Bernt
1c4158c373
- Created FIELD_TRIAL_LOG.md — observation protocol for real customer cases - Log entry template with 10 fields - Two example entries (accepted and rejected decisions) - Questions the log answers: adoption, accuracy, calibration, rejection analysis - Future KPIs: Decision Adoption Rate, Decision Accuracy, Time to Decision - Updated MEMORY.md with Control Intelligence & Decision Model section - Decision Pipeline v1.0 summary - Decision Object v1.0 (STRUCTURE FROZEN) - Status: VALIDATED FOR FIELD TRIALS - Three target customer cases - Future KPIs - Key principle: No more modeling until first real customer case - Added Decision Adoption Rate and Decision Accuracy as future product KPIs - Not implemented yet — start collecting data now, calculate later - Decision Adoption = accepted / total recommendations - Decision Accuracy = correct / total decisions Rationale: Stop modeling, start observing. First real customer case will teach more than the last twenty documents combined.
114 lines
2.7 KiB
Markdown
114 lines
2.7 KiB
Markdown
# FIELD TRIAL LOG
|
|
|
|
**Observation Protocol for Decision Model Validation**
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Version** | 1.0 |
|
|
| **Status** | ACTIVE |
|
|
| **Purpose** | Collect raw data from real customer cases |
|
|
|
|
---
|
|
|
|
## Log Entry Template
|
|
|
|
| Field | Example | Notes |
|
|
|-------|---------|-------|
|
|
| **Date** | 2026-07-15 | |
|
|
| **Customer Type** | Municipality / Property Owner / Contractor | |
|
|
| **Decision** | Inspect Road A12 | What the system recommended |
|
|
| **Decision Accepted** | Yes / No | Did the user accept? |
|
|
| **Action Executed** | Yes / No / Pending | Was the action carried out? |
|
|
| **Outcome** | Crack confirmed | What actually happened |
|
|
| **Time to Decision** | 3 minutes | How long from login to decision |
|
|
| **User Feedback** | "We trusted the recommendation." | Verbatim or summary |
|
|
| **Confidence** | 87% | System confidence at decision time |
|
|
| **Notes** | Free text | Anything else observed |
|
|
|
|
---
|
|
|
|
## Example Entries
|
|
|
|
### Entry 1
|
|
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| Date | 2026-07-15 |
|
|
| Customer Type | Municipality |
|
|
| Decision | Inspect Road A12 |
|
|
| Decision Accepted | Yes |
|
|
| Action Executed | Yes |
|
|
| Outcome | Crack confirmed, repaired within 14 days |
|
|
| Time to Decision | 3 minutes |
|
|
| User Feedback | "We trusted the recommendation." |
|
|
| Confidence | 87% |
|
|
| Notes | User clicked through Explainability chain |
|
|
|
|
### Entry 2
|
|
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| Date | 2026-07-16 |
|
|
| Customer Type | Property Owner |
|
|
| Decision | Repair facade immediately |
|
|
| Decision Accepted | No |
|
|
| Action Executed | No |
|
|
| Outcome | Deferred, user chose to wait |
|
|
| Time to Decision | 5 minutes |
|
|
| User Feedback | "Too expensive right now." |
|
|
| Confidence | 92% |
|
|
| Notes | Confidence was high but user rejected due to cost |
|
|
|
|
---
|
|
|
|
## Questions This Log Answers
|
|
|
|
After collecting entries:
|
|
|
|
- Were recommendations accepted? (Decision Adoption Rate)
|
|
- Were actions executed? (Execution Rate)
|
|
- Did outcomes match predictions? (Decision Accuracy)
|
|
- Was confidence reasonable? (Calibration)
|
|
- Which recommendations were ignored? (Rejection Analysis)
|
|
- How long did decisions take? (Time to Decision)
|
|
|
|
---
|
|
|
|
## Future KPIs
|
|
|
|
### Decision Adoption Rate
|
|
|
|
```
|
|
Recommendations accepted
|
|
────────────────────────
|
|
Total recommendations
|
|
```
|
|
|
|
Example: 72 of 100 = **72%**
|
|
|
|
### Decision Accuracy
|
|
|
|
```
|
|
Decisions that proved correct
|
|
─────────────────────────────
|
|
Total decisions
|
|
```
|
|
|
|
Example: 65 of 72 executed = **90%**
|
|
|
|
**Note:** These require months of data. Start collecting now, calculate later.
|
|
|
|
---
|
|
|
|
## Status
|
|
|
|
**ACTIVE — Awaiting first customer case**
|
|
|
|
---
|
|
|
|
## ändringshistoria
|
|
|
|
| Version | Datum | Beskrivning |
|
|
|---------|-------|-------------|
|
|
| 1.0 | 2026-07-02 | Initial field trial log template |
|