boc/docs/design/FIELD_TRIAL_LOG.md

# FIELD TRIAL LOG

**Observation Protocol for Decision Model Validation**

| | |
|---|---|
| **Version** | 1.0 |
| **Status** | ACTIVE |
| **Purpose** | Collect raw data from real customer cases |

---

## Log Entry Template

| Field | Example | Notes |
|-------|---------|-------|
| **Date** | 2026-07-15 | |
| **Customer Type** | Municipality / Property Owner / Contractor | |
| **Decision** | Inspect Road A12 | What the system recommended |
| **Decision Accepted** | Yes / No | Did the user accept? |
| **Action Executed** | Yes / No / Pending | Was the action carried out? |
| **Outcome** | Crack confirmed | What actually happened |
| **Time to Decision** | 3 minutes | How long from login to decision |
| **User Feedback** | "We trusted the recommendation." | Verbatim or summary |
| **Confidence** | 87% | System confidence at decision time |
| **Notes** | Free text | Anything else observed |

---

## Example Entries

### Entry 1

| Field | Value |
|-------|-------|
| Date | 2026-07-15 |
| Customer Type | Municipality |
| Decision | Inspect Road A12 |
| Decision Accepted | Yes |
| Action Executed | Yes |
| Outcome | Crack confirmed, repaired within 14 days |
| Time to Decision | 3 minutes |
| User Feedback | "We trusted the recommendation." |
| Confidence | 87% |
| Notes | User clicked through Explainability chain |

### Entry 2

| Field | Value |
|-------|-------|
| Date | 2026-07-16 |
| Customer Type | Property Owner |
| Decision | Repair facade immediately |
| Decision Accepted | No |
| Action Executed | No |
| Outcome | Deferred, user chose to wait |
| Time to Decision | 5 minutes |
| User Feedback | "Too expensive right now." |
| Confidence | 92% |
| Notes | Confidence was high but user rejected due to cost |

---

## Questions This Log Answers

After collecting entries:

- Were recommendations accepted? (Decision Adoption Rate)
- Were actions executed? (Execution Rate)
- Did outcomes match predictions? (Decision Accuracy)
- Was confidence reasonable? (Calibration)
- Which recommendations were ignored? (Rejection Analysis)
- How long did decisions take? (Time to Decision)

---

## Future KPIs

### Decision Adoption Rate

```
Recommendations accepted
────────────────────────
Total recommendations
```

Example: 72 of 100 = **72%**

### Decision Accuracy

```
Decisions that proved correct
─────────────────────────────
Total decisions
```

Example: 65 of 72 executed = **90%**

**Note:** These require months of data. Start collecting now, calculate later.

---

## Pilot Structure

**Four phases — product research, not marketing:**

| Phase | What You Do | What You Learn |
|-------|-------------|----------------|
| **1. Collection** | Film, photograph, create observations | What can actually be detected? |
| **2. Analysis** | Let pipeline create Findings and Decisions | Are decisions understandable and relevant? |
| **3. Verification** | Compare with reality on site | Was the recommendation correct? |
| **4. Reflection** | Document what worked and what didn't | What needs to change in the model? |

**Full chain captured:**
```
Reality → Observation → Evidence → Finding → Decision → Action → Outcome → Learning
```

## What to Measure

**Not:** Did AI find a crack?
**But:** Did this become a decision a real person could act on?

**Example:**
- AI finds 23 cracks
- LandveX says: "Inspect Road A12 within 30 days"
- **The second is the product.**

## Document "Non-Decisions"

Also valuable:
- "No action needed"
- "More data required"
- "Cannot recommend anything yet"

If the system always tries to give advice even when evidence is weak, you risk building a system that feels confident when it shouldn't.

## Film the Workflow

Since you are conducting the pilots, film more than just infrastructure. Also film:
- How you find an area
- Why you choose a mission
- How you document
- How long it takes
- What feels unclear
- When you become uncertain
- When the system saves time

This material becomes invaluable for product development, onboarding, training, and sales.

## Observer Mindset

**Principle:** If data contradicts the model, the model should change.

Use the same approach during pilots. If you notice:
- A step feels unnecessary
- A decision becomes unclear
- A recommendation cannot be acted on

**Document first, change model later.** Build the model from real workflows instead of assumptions.

**The first 20–50 real missions will give you more valuable product knowledge than months of additional modeling.**

## Status

**ACTIVE — Awaiting first customer case**

---

## ändringshistoria

| Version | Datum | Beskrivning |
|---------|-------|-------------|
| 1.0 | 2026-07-02 | Initial field trial log template |
| 1.1 | 2026-07-02 | Added pilot phases, measurement principles, filming protocol |