boc/workcamp/VOL3-Infrastructure-Operations.md

# AMOS Development Workcamp 2026
# Volume 3: Infrastructure and Operations History
# Period: 11 maj – 7 juni 2026
# Genererat: 2026-06-07 | Klassifikation: INTERN – KONFIDENTIELL

---

## 1. AWS-INFRASTRUKTUR ÖVERSIKT

### 1.1 AWS-konto och Region

| Parameter | Värde |
|-----------|-------|
| **AWS-konto** | 155407238699 |
| **Primär region** | eu-north-1 (Stockholm) |
| **Sekundär region** | eu-west-1 (Irland) – Mac Build Server |
| **IAM-profil (agent)** | platform-ai-agent |
| **Instans-roll** | platform-ec2-ssm-role (AmazonSSMManagedInstanceCore) |

### 1.2 EC2-instanser

#### server-2 (primär produktionsserver)
| Parameter | Värde |
|-----------|-------|
| **Instance ID** | i-09b2204a52c2f33c9 |
| **Typ** | ARM64 (aarch64) |
| **Region/AZ** | eu-north-1b |
| **Elastic IP** | 16.170.83.169 |
| **Hostname** | bernt.wavult.com / bernt.aamos.systems |
| **OS** | Amazon Linux 2023 (ARM64), kernel 6.18.30-61.116.amzn2023.aarch64 |
| **RAM** | 6.0GB (Node.js) |
| **SSM** | Online (AmazonSSMManagedInstanceCore) |
| **SG** | sg-070aca8fc94a5a91b |

#### host-2 / server-3 (redundanshost, RES-004)
| Parameter | Värde |
|-----------|-------|
| **Instance ID** | i-0fc20ddba6bb17ead (v2, DNS-fix) |
| **Privat IP** | 172.31.20.206 |
| **Region/AZ** | eu-north-1a |
| **AMI-origin** | ami-0fc5c75dc45812df9 (server-2 baseline snapshot) |
| **Skapad** | 2026-06-05 03:09 UTC |
| **SSM** | Online |
| **DNS-fix** | /etc/resolv.conf: nameserver 172.31.0.2, chattr +i |

#### GPU-box / gpu-training (ALETHEIA-träning)
| Parameter | Värde |
|-----------|-------|
| **Instance ID** | i-04239ef03d26b218b |
| **Typ** | A10G 23GB VRAM |
| **Publik IP** | 13.61.176.142 |
| **Privat IP** | 172.31.36.61 |
| **SSH-nyckel** | gpu_deploy_key |
| **EJ SSM-styrd** | SSH-baserad åtkomst |
| **Status** | STANNAR igång (Erik: fortsatt träning) |
| **Serverar** | vLLM :8000 (ALETHEIA) |

#### p4d.24xlarge SPOT (DDP-träning, TERMINERAD)
| Parameter | Värde |
|-----------|-------|
| **Instance ID** | i-0e9e845b6d85af5a0 |
| **Typ** | p4d.24xlarge (8x A100-SXM4-40GB) |
| **Region/AZ** | eu-north-1c |
| **Publik IP** | 13.48.129.246 |
| **Privat IP** | 172.31.1.199 |
| **Kostnad** | $8/h SPOT (cap $12/h) |
| **DLAMI** | PyTorch 2.7 |
| **Status** | TERMINERAD efter v5-träning (ALETHEIA klar) |

#### Mac Build Server (eu-west-1)
| Parameter | Värde |
|-----------|-------|
| **Instance ID** | i-0fde74767ebfcbdd8 |
| **AMI** | ami-02ab2839514384495 |
| **Dedicated Host** | h-0298aacec652d75b3 (eu-west-1a) |
| **Snapshot** | snap-00d2f39d8775caa67 (completed) |
| **Publik IP** | 3.249.41.127 |
| **SSH** | `ssh -i ~/.openclaw/workspace/mac-build-key.pem ec2-user@3.249.41.127` |
| **macOS** | 26.4.1 (BuildVersion: 25E253) |
| **Runner** | wavult-mac-build-eu-01, Runner ID: 2 |

### 1.3 RDS (PostgreSQL)

#### platform-identity-core (primär produktions-DB)
| Parameter | Värde |
|-----------|-------|
| **Endpoint** | platform-identity-core.cvi0qcksmsfj.eu-north-1.rds.amazonaws.com |
| **DB-namn** | amos (schema: quixzoom, wavult_identity, m.fl.) |
| **Initial klass** | db.t4g.small (2GB RAM) |
| **Klass efter RES-003** | **db.r6g.large (16GB RAM)** – 8x ökning |
| **Multi-AZ** | **TRUE** (aktiverat RES-001, 2026-06-04) |
| **Backup retention** | 35 dagar + daglig auto-snapshot |
| **SG** | sg-0feb828ee1ff94071 |
| **SSM-access** | Instansroll saknar secrets-access → använd platform-ai-agent |

> **KRITISK LÄRDOM:** DNS-resolution av RDS sker via Tailscale MagicDNS (100.100.100.100) på server-2. host-2 saknade Tailscale → ENOTFOUND. Löst med statisk VPC-resolver (172.31.0.2) via chattr.

#### aamos-postgres (K8s/EKS DB)
- Endpoint: aamos-postgres/aamos_prod (intern EKS-resurs)
- OBS: Skiljer sig från identity-core – restaurangtjänster använder identity-core/amos

### 1.4 Application Load Balancer (ALB)

| Parameter | Värde |
|-----------|-------|
| **Namn** | aamos-prod-alb |
| **DNS** | aamos-prod-alb-709257248.eu-north-1.elb.amazonaws.com |
| **AZ:er** | eu-north-1a, eu-north-1b, eu-north-1c |
| **Target Group** | aamos-prod-tg |
| **Targets** | server-2 (1b, HEALTHY) + host-2 (1a, HEALTHY) |
| **Health check** | HTTP:3209/health/ready (djup DB-check) |
| **HTTPS listener** | *.wavult.com + quixzoom.com cert (SNI) |
| **HTTP redirect** | 301 → HTTPS |
| **ALB-SG** | sg-0cac842012eae85dd |
| **DNS live** | quixzoom-api.wavult.com → ALB alias |

### 1.5 S3

| Bucket | Användning |
|--------|-----------|
| `quixzoom-landing-prod` | Statisk sajt (CloudFront EE30B9WM5ZYM7) |
| `platform-ouroboros-models` | AI-modeller. `aletheia-eslm-v5-20260607/` (14 objekt) |

### 1.6 EKS-kluster

| Parameter | Värde |
|-----------|-------|
| **Klusternamn** | wavult |
| **Version** | v1.31 |
| **Auth-mode** | API_AND_CONFIG_MAP |
| **Noder** | 3 Ready (graviton r6g/r8g, arm64) |
| **x86-pool** | c6i (desired=0, vilande) |
| **ECR** | 155407238699.dkr.ecr.eu-north-1.amazonaws.com |

**Installerade controllers/operators:**
- ArgoCD (GitOps)
- cert-manager
- external-dns
- external-secrets (ClusterSecretStore: `aws-ssm-store` → AWS SSM)
- Kong gateway (NLB k8s-gateway-konggate)
- aws-load-balancer-controller
- Karpenter (graviton-spot nodepool)
- metrics-server
- Kafka (KRaft, 3-broker StatefulSet `wavult-kafka`, wavult ns)

**Namespaces:**
- `rust` – Rust-workloads (7 ClusterIP-services definierade)
- `gateway` – Kong Ingress
- `wavult` – Kafka + bakgrundsinfra
- `argocd` – GitOps
- `aamos` – aamos-event-bus-svc :3383

**Migrationsstatus 2026-06-07:**
- tickets-tjänst: KLAR i EKS (2 pods, arm64, Kong-route, RDS-verifierad)
- ledger: ECR-repo skapad, EJ migrerad
- 15 övriga restaurant-tjänster: EJ migrerade

### 1.7 AWS Secrets Manager

| Secret | ARN-suffix | Innehåll |
|--------|-----------|---------|
| `/wavult/prod/core` | — | Core-nycklar |
| `/wavult/prod/identity` | — | Identity-nycklar |
| `/wavult/prod/amos-db-url` | — | RDS URL |
| `/wavult/prod/ledger-db-url` | — | Ledger DB URL |
| `wavult/codebuild/gitea-token` | — | Gitea CI-token |
| `aamos/prod/mail-accounts` | ...NCx2TX | 21 Mailu-kontolösenord |
| `/aamos/prod/restaurant/DATABASE_URL` | — | Restaurant-tjänsters DB |

> **OBS:** Instansroll saknar secrets-access. Åtkomst VIA profil `platform-ai-agent`.

### 1.8 Cloudfront

| Distribution | Origin | Domäner |
|-------------|--------|---------|
| EE30B9WM5ZYM7 | S3 quixzoom-landing-prod | quixzoom.com, www.quixzoom.com |

### 1.9 CodeBuild

- build-wavult-core
- build-amos-core
- mirror-gitea
- hypbit-build-landvex-api

---

## 2. NÄTVERKSARKITEKTUR

### 2.1 VPC och Subnets

- VPC: eu-north-1 (Standard AWS VPC)
- Subnets: eu-north-1a, eu-north-1b, eu-north-1c
- DNS: AmazonProvidedDNS (172.31.0.2) – CANONICAL resolver för prod
- Tailscale: MagicDNS 100.100.100.100 – server-2 primär resolver (BRÄCKLIGT för multi-host)

### 2.2 Tailscale

- server-2 är Tailscale-ansluten (100.100.100.100)
- **ARKITEKTUR-FLAGGA:** Prod DB-resolution beror på Tailscale MagicDNS – bräckligt
- **ÅTGÄRD (2026-06-05):** VPC-resolver som fallback via systemd-resolved.conf.d/99-vpc-fallback.conf på server-2. FallbackDNS=172.31.0.2 + Domains ~rds.amazonaws.com.

### 2.3 Security Groups

| SG | Roll | Kritiska regler |
|----|------|----------------|
| sg-070aca8fc94a5a91b | server-2 + host-2 (hosts) | Self-reference inlagd 2026-06-05 (443/3100/3209 från sig själv) |
| sg-0cac842012eae85dd | ALB | — |
| sg-0feb828ee1ff94071 | RDS | Tillåter host-SG (070a) på 5432 |

> **KRITISK FIX (2026-06-05 02:38 UTC):** sg-070aca8fc94a5a91b saknade self-reference → host→host (och host→egen-priv-IP) blockerat. Tillagd: inbound 443/3100/3209 från egen SG. host-2 fick omedelbart RDS-access på 3209.

### 2.4 DNS (Route53)

| Zon | Zone ID |
|-----|--------|
| aamos.systems | Z03908683A2AIMRZR85BQ |
| wavult.com | 5bed27e91d719b3f9d82c234d191ad99 (Cloudflare, avvecklat 2026-05-16) |
| quixzoom.com | e9a9520b64cd67eca1d8d926ca9daa79 (Cloudflare) |

**DNS-migrationsstatus (2026-06-05/06):**
- 66 Route53-zoner inventerade (C1 2026-06-06)
- Kategori EU: 38 zoner (hanteras av teamet)
- Kategori övriga marknader: 13 zoner (partner-modell, hold tills Erik beslutar)
- Dubbletter/döda: 4 zoner

---

## 3. TJÄNSTER OCH PORTAR

### 3.1 Systemd-tjänster (Rust – ARM64)

| Tjänst | Port | Kategori |
|--------|------|---------|
| aamos-api-gateway | 3206 | Core |
| aamos-rule-engine | 3201 | Core |
| aamos-tolkserver | 3204 | Core (Primal Charter §1.1-1.8) |
| aamos-stammregister | 3205 | Core |
| aamos-gecl-chain | 3263 | Core (Governed Evidence Chain Ledger) |
| aamos-health-aggregator | 3299 | Core (hårdkodad port) |
| aamos-financial-engine | 3203 | Finance |
| aamos-ledger | 3250 | Finance (dubbel bokföring, BAS, SIE4) |
| aamos-sie-import | 3254 | Finance |
| aamos-bank-import | 3253 | Finance |
| aamos-billing-engine | 3392 | Finance |
| aamos-payroll-rust | 3285 | Finance/HR |
| aamos-invoice-rust | 3286 | Finance |
| aamos-people-core | 3271 | HR |
| aamos-hr-rust | 3284 | HR |
| aamos-operations-core | 3272 | Ops |
| aamos-commerce-rust | 3280 | Commerce |
| aamos-crm-rust | 3282 | Commerce |
| aamos-campaign-rust | 3283 | Commerce |
| aamos-compliance-rust | 3287 | Compliance |
| aamos-dynasty-engine | 3210 | Governance (hårdkodad port) |
| aamos-venture | 3316 | Governance |
| aamos-analytics-engine | 3211 | Analytics |
| aamos-analytics-rust | 3281 | Analytics |
| aamos-meeting-service | 3308 | Collab |
| aamos-quixzoom-engine | 3390 | quiXzoom |
| aamos-landvex-engine | 3391 | LandveX |
| aamos-dissg-engine | 3360 | Gov/Intel (DISSG – Kolada) |
| aamos-seep | 3350 | Gov/Intel (SEEP – Kvittovalvet Pan-EU) |
| aamos-mlcs-engine | 3370 | Intelligence |
| amos-core | 3100 | Core (AI Brain, LLM-router) |

#### Restaurant-systemd (Rust – ARM64, server-2)
| Tjänst | Port |
|--------|------|
| aamos-restaurant-pos | 3300 |
| aamos-restaurant-inventory | 3301 |
| aamos-restaurant-hr | 3302 |
| aamos-restaurant-ops | 3303 |
| aamos-restaurant-onboarding | 3304 |
| aamos-restaurant-menu | 3305 |
| aamos-restaurant-compliance | 3306 |
| aamos-restaurant-analytics | 3307 |
| aamos-restaurant-delivery | 3309 |
| aamos-restaurant-licensing | 3310 |
| aamos-restaurant-kiosk | 3311 |
| aamos-restaurant-estax | 3312 |
| aamos-restaurant-eslocal | 3313 |
| aamos-restaurant-screens | 3314 |
| aamos-restaurant-designer | 3315 |
| aamos-restaurant-tickets | 3318 → **EKS (rust ns, 2 pods)** |

### 3.2 Systemd-tjänster (Node.js)

| Tjänst | Port |
|--------|------|
| ouroboros-finance | 3202 |
| ouroboros-identity | 3207 |
| ouroboros-payroll | 3231 |
| ouroboros-expenses | 3226 |
| aamos-audit-engine | 3251 |

### 3.3 PM2-processer (Node.js)

| Namn | Port | Roll |
|------|------|------|
| quixzoom-api | 3209 | quiXzoom API (bakom ALB) |
| ai-gateway | 3270 | LLM-routing (6 providers, auto-failover) |
| provider-monitor | — | Kredit/health-bevakning |
| audit-proxy | — | Audit-proxy (bryggar 3251→3260) |
| ouroboros-multiorg | — | Multi-org orchestration |
| ouroboros-finance-proxy | — | Finance proxy |
| ouroboros-identity-proxy | — | Identity proxy |
| ouroboros-people-proxy | — | People proxy |
| ouroboros-commerce-proxy | — | Commerce proxy |
| ouroboros-time | — | Time tracking |
| mail-admin | — | Mailadmin |

### 3.4 Externa tjänster (vLLM, NATS)

| Tjänst | Host | Port | Status |
|--------|------|------|--------|
| vLLM (ALETHEIA) | 172.31.36.61 (gpu-training) | 8000 | LIVE, nohup (EJ systemd) |
| NATS | server-2 | — | Systemd |
| Kafka | EKS wavult ns | — | 3/3 brokers healthy |

---

## 4. DOMÄNMIGRATION wavult.com → aamos.systems

### 4.1 Bakgrund

Erik (2026-06-05): "Vi driver affären Aamos i Aamos. wavult ska ALDRIG synas i domäner."

### 4.2 Migreringsstatus (2026-06-05/06)

**Live på aamos.systems (200+):**
- os, pro, ai, api, analytics, commerce, amos, bernt, git, gitea, grafana/monitor, watchman, sales, tasks, identity, payments, gfiae, audit

**Produktdomäner (live):**
- aamos.ai (+www +api)
- api.quixzoom.com

**301-redirects (gammal wavult.com → ny):**
- ouroboros→os, argus→watchman, staff→team, jira/plane→tasks, quixzoom-api→api.quixzoom.com, gate/kong-admin→api.aamos.systems

**Maintenance (503):**
- team (port 3510 ej deployad)
- vault (172.31.42.221 nedlagd)

### 4.3 nginx-konfiguration

Huvudfil: `/etc/nginx/sites-available/aamos.systems`

**Subdomäner i aamos.systems-filen:**
- `os.aamos.systems` → proxy 127.0.0.1:3100
- `pro.aamos.systems` → proxy 127.0.0.1:3100
- `ai.aamos.systems` → proxy 127.0.0.1:3100
- `api.aamos.systems` → microservices gateway

**Egna dedikerade vhost-filer:**
- audit.aamos.systems, gfiae.aamos.systems, git.aamos.systems, gitea.aamos.systems
- grafana.aamos.systems, identity.aamos.systems, payments.aamos.systems, watchman.aamos.systems

> **NGINX-PRIVKEY-BUGG (fixad):** Alla certs i /etc/letsencrypt/archive/*/privkey*.pem hade 640 (ec2-user) → nginx-worker (uid 101) kunde ej läsa → krasch vid reload. Fixat till 644. Viktigt vid framtida cert-skapande.

---

## 5. MAILINFRASTRUKTUR (MAILU)

### 5.1 Stack

| Komponent | Detalj |
|-----------|--------|
| **Stack** | Mailu 2.0 (postfix/dovecot/rspamd/admin/webmail) |
| **Mailserver** | mail.wavult.com @ 16.170.83.169 |
| **Utgående relay** | Resend (resend.com) |
| **TLS-cert** | mail.wavult.com (EJ mail.aamos.systems – SAN saknas) |
| **Admin** | Mailu admin-panel |
| **Docker** | Docker-stack live |

### 5.2 Brevlådor (17 st)

**aamos.systems (10):**
- erik, winston, dennis, johan, leon (RADERAD 2026-06-06), bernt
- info, support, hr, no-reply
- Roll-brevlådor: ceo, cfo, coo, cto (+ cmo-alias)

**landvex.com (3):** info, support, no-reply

**quixzoom.com (4):** info, support, no-reply, zoomers

### 5.3 Alias-routing (24 alias, multi-hop)

**Funktionsadress → ROLL → Person-arkitektur:**
- sales/marketing@aamos.systems → cmo → cto (CMO vakant, temporärt)
- billing/finance/invoice@aamos.systems → cfo → winston
- legal@aamos.systems → coo → dennis
- security/cto@aamos.systems → cto → johan
- ceo@aamos.systems → ceo → erik

**LIVE-test (verifierat):**
- sales@aamos.systems → cmo → ceo → erik: status=sent ✅
- billing@landvex.com → cfo → winston: status=sent ✅

### 5.4 DNS-konfiguration (Route53)

| Record | Värde |
|--------|-------|
| MX | 10 mail.wavult.com |
| SPF | v=spf1 mx a:mail.wavult.com include:spf.resend.com ~all |
| DKIM | wavult._domainkey.aamos.systems (RSA 2048, chunkad 255-bitar) |
| DMARC | p=none (mjuk start, rua→postmaster@aamos.systems) |

### 5.5 Kvarstående Mailuppgifter

- TLS-cert: lägg till mail.aamos.systems som SAN
- DMARC: höj p=none → p=quarantine/reject (efter 1-2v verifiering)
- ~60 övriga domäner: partner-modell eller catch-all-redirect, väntar Eriks beslut

---

## 6. AI-INFRASTRUKTUR

### 6.1 AI-Gateway (ai-gateway, PM2, port 3270)

**Fil:** `/opt/amos/data/ai-gateway/gateway.mjs`

**Failover-kedjor (Grok exkluderad tills påfylld):**
| Task | Primär → Fallback 1 → Fallback 2 |
|------|----------------------------------|
| reasoning | openai(o3) → anthropic(opus) → deepseek |
| fast | groq → gemini → openai(4o-mini) |
| search | perplexity (ingen fallback) |
| creative | openai(gpt-4o) → anthropic(sonnet) → gemini |
| financial | anthropic(claude) → openai(o3) → groq |
| default | anthropic(sonnet) → openai(4o) → groq |

**Endpoints:** POST /v1/chat, GET /health, /metrics, /providers
**Circuit-breaker:** Per provider (öppnas efter N fel, halvöppen efter cooldown)

### 6.2 Provider-monitor (PM2)

**Fil:** `/opt/amos/data/ai-gateway/monitor.mjs`
- Pollar 7 providers var 5:e min
- Auto-larm CRITICAL: Winston (kredit) + Johan (nedtid) via HomoDeus
- Output: provider-status.json

### 6.3 AWS Bedrock

| Modell | Status |
|--------|--------|
| Qwen3-32B (aws) | LIVE – primär för /api/aamos/chat |
| Claude Sonnet 4.5 (Bedrock eu-north-1) | UPPLÅST (FTU-formulär inskickat 2026-06-06 via API, root) |
| Claude Opus 4.6 (Bedrock eu-north-1) | UPPLÅST |
| Claude Opus 4.8 (direkt Anthropic-nyckel) | LIVE – poliskontroll/domare |

> **OBS:** Bedrock-Claude var gated (use-case-formulär ej ifyllt). Lösning: PutUseCaseForModelAccess API anropades direkt med root-access. Nu upplåst för konto 155407238699 eu-north-1.

### 6.4 ALETHEIA (ESLM v5)

| Parameter | Värde |
|-----------|-------|
| **Basmodell** | Qwen2.5-7B-Instruct (unsloth/Qwen2.5-7B-Instruct-bnb-4bit) |
| **Träningsdata** | v5: 20807 train / 2319 val, 0 företagstermer |
| **LoRA config** | r=16, alpha=32, dropout=0.05, 7 target_modules |
| **Träningsresultat** | train_loss 0.35, token-accuracy 94%, 3 epoker, 57 min (8x A100) |
| **Serving** | vLLM fp16, VRAM 21GB, served-name: aletheia |
| **Endpoint** | 172.31.36.61:8000/v1 |
| **Bugfix** | VLLM_USE_FLASHINFER_SAMPLER=0 + --enforce-eager |
| **Säkrad** | server-2 /opt/amos/models/aletheia-v5/ (15GB) + S3 + A10G |

**Versionshistorik:**
- v1 (amos-r2 original): 33473 ex, fel org-nr (559316-3881), LandveX=bostad → BLOCKED 33%
- v2: 33352 ex, org-nr fixat, ej omtränats
- v3: 33352 ex (canon-datafix), Leon/Vincento kvar → KASSERAD (kontaminerad)
- v4: 33178 ex (Leon/Vincento droppade) → KASSERAD (v4 hade fortf. företagstermer)
- **v5 (KLAR): 20807 ex, 0 företagstermer, generell evidensmotor → DEPLOYAD**

---

## 7. SÄKERHETSHÄNDELSER OCH ÅTGÄRDER

### 7.1 Kritiska Säkerhetssårbarheter – Reparerade

| CVSS | Sårbarhet | Status | Datum |
|------|-----------|--------|-------|
| **10.0** | JWT Algorithm Confusion – jwt.decode() istf jwt.verify() i 6 filer | ✅ Fixad | 2026-05-06 |
| **Kritisk** | POST /api/qz/payments/credits/add – INGEN auth (obegränsad credits-minting) | ✅ Fixad | 2026-06-05 |
| **Kritisk** | Stripe-webhook – ingen idempotens (dubbel kreditering vid retry/multi-host) | ✅ Fixad | 2026-06-05 |
| **Hög** | /api/qz/assistant – öppen LLM-endpoint (AI-kostnadsabuse) | ✅ Rate-limited | 2026-06-05 |
| **Medel** | quiXzoom.submissions saknade approval-endpoint (zoomers aldrig betalade) | ✅ Fixad | 2026-06-05 |

### 7.2 Latenta SSL-buggar (Fixade)

Alla `.mjs`-filer med `new Pool({connectionString: ...})` utan SSL:
- tool-registry.mjs → FIXED (ssl:{rejectUnauthorized:false})
- ouroboros-notifications-routes.mjs → FIXED
- timetrack/analytics-engine.mjs → FIXED
- scaffold/generator.mjs + transitions.mjs → Skippade (code templates, ej mot RDS)

### 7.3 nginx privkey chmod-bugg (Fixad)

Privkey-filer i /etc/letsencrypt/archive/*/privkey*.pem hade chmod 640 (ec2-user:ec2-user) → nginx-worker (uid 101) kraschade vid reload. Fixat till 644.

---

## 8. INCIDENTER MED RCA

### INC-001: xAI/Grok kredit-haveri (2026-06-04 21:43 UTC)
**Symptom:** AI-orkestrering #1c stoppade utan förvarning  
**Rotorsak:** xAI/Grok API-nyckel hade 0 kr kvar, ingen auto-failover  
**Impact:** Strategisk AI-orkestrering stoppades (ej kundpåverkande)  
**Fix:** AI-Gateway med auto-failover + circuit-breaker. Provider-monitor med CRITICAL-larm.  
**Permanent åtgärd:** ABSOLUT REGEL – varje provider = primärt Revolut-kort + fallback-kort + auto-topup + buffert (Tickets: Johan PROV-001..006, Winston PROV-W1..6)

### INC-002: LandveX-systemkunskap-haveri (2026-06-06)
**Symptom:** AI-systemet trodde LandveX = "bostad/mäklare"  
**Rotorsak (4 lager):**
1. RAG-index existerade inte på disk
2. rag.mjs indexerade bara 3 filer (ej styrningspaketet)
3. Stubs svarade ok:true (dolde att RAG var dött)
4. 39 konkurrerande truth-filer (ingen enskild sanningskälla)
**Impact:** Felaktig produktdefinition i AI-assistenten (produktionschat)  
**Fix:** AAMOS_CANON.md skapad, rag.mjs patchad, RAG-index byggt om (94 chunks), wavult-facts.mjs fixad, orphan-fil neutraliserad  
**Permanent åtgärd:** AAMOS_CANON.md = ENDA kanoniska sanningskällan (E4). Separation: wavult-facts.mjs (manuell) + CANON/RAG separata källor.

### INC-003: host-2 DNS/RDS-fel (2026-06-05 03:00 UTC)
**Symptom:** host-2 (server-3) fick ENOTFOUND på RDS-endpoint  
**Rotorsak:** AMI-snapshot fångade server-2:s resolv.conf med 100.100.100.100 (Tailscale). host-2 utan Tailscale → resolvern ej nåbar → RDS-DNS ouppslag  
**Impact:** host-2 hölls automatiskt ur ALB-rotation (djup health-check 503 → korrekt beteende)  
**Fix:** Terminerade trasig klon, ny host-2 v2 med user-data som skriver resolv.conf + chattr +i (nameserver 172.31.0.2)  
**Permanent åtgärd:** server-2 fick FallbackDNS via systemd-resolved.conf.d/99-vpc-fallback.conf. ARKITEKTUR: Lägg VPC-resolver som primär/fallback på ALLA prod-hostar.

### INC-004: amos-core OOM via SocialAuto restart-storm (2026-06-05)
**Symptom:** Alla domäner mot port 3100 timeout. amos-core: 5.2G/6.0G RAM, 1850 tasks, NATS 503-flod  
**Rotorsak:** SocialAuto (social-account-automation.mjs) relanserade vid varje restart ALLA QUEUED/FAILED-jobb utan concurrency-tak → 137 stealth-Chromium-instanser (13.4 GB) → OOM → event-loop svalt  
**Fix:** Cancellerade 274 skenande jobb. PATCH: global semafor SOCIALAUTO_MAX_CONCURRENT=3 runt runAutomation()  
**Lärdom:** Vid amos-core-strul – kolla ALLTID `pgrep -fc chrome` och social-account-jobs.json status-counts FÖRST.

### INC-005: Kafka degraderad 8 dagar (2/3 brokers) (Uppdagad 2026-06-07)
**Symptom:** kafka-1 saknad, 2/3 brokers i ≥8 dagar  
**Rotorsak 1:** kafka-1 init-container `fix-volume` saknade resource limits → ResourceQuota `wavult-quota` blockerade (FailedCreate x721 över 8 dagar)  
**Rotorsak 2:** kafka-1:s PVC bunden till raderad EBS-volym vol-0891ca1531dc90def (InvalidVolume.NotFound)  
**Fix:** Patchade initContainer med resource limits. Raderade orphaned PVC + pod. EBS-CSI provisionerade ny 20Gi gp3. Broker rejoinade via RF=3-replikation.  
**Permanent åtgärd:** Broker-hälso-monitoring saknas → observability-gap att åtgärda.

---

## 9. CERTIFIKAT

| Domän | CA | Giltig t.o.m. | Nyckeltyp |
|-------|-----|--------------|----------|
| landvex.com + www | ZeroSSL | 2026-09-02 | EC |
| *.wavult.com | Let's Encrypt | Rollande | RSA |
| mail.wavult.com | Let's Encrypt | Rollande | RSA |
| quixzoom.com | ACM (ALB) | Hanteras av AWS | RSA |

---

*Volume 3 av 7 | AMOS Development Workcamp Reconstruction Package*