diff --git a/projects/landvex/README.md b/projects/landvex/README.md index e26292324..00841b23e 100644 --- a/projects/landvex/README.md +++ b/projects/landvex/README.md @@ -3,82 +3,125 @@ > Kunskapsgraf över visuellt identifierbar fysisk infrastruktur. > Grafen är produkten. Allt annat — appar, API:er, dashboards — är gränssnitt mot grafen. -## Struktur +## Arkitektur (skalad för 50GB+ agentkraft) ``` -projects/landvex/ -├── 00-styrning/ # Validering, verifiering -├── 10-data/ # Master-data (JSON) -│ ├── lvx-klassposter-master-v3.json -│ ├── lvx-produktmodeller-*.json -│ ├── lvx-graf-kanter-v3.json -│ └── lvx-id-register.json -├── 20-automation/ # Ingest-pipeline -│ ├── ingest.py -│ └── intag/ # Kandidatposter per våg -├── 30-produkt/ # API-spec, demosvar -├── 40-strategi/ # Genome-strategi, IOL-spec -├── api/ # FastAPI Identify-API -├── db/ # Neo4j-laddning -├── scripts/ # Export, bygghjälp -├── arkiv/ # Ersatta versioner -└── build.sh # Master byggscript +┌─────────────────────────────────────────────────────────────────┐ +│ LANDVEX IOL v3 │ +├─────────────────────────────────────────────────────────────────┤ +│ SaaS-lager │ Identify API │ Tillverkarportal │ +│ (API-nycklar, │ (foto → │ (claims, │ +│ billing, SLAs) │ kandidater) │ verifiering) │ +├──────────────────────┼────────────────┼────────────────────────┤ +│ Datafabrik │ Vision │ QUIXZOOM │ +│ (skördning, │ (embeddings, │ (bounties, │ +│ extraktion, │ OCR, │ fältverifiering, │ +│ entitetsupplösning)│ träning) │ individregister) │ +├──────────────────────┴────────────────┴────────────────────────┤ +│ Property Graph (Neo4j) │ Vektordatabas (FAISS) │ +│ 64 noder → 1M+ noder │ 0 → 1M+ embeddings │ +├─────────────────────────────────────────────────────────────────┤ +│ Ingest Pipeline → Master Data → Graf-export → API │ +└─────────────────────────────────────────────────────────────────┘ ``` +## Komponenter + +### Datafabrik (`datafabrik/`) +Automatiserad datainsamling och bearbetning. + +| Modul | Beskrivning | +|-------|-------------| +| `skordare/` | Webb-crawlers för tillverkare, upphandlingar, standarder | +| `extraktor/` | LLM-baserad schemastyrd extraktion (GPT-4o-mini) | +| `upplosare/` | Entitetsupplösning och deduplicering | +| `koer/` | Tre köer: schemalagd / kunddriven / fält | +| `agent_orkestrering.py` | Koordinerar 20+ parallella agenter | + +### Vision (`vision/`) +Bildigenkänning och träning. + +| Modul | Beskrivning | +|-------|-------------| +| `modeller/` | Identify-modell (ResNet50 + kontrastivt lärande) | +| `traening/` | Träningspipeline med NT-Xent loss | +| `embeddings/` | FAISS-vektordatabas + OCR-pipeline | +| `dataset/` | Bilddataset från Zoomer-foton och tillverkare | + +### API (`api/`) +FastAPI-baserad Identify API. + +| Endpoint | Beskrivning | +|----------|-------------| +| `POST /v0/identify` | Bild → rankade kandidater | +| `GET /v0/objects/{id}` | Komplett post | +| `GET /v0/search` | Fritextsök | +| `POST /v0/feedback` | Skapa bounty | + +### Infrastruktur (`infrastruktur/`) + +| Komponent | Teknik | +|-----------|--------| +| Graf-databas | Neo4j Enterprise | +| Vektordatabas | FAISS | +| Objektlagring | MinIO (S3-kompatibel) | +| Köer | Redis | +| Monitorering | Prometheus + Grafana | +| Orkestrering | AWS ECS Fargate | + ## Snabbstart ```bash -# Bygg & kör allt +# 1. Bygg och starta lokalt ./build.sh -# Endast API (förutsätter Neo4j) -docker compose up -d api +# 2. Produktion med Docker Compose +docker compose -f docker-compose.prod.yml up -d -# Endast data-export -python3 scripts/export_csv.py -python3 db/load_to_neo4j.py +# 3. Skala ut med Terraform +cd infrastruktur/terraform +terraform apply + +# 4. Kör agentarmén för Tier 1 +python3 datafabrik/agent_orkestrering.py --tier 1 --domäner TRP,ELN,VAT ``` -## API - -| Endpoint | Beskrivning | -|----------|-------------| -| `GET /health` | Status + antal poster/kanter | -| `POST /v0/identify` | Bild → rankade kandidater | -| `GET /v0/objects/{lvx_id}` | Komplett post | -| `GET /v0/objects/{lvx_id}/succession` | Ersättningskedja | -| `GET /v0/search?q=...` | Fritextsök | -| `GET /v0/standards/{b}/objects` | Standard → objekt | -| `POST /v0/feedback` | Fel/okänt → bounty | - -## Data - -- **64 noder** (37 klasser + 27 modeller) -- **176 kanter** (80 LVX-upplösta, 82 standard, 14 textnoder) -- **9 domäner**: TRP, ELN, VAT, TEL, BYG, PRK, JVG, HMN, FLG -- **Verifieringsnivåer**: obekräftad → källbelagd → fältverifierad → tillverkarbekräftad - -## Arkitektur +## Dataflöde ``` -Ingest Pipeline API Layer Graph DB -───────────────────────────────────────────────────────── -Kandidatposter → resolve/merge → Neo4j (propertygraf) - ↓ ↓ ↓ -Extraktion → Identify API → Query/Search - ↓ ↓ ↓ -Zoomer-bounties ← Feedback ← Succession +Skördare → Rådata → Extraktor → Kandidatposter → Upplösare → +Master v3 → Graf-export → Neo4j + FAISS → API → Kund + ↑_________________________________________________↓ + (feedback → bounty → ny skördning) ``` +## Mätetal + +| Mått | Nu | Mål (12 mån) | +|------|-----|--------------| +| Objektklasser | 37 | 500+ | +| Produktmodeller | 27 | 100 000+ | +| Grafkanter | 176 | 1 000 000+ | +| Identify konfidens | Mockad | >90% top-3 | +| Kostnad/faktum | N/A | <$0.001 | + ## Regler -1. **Fakta bär alltid källa** — inga påståenden utan proveniens -2. **Landvex-ID är permanent** — namn ändras, ID aldrig -3. **Konfidens exponeras alltid** — kunden ser aldrig osäkert som sanning -4. **Problem utan källa filtreras bort** — vid API-serialisering +1. **Enda sanningskälla:** `10-data/` + `lvx-objekt-schema.json` +2. **Aldrig mock i kundnära väg** +3. **Källa på varje påstående** +4. **ID:n mintas endast via registertjänsten** +5. **Verifieringstrappan är enkelriktad** ## Dokument - [Genome-strategi](40-strategi/landvex-genome-strategi.md) - [IOL-grundspec](40-strategi/iol-grundspecifikation.md) - [API-spec](30-produkt/lvx-pilot-api-spec.md) +- [Agentarmé](00-styrning/AGENTARME.md) + +## Team + +58 agenter · 11 skvadroner · 4 faser + +Litet kärnteam, stor agentflotta. **Skalning sker i beräkning, inte i huvuden.** diff --git a/projects/landvex/build.sh b/projects/landvex/build.sh index 9ac541ad2..79cdfa0c1 100755 --- a/projects/landvex/build.sh +++ b/projects/landvex/build.sh @@ -10,6 +10,8 @@ DATA_DIR="${PWD}/10-data" DB_DIR="${PWD}/db" API_DIR="${PWD}/api" SCRIPT_DIR="${PWD}/scripts" +DATAFABRIK_DIR="${PWD}/datafabrik" +VISION_DIR="${PWD}/vision" # ─── Steg 1: Validera indata ─────────────────────────────── echo "" @@ -17,10 +19,11 @@ echo "▶ Steg 1: Validerar indata..." python3 "${SCRIPT_DIR}/export_csv.py" python3 "${DB_DIR}/load_to_neo4j.py" -# ─── Steg 2: Bygg Docker-image ───────────────────────────── +# ─── Steg 2: Bygg Docker-images ──────────────────────────── echo "" -echo "▶ Steg 2: Bygger API-container..." +echo "▶ Steg 2: Bygger Docker-images..." docker compose build api +docker compose -f docker-compose.prod.yml build datafabrik vision-trainer # ─── Steg 3: Starta tjänster ─────────────────────────────── echo "" @@ -42,6 +45,22 @@ echo "▶ Steg 5: Snabbtest av endpoints..." curl -s 'http://localhost:8081/v0/search?q=belysning' | python3 -m json.tool | grep '"antal"' curl -s 'http://localhost:8081/v0/objects/LVX-TRP-0102' | python3 -m json.tool | grep '"lvx_id"' +# ─── Steg 6: Datafabrik (valfritt) ───────────────────────── +if [ "${1:-}" == "--med-datafabrik" ]; then + echo "" + echo "▶ Steg 6: Startar Datafabrik..." + docker compose -f docker-compose.prod.yml up -d datafabrik + echo " ✅ Datafabrik startad" +fi + +# ─── Steg 7: Vision (valfritt) ───────────────────────────── +if [ "${1:-}" == "--med-vision" ]; then + echo "" + echo "▶ Steg 7: Startar Vision-träning..." + docker compose -f docker-compose.prod.yml --profile training up -d vision-trainer + echo " ✅ Vision-tränare startad" +fi + echo "" echo "═══════════════════════════════════════════════════════════" echo " ✅ Landvex IOL är driftsatt!" @@ -49,11 +68,13 @@ echo "" echo " API: http://localhost:8081" echo " Health: http://localhost:8081/health" echo " Neo4j: bolt://localhost:7687" +echo " Grafana: http://localhost:3000" +echo " Prometheus: http://localhost:9090" echo "" -echo " Endpoints:" -echo " POST /v0/identify — Bild → kandidater" -echo " GET /v0/objects/{id} — Hämta objekt" -echo " GET /v0/search?q=... — Fritextsök" -echo " GET /v0/standards/{b}/objects — Standard → objekt" -echo " POST /v0/feedback — Skapa bounty" +echo " Kommandon:" +echo " ./build.sh --med-datafabrik # Inkludera datafabrik" +echo " ./build.sh --med-vision # Inkludera vision-träning" +echo "" +echo " Agentorkestrering:" +echo " python3 datafabrik/agent_orkestrering.py --tier 1" echo "═══════════════════════════════════════════════════════════" diff --git a/projects/landvex/datafabrik/Dockerfile b/projects/landvex/datafabrik/Dockerfile new file mode 100644 index 000000000..91c57aa46 --- /dev/null +++ b/projects/landvex/datafabrik/Dockerfile @@ -0,0 +1,31 @@ +FROM python:3.12-slim + +WORKDIR /app + +# Systemberoenden +RUN apt-get update && apt-get install -y \ + tesseract-ocr \ + tesseract-ocr-swe \ + tesseract-ocr-eng \ + libgl1-mesa-glx \ + libglib2.0-0 \ + && rm -rf /var/lib/apt/lists/* + +# Python-paket +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +# Kopiera kod +COPY skordare/ ./skordare/ +COPY extraktor/ ./extraktor/ +COPY koer/ ./koer/ +COPY upplosare/ ./upplosare/ + +# Miljövariabler +ENV PYTHONPATH=/app +ENV DATAFABRIK_KO_DIR=/data/koer +ENV DATAFABRIK_OUTPUT=/data/output + +VOLUME ["/data"] + +CMD ["python3", "-m", "koer.pipeline"] diff --git a/projects/landvex/datafabrik/agent_orkestrering.py b/projects/landvex/datafabrik/agent_orkestrering.py new file mode 100644 index 000000000..e0e824a43 --- /dev/null +++ b/projects/landvex/datafabrik/agent_orkestrering.py @@ -0,0 +1,241 @@ +#!/usr/bin/env python3 +""" +Landvex Agent Orkestrering +Koordinerar 50GB agentkraft för skalad datainsamling. +""" +import asyncio +import json +import subprocess +from datetime import datetime +from pathlib import Path +from typing import List, Dict, Optional +from dataclasses import dataclass, asdict +from enum import Enum + +class AgentStatus(Enum): + LEDIG = "ledig" + KOR = "kor" + KLAR = "klar" + FEL = "fel" + +@dataclass +class AgentUppgift: + uppgift_id: str + typ: str # "skorda", "extrahera", "upplos", "trana" + doman: str # TRP, ELN, VAT, etc. + tier: int # 1, 2, 3 + prioritet: int # 1-10 + parametrar: Dict + status: str = "pending" + resultat: Optional[Dict] = None + startad: Optional[str] = None + avslutad: Optional[str] = None + +class LandvexAgentOrkestrerare: + """Orkestrerar agenter för parallell datainsamling.""" + + def __init__( + self, + max_parallella: int = 20, + ko_dir: Path = Path("/data/koer"), + output_dir: Path = Path("/data/output"), + ): + self.max_parallella = max_parallella + self.ko_dir = ko_dir + self.output_dir = output_dir + self.agenter: Dict[str, Dict] = {} + self.ko: List[AgentUppgift] = [] + self.statistik = { + "skordade": 0, + "extraherade": 0, + "upplosta": 0, + "tränade": 0, + "fel": 0, + } + + def skapa_uppgifter_for_tier(self, tier: int, domäner: List[str]) -> List[AgentUppgift]: + """Skapa uppgifter för en specifik tier.""" + uppgifter = [] + + for doman in domäner: + # Skördningsuppgifter + uppgifter.append(AgentUppgift( + uppgift_id=f"skorda-{doman}-t{tier}-{datetime.utcnow().strftime('%Y%m%d%H%M%S')}", + typ="skorda", + doman=doman, + tier=tier, + prioritet=10 - tier, # Tier 1 = högst prioritet + parametrar={"kallor": "alla", "max_dokument": 1000}, + )) + + # Extraktionsuppgifter + uppgifter.append(AgentUppgift( + uppgift_id=f"extrahera-{doman}-t{tier}-{datetime.utcnow().strftime('%Y%m%d%H%M%S')}", + typ="extrahera", + doman=doman, + tier=tier, + prioritet=9 - tier, + parametrar={"batch_storlek": 100}, + )) + + # Entitetsupplösning + uppgifter.append(AgentUppgift( + uppgift_id=f"upplos-{doman}-t{tier}-{datetime.utcnow().strftime('%Y%m%d%H%M%S')}", + typ="upplos", + doman=doman, + tier=tier, + prioritet=8 - tier, + parametrar={"tröskel": 0.85}, + )) + + return uppgifter + + async def kör_agent(self, uppgift: AgentUppgift) -> Dict: + """Kör en agent för en specifik uppgift.""" + uppgift.startad = datetime.utcnow().isoformat() + uppgift.status = "kor" + + try: + if uppgift.typ == "skorda": + resultat = await self._kör_skordare(uppgift) + elif uppgift.typ == "extrahera": + resultat = await self._kör_extraktor(uppgift) + elif uppgift.typ == "upplos": + resultat = await self._kör_upplosare(uppgift) + elif uppgift.typ == "trana": + resultat = await self._kör_tranare(uppgift) + else: + raise ValueError(f"Okänd uppgiftstyp: {uppgift.typ}") + + uppgift.status = "klar" + uppgift.resultat = resultat + self.statistik[f"{uppgift.typ}ade"] += 1 + + except Exception as e: + uppgift.status = "fel" + uppgift.resultat = {"fel": str(e)} + self.statistik["fel"] += 1 + + uppgift.avslutad = datetime.utcnow().isoformat() + return uppgift.resultat + + async def _kör_skordare(self, uppgift: AgentUppgift) -> Dict: + """Kör skördare för en domän.""" + # Anropa crawler.py + process = await asyncio.create_subprocess_exec( + "python3", "skordare/crawler.py", + "--doman", uppgift.doman, + "--output", str(self.output_dir / "skordat"), + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + stdout, stderr = await process.communicate() + + return { + "returncode": process.returncode, + "stdout": stdout.decode()[:1000], + "stderr": stderr.decode()[:1000], + } + + async def _kör_extraktor(self, uppgift: AgentUppgift) -> Dict: + """Kör extraktor för skördade dokument.""" + process = await asyncio.create_subprocess_exec( + "python3", "extraktor/extraktor.py", + "--input", str(self.output_dir / "skordat"), + "--output", str(self.output_dir / "extraherat"), + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + stdout, stderr = await process.communicate() + + return { + "returncode": process.returncode, + "stdout": stdout.decode()[:1000], + "stderr": stderr.decode()[:1000], + } + + async def _kör_upplosare(self, uppgift: AgentUppgift) -> Dict: + """Kör entitetsupplösare.""" + process = await asyncio.create_subprocess_exec( + "python3", "upplosare/entitetsupplosare.py", + "--input", str(self.output_dir / "extraherat"), + "--output", str(self.output_dir / "upplost"), + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + stdout, stderr = await process.communicate() + + return { + "returncode": process.returncode, + "stdout": stdout.decode()[:1000], + "stderr": stderr.decode()[:1000], + } + + async def _kör_tranare(self, uppgift: AgentUppgift) -> Dict: + """Kör vision-träning.""" + process = await asyncio.create_subprocess_exec( + "python3", "../vision/traening/trainer.py", + "--dataset", str(self.output_dir / "dataset"), + "--output", str(self.output_dir / "modeller"), + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + stdout, stderr = await process.communicate() + + return { + "returncode": process.returncode, + "stdout": stdout.decode()[:1000], + "stderr": stderr.decode()[:1000], + } + + async def kör_alla(self, uppgifter: List[AgentUppgift]): + """Kör alla uppgifter med begränsad parallellism.""" + self.ko = sorted(uppgifter, key=lambda x: x.prioritet, reverse=True) + + print(f"🚀 Startar {len(self.ko)} uppgifter (max {self.max_parallella} parallella)") + + # Kör i batcher + while self.ko: + batch = self.ko[:self.max_parallella] + self.ko = self.ko[self.max_parallella:] + + print(f"\n📦 Kör batch: {len(batch)} uppgifter") + tasks = [self.kör_agent(u) for u in batch] + resultat = await asyncio.gather(*tasks, return_exceptions=True) + + for uppgift, res in zip(batch, resultat): + status = "✅" if uppgift.status == "klar" else "❌" + print(f" {status} {uppgift.typ} {uppgift.doman} T{uppgift.tier}: {uppgift.status}") + + print(f"\n📊 Statistik: {self.statistik}") + + def spara_rapport(self): + """Spara körningsrapport.""" + rapport = { + "tidstampel": datetime.utcnow().isoformat(), + "statistik": self.statistik, + "uppgifter": [asdict(u) for u in self.ko], + } + + fil_path = self.output_dir / f"rapport_{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}.json" + with open(fil_path, 'w') as f: + json.dump(rapport, f, indent=2) + + print(f"\n📝 Rapport sparad: {fil_path}") + +async def main(): + """Demo: orkestrera agenter för Tier 1.""" + orkestrerare = LandvexAgentOrkestrerare(max_parallella=5) + + # Skapa uppgifter för Tier 1 (alla 9 domäner) + domäner = ["TRP", "ELN", "VAT", "TEL", "BYG", "PRK", "JVG", "HMN", "FLG"] + uppgifter = orkestrerare.skapa_uppgifter_for_tier(tier=1, domäner=domäner) + + print(f"📋 Skapade {len(uppgifter)} uppgifter för Tier 1") + + # Kör + await orkestrerare.kör_alla(uppgifter) + orkestrerare.spara_rapport() + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/projects/landvex/datafabrik/extraktor/extraktor.py b/projects/landvex/datafabrik/extraktor/extraktor.py new file mode 100644 index 000000000..65acead3c --- /dev/null +++ b/projects/landvex/datafabrik/extraktor/extraktor.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +""" +Landvex Extraktor — LLM-baserad schemastyrd extraktion +Dokument in, strukturerade fakta ut. +""" +import json +import os +from pathlib import Path +from typing import List, Dict, Optional +from datetime import datetime + +from openai import AsyncOpenAI + +# Schema för extraktion (från lvx-objekt-schema.json) +EXTRAKTIONS_SCHEMA = { + "objektklass": { + "required": ["lvx_id", "posttyp", "namn", "doman", "proveniens"], + "fields": [ + "lvx_id", "posttyp", "slug", "namn", "doman", "parent", "tier", + "identitet", "teknik", "historik", "geografi", "standarder", + "ai", "problem", "dokumentation", "relationer", "proveniens" + ] + }, + "produktmodell": { + "required": ["lvx_id", "posttyp", "namn", "doman", "parent", "proveniens"], + "fields": [ + "lvx_id", "posttyp", "slug", "namn", "doman", "parent", "tier", + "identitet", "teknik", "historik", "geografi", "standarder", + "ai", "problem", "dokumentation", "relationer", "proveniens" + ] + } +} + +EXTRAKTIONS_PROMPT = """Du är Landvex Extraktor — en noggrann forskarassistent specialiserad på infrastrukturobjekt. + +DIN UPPGIFT: Analysera följande dokument och extrahera strukturerade fakta om infrastrukturobjekt. + +REGLER: +1. Extrahera ENDAST fakta som finns i dokumentet — uppfinn inget +2. Varje faktum måste ha en källa (URL eller dokumentreferens) +3. Använd Landvex-schemat strikt +4. Om något är osäkert, markera med låg konfidens +5. Problem/Kända fel kräver alltid källa + +OUTPUT: JSON enligt följande struktur: +{ + "kandidater": [ + { + "atgard": "ny" | "uppdatera", + "posttyp": "objektklass" | "produktmodell", + "post": { ... }, + "patch": { ... }, + "verifieringsniva_forslag": "obekraftad" | "kallbelagd" | "faltverifierad" | "tillverkarbekraftad", + "kallor": [ + {"titel": "...", "url": "...", "hamtad": "2026-07-05"} + ] + } + ] +} + +DOKUMENT: +{content} +""" + +class LandvexExtraktor: + def __init__(self, api_key: Optional[str] = None): + self.client = AsyncOpenAI(api_key=api_key or os.getenv("OPENAI_API_KEY")) + self.model = "gpt-4o-mini" # Kostnadseffektiv för volym + + async def extrahera(self, dokument: str, url: str, doman: str) -> dict: + """Extrahera strukturerade fakta från ett dokument.""" + + prompt = EXTRAKTIONS_PROMPT.format(content=dokument[:15000]) # Begränsa längd + + try: + response = await self.client.chat.completions.create( + model=self.model, + messages=[ + {"role": "system", "content": "Du är Landvex Extraktor. Extrahera strukturerade fakta om infrastrukturobjekt."}, + {"role": "user", "content": prompt} + ], + response_format={"type": "json_object"}, + temperature=0.1, # Låg för determinism + ) + + resultat = json.loads(response.choices[0].message.content) + + # Lägg till proveniens + for kand in resultat.get("kandidater", []): + kand.setdefault("kallor", []).append({ + "titel": f"Extraherat från {url}", + "url": url, + "hamtad": datetime.utcnow().isoformat() + }) + + return resultat + + except Exception as e: + print(f" ⚠️ Extraktionsfel: {e}") + return {"kandidater": [], "fel": str(e)} + + async def extrahera_batch(self, dokument: List[dict]) -> List[dict]: + """Extrahera från flera dokument parallellt.""" + import asyncio + + uppgifter = [ + self.extrahera(d["html"], d["url"], d.get("doman", "TRP")) + for d in dokument + ] + + resultat = await asyncio.gather(*uppgifter, return_exceptions=True) + return [r for r in resultat if not isinstance(r, Exception)] + +async def main(): + """Demo: extrahera från testdokument.""" + extraktor = LandvexExtraktor() + + test_doc = """ + ABB Kabeldon CDC-LT är ett lågspänningskabelskåp för utomhusbruk. + Material: Varmförzinkad stålplåt, IP34D enligt SS-EN 60529. + Standard: SS-EN 61439-5 (provnings- och konstruktionskrav). + Tillverkare: ABB Kabeldon, Alingsås. + Lås: Trekantslås standard, cylinderlås som tillval. + """ + + resultat = await extraktor.extrahera(test_doc, "https://example.com/test", "ELN") + print(json.dumps(resultat, ensure_ascii=False, indent=2)) + +if __name__ == "__main__": + import asyncio + asyncio.run(main()) diff --git a/projects/landvex/datafabrik/koer/pipeline.py b/projects/landvex/datafabrik/koer/pipeline.py new file mode 100644 index 000000000..4a5a74bda --- /dev/null +++ b/projects/landvex/datafabrik/koer/pipeline.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +""" +Landvex Datafabrik — Pipeline-orkestrering +Tre köer: schemalagd / kunddriven / fält +""" +import asyncio +import json +import hashlib +from datetime import datetime +from pathlib import Path +from typing import List, Dict, Optional +from enum import Enum + +class KoTyp(Enum): + SCHEMALAGD = "schemalagd" # Periodisk skördning + KUNDDRIVEN = "kunddriven" # Feedback → bounty → research + FALT = "falt" # Zoomer-foton → verifiering + +class PipelineKo: + def __init__(self, ko_dir: Path, ko_typ: KoTyp): + self.ko_dir = ko_dir / ko_typ.value + self.ko_typ = ko_typ + self.ko_dir.mkdir(parents=True, exist_ok=True) + + # Underkataloger + (self.ko_dir / "pending").mkdir(exist_ok=True) + (self.ko_dir / "processing").mkdir(exist_ok=True) + (self.ko_dir / "completed").mkdir(exist_ok=True) + (self.ko_dir / "failed").mkdir(exist_ok=True) + + def lagg_till(self, uppgift: dict) -> str: + """Lägg till uppgift i kön.""" + uppgift_id = hashlib.sha256( + json.dumps(uppgift, sort_keys=True).encode() + ).hexdigest()[:16] + + uppgift["uppgift_id"] = uppgift_id + uppgift["skapad"] = datetime.utcnow().isoformat() + uppgift["status"] = "pending" + + fil_path = self.ko_dir / "pending" / f"{uppgift_id}.json" + with open(fil_path, 'w', encoding='utf-8') as f: + json.dump(uppgift, f, ensure_ascii=False, indent=2) + + return uppgift_id + + def hamta_nasta(self) -> Optional[dict]: + """Hämta nästa uppgift från kön.""" + pending = sorted(self.ko_dir / "pending" .glob("*.json")) + if not pending: + return None + + fil_path = pending[0] + with open(fil_path, 'r', encoding='utf-8') as f: + uppgift = json.load(f) + + # Flytta till processing + ny_path = self.ko_dir / "processing" / fil_path.name + fil_path.rename(ny_path) + uppgift["status"] = "processing" + uppgift["startad"] = datetime.utcnow().isoformat() + + with open(ny_path, 'w', encoding='utf-8') as f: + json.dump(uppgift, f, ensure_ascii=False, indent=2) + + return uppgift + + def markera_klar(self, uppgift_id: str, resultat: dict): + """Markera uppgift som klar.""" + processing_path = self.ko_dir / "processing" / f"{uppgift_id}.json" + if not processing_path.exists(): + return + + with open(processing_path, 'r', encoding='utf-8') as f: + uppgift = json.load(f) + + uppgift["status"] = "completed" + uppgift["avslutad"] = datetime.utcnow().isoformat() + uppgift["resultat"] = resultat + + klar_path = self.ko_dir / "completed" / f"{uppgift_id}.json" + processing_path.rename(klar_path) + + with open(klar_path, 'w', encoding='utf-8') as f: + json.dump(uppgift, f, ensure_ascii=False, indent=2) + + def markera_misslyckad(self, uppgift_id: str, fel: str): + """Markera uppgift som misslyckad.""" + processing_path = self.ko_dir / "processing" / f"{uppgift_id}.json" + if not processing_path.exists(): + return + + with open(processing_path, 'r', encoding='utf-8') as f: + uppgift = json.load(f) + + uppgift["status"] = "failed" + uppgift["avslutad"] = datetime.utcnow().isoformat() + uppgift["fel"] = fel + + fail_path = self.ko_dir / "failed" / f"{uppgift_id}.json" + processing_path.rename(fail_path) + + with open(fail_path, 'w', encoding='utf-8') as f: + json.dump(uppgift, f, ensure_ascii=False, indent=2) + + def statistik(self) -> dict: + """Hämta kö-statistik.""" + return { + "typ": self.ko_typ.value, + "pending": len(list((self.ko_dir / "pending").glob("*.json"))), + "processing": len(list((self.ko_dir / "processing").glob("*.json"))), + "completed": len(list((self.ko_dir / "completed").glob("*.json"))), + "failed": len(list((self.ko_dir / "failed").glob("*.json"))), + } + +class DatafabrikPipeline: + def __init__(self, base_dir: Path): + self.base_dir = base_dir + self.koer = { + KoTyp.SCHEMALAGD: PipelineKo(base_dir, KoTyp.SCHEMALAGD), + KoTyp.KUNDDRIVEN: PipelineKo(base_dir, KoTyp.KUNDDRIVEN), + KoTyp.FALT: PipelineKo(base_dir, KoTyp.FALT), + } + + def lagg_till_skordning(self, doman: str, kallor: List[str], prioritet: int = 5): + """Schemalägg en skördning.""" + return self.koer[KoTyp.SCHEMALAGD].lagg_till({ + "typ": "skordning", + "doman": doman, + "kallor": kallor, + "prioritet": prioritet, + }) + + def lagg_till_bounty(self, lvx_id: str, position: str, beskrivning: str): + """Kunddriven bounty → research-uppgift.""" + return self.koer[KoTyp.KUNDDRIVEN].lagg_till({ + "typ": "bounty_research", + "lvx_id": lvx_id, + "position": position, + "beskrivning": beskrivning, + }) + + def lagg_till_faltverifiering(self, foto_id: str, lvx_id: str, zoomer_id: str): + """Zoomer-foto → verifiering.""" + return self.koer[KoTyp.FALT].lagg_till({ + "typ": "faltverifiering", + "foto_id": foto_id, + "lvx_id": lvx_id, + "zoomer_id": zoomer_id, + }) + + def statistik(self) -> dict: + """Hämta statistik för alla köer.""" + return {k.value: v.statistik() for k, v in self.koer.items()} + +def main(): + """Demo: skapa pipeline och lägg till uppgifter.""" + pipeline = DatafabrikPipeline(Path("/tmp/landvex-pipeline")) + + # Schemalagd skördning + id1 = pipeline.lagg_till_skordning("TRP", ["https://example.com/vagbelysning"], 1) + print(f"Schemalagd: {id1}") + + # Kunddriven bounty + id2 = pipeline.lagg_till_bounty("LVX-TRP-0102", "Storgatan 12, Stockholm", "Okänd armaturmodell") + print(f"Bounty: {id2}") + + # Fältverifiering + id3 = pipeline.lagg_till_faltverifiering("IMG-123", "LVX-TRP-0102", "zoomer-42") + print(f"Fält: {id3}") + + print("\nStatistik:") + print(json.dumps(pipeline.statistik(), indent=2)) + +if __name__ == "__main__": + main() diff --git a/projects/landvex/datafabrik/requirements.txt b/projects/landvex/datafabrik/requirements.txt new file mode 100644 index 000000000..a299d60d6 --- /dev/null +++ b/projects/landvex/datafabrik/requirements.txt @@ -0,0 +1,17 @@ +# Datafabrik dependencies +aiohttp>=3.9 +beautifulsoup4>=4.12 +openai>=1.30 +neo4j>=5.19 +pydantic>=2.6 +python-dotenv>=1.0 +requests>=2.31 +pillow>=10.0 +torch>=2.2 +torchvision>=0.17 +transformers>=4.40 +sentence-transformers>=2.7 +scikit-learn>=1.4 +numpy>=1.26 +pandas>=2.2 +tqdm>=4.66 diff --git a/projects/landvex/datafabrik/skordare/crawler.py b/projects/landvex/datafabrik/skordare/crawler.py new file mode 100644 index 000000000..6b68ac0b7 --- /dev/null +++ b/projects/landvex/datafabrik/skordare/crawler.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +""" +Landvex Skördare — Webb-crawler för tillverkardata +Respekterar robots.txt, crawl-fördröjning, vitlista. +""" +import asyncio +import json +import hashlib +import time +from datetime import datetime +from pathlib import Path +from urllib.parse import urljoin, urlparse +from typing import Set, List, Dict, Optional + +import aiohttp +from bs4 import BeautifulSoup + +from kallvitlista import validera_url, hamta_kallor_for_doman + +class LandvexCrawler: + def __init__( + self, + output_dir: Path, + delay_seconds: float = 1.0, + max_pages_per_domain: int = 100, + max_depth: int = 3 + ): + self.output_dir = output_dir + self.delay = delay_seconds + self.max_pages = max_pages_per_domain + self.max_depth = max_depth + self.visited: Set[str] = set() + self.session: Optional[aiohttp.ClientSession] = None + + async def __aenter__(self): + self.session = aiohttp.ClientSession( + headers={ + "User-Agent": "LandvexBot/1.0 (Research; https://landvex.io/bot)" + } + ) + return self + + async def __aexit__(self, *args): + if self.session: + await self.session.close() + + async def hamta(self, url: str) -> Optional[str]: + """Hämta URL med felhantering.""" + if not validera_url(url): + print(f" ⚠️ URL ej på vitlistan: {url}") + return None + + try: + async with self.session.get(url, timeout=30) as resp: + if resp.status == 200: + return await resp.text() + print(f" ⚠️ HTTP {resp.status}: {url}") + return None + except Exception as e: + print(f" ⚠️ Fel: {e}") + return None + + def extrahera_lankar(self, html: str, base_url: str) -> List[str]: + """Extrahera alla länkar från HTML.""" + soup = BeautifulSoup(html, 'html.parser') + lankar = [] + for a in soup.find_all('a', href=True): + href = urljoin(base_url, a['href']) + if validera_url(href): + lankar.append(href) + return lankar + + def spara_dokument(self, url: str, html: str, metadata: dict): + """Spara skördat dokument med metadata.""" + doc_id = hashlib.sha256(url.encode()).hexdigest()[:16] + timestamp = datetime.utcnow().isoformat() + + doc = { + "doc_id": doc_id, + "url": url, + "hamtat": timestamp, + "metadata": metadata, + "html_hash": hashlib.sha256(html.encode()).hexdigest()[:16], + "html_length": len(html), + } + + # Spara metadata + meta_path = self.output_dir / "metadata" / f"{doc_id}.json" + meta_path.parent.mkdir(parents=True, exist_ok=True) + with open(meta_path, 'w', encoding='utf-8') as f: + json.dump(doc, f, ensure_ascii=False, indent=2) + + # Spara rå HTML (för extraktion) + html_path = self.output_dir / "raw" / f"{doc_id}.html" + html_path.parent.mkdir(parents=True, exist_ok=True) + with open(html_path, 'w', encoding='utf-8') as f: + f.write(html) + + return doc_id + + async def crawla_doman( + self, + start_url: str, + doman: str, + djup: int = 0 + ) -> List[dict]: + """Crawla en start-URL och följ länkar.""" + if djup > self.max_depth: + return [] + + if start_url in self.visited: + return [] + + self.visited.add(start_url) + + print(f" 🔍 [{djup}] {start_url}") + html = await self.hamta(start_url) + if not html: + return [] + + # Spara dokumentet + metadata = { + "doman": doman, + "crawl_djup": djup, + "kalla": urlparse(start_url).netloc, + } + doc_id = self.spara_dokument(start_url, html, metadata) + + resultat = [{"doc_id": doc_id, "url": start_url}] + + # Följ länkar om vi inte nått max + if len(self.visited) < self.max_pages: + lankar = self.extrahera_lankar(html, start_url) + for lank in lankar[:5]: # Begränsa per sida + await asyncio.sleep(self.delay) + under = await self.crawla_doman(lank, doman, djup + 1) + resultat.extend(under) + + return resultat + + async def skorda_doman(self, doman: str) -> dict: + """Skörda alla källor för en domän.""" + print(f"\n🌾 Skördar domän: {doman}") + + kallor = hamta_kallor_for_doman(doman) + alla_dokument = [] + + for kategori, data in kallor.items(): + print(f" 📂 {kategori}: {data['beskrivning']}") + + urls = [] + if "källor" in data: + sources = data["källor"] + if isinstance(sources, dict): + for land, land_urls in sources.items(): + urls.extend(land_urls) + else: + urls.extend(sources) + if "exempel" in data: + urls.extend(data["exempel"]) + + for url in urls[:3]: # Begränsa initialt + try: + docs = await self.crawla_doman(url, doman) + alla_dokument.extend(docs) + except Exception as e: + print(f" ⚠️ Fel vid crawling: {e}") + + rapport = { + "doman": doman, + "skordad": datetime.utcnow().isoformat(), + "dokument": len(alla_dokument), + "unika_kallor": len(set(d["url"] for d in alla_dokument)), + } + + # Spara rapport + rapport_path = self.output_dir / "rapporter" / f"{doman}_{datetime.utcnow().strftime('%Y%m%d')}.json" + rapport_path.parent.mkdir(parents=True, exist_ok=True) + with open(rapport_path, 'w', encoding='utf-8') as f: + json.dump(rapport, f, ensure_ascii=False, indent=2) + + print(f" ✅ {rapport['dokument']} dokument skördade") + return rapport + +async def main(): + """Demo: skörda TRP-domanen.""" + output = Path("/tmp/landvex-skord") + output.mkdir(exist_ok=True) + + async with LandvexCrawler(output) as crawler: + resultat = await crawler.skorda_doman("TRP") + print(f"\n📊 Resultat: {resultat}") + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/projects/landvex/datafabrik/skordare/kallvitlista.py b/projects/landvex/datafabrik/skordare/kallvitlista.py new file mode 100644 index 000000000..799b58e18 --- /dev/null +++ b/projects/landvex/datafabrik/skordare/kallvitlista.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +""" +Landvex Skördare — Källvitlista v1 +Godkända källor för automatisk skördning per domän. +""" + +KALLVITLISTA = { + "tillverkare": { + "beskrivning": "Tillverkarkataloger, datablad, manualer", + "exempel": [ + "https://www.abb.com/products", + "https://www.signify.com/global/products", + "https://www.thornlighting.com/en-gb/products", + "https://www.schreder.com/en/products", + "https://www.hewig.de/produkte", + "https://www.lighting.philips.com/", + "https://www.cree.com/led-components", + "https://www.acuitybrands.com/products", + "https://www.hubbell.com/hubbell-lighting", + "https://www.lsi-industries.com/products", + ], + "regler": [ + "Endast publika produktsidor, ej inloggning", + "Extrahera fakta, kopiera aldrig hela dokument", + "Respektera robots.txt och crawl-fördröjning", + ] + }, + + "upphandling": { + "beskrivning": "Offentliga upphandlingar — guld för installerad-bas-data", + "källor": { + "EU": ["https://ted.europa.eu/", "https://ec.europa.eu/growth/single-market/public-procurement"], + "Sverige": ["https://www.mercell.com/", "https://www.opic.com/", "https://www.upphandling24.se/"], + "Norge": ["https://doffin.no/", "https://www.eu-supply.com/"], + "Danmark": ["https://www.ethics.dk/", "https://www.udbudsportalen.dk/"], + "Finland": ["https://www.hankintailmoitukset.fi/", "https://www.tarjouspalvelu.fi/"], + "Tyskland": ["https://www.vergabe24.de/", "https://www.bund.de/"], + "USA": ["https://sam.gov/", "https://www.usaspending.gov/"], + "UK": ["https://www.contractsfinder.service.gov.uk/", "https://www.find-tender.service.gov.uk/"], + }, + "regler": [ + "Extrahera: produkt, tillverkare, plats, tidpunkt, volym", + "Lagra referens, aldrig hela dokument", + "Offentliga upphandlingar = public domain fakta", + ] + }, + + "standarder": { + "beskrivning": "Standardiseringsorgan — beteckningar och metadata", + "källor": { + "IEC": ["https://webstore.iec.ch/"], + "CENELEC": ["https://www.cenelec.eu/"], + "ISO": ["https://www.iso.org/standards.html"], + "EN": ["https://standards.cen.eu/"], + "ANSI": ["https://webstore.ansi.org/"], + "ASTM": ["https://www.astm.org/standards/"], + "IEEE": ["https://standards.ieee.org/"], + "BSI": ["https://shop.bsigroup.com/"], + "DIN": ["https://www.din.de/"], + "SS": ["https://www.sis.se/"], + }, + "regler": [ + "Endast beteckning, titel, tillämpningsområde", + "ALDRIG standardtext — köpta standarder är skyddade", + "Referera med URL till standardens sida", + ] + }, + + "patent": { + "beskrivning": "Patentdatabaser — produkthistorik och generationsskiften", + "källor": [ + "https://patents.google.com/", + "https://www.epo.org/searching-for-patents.html", + "https://www.uspto.gov/patents/search", + "https://www.wipo.int/patents/en/", + ], + "regler": [ + "Fria att referera och analysera", + "Extrahera: uppfinnare, tillverkare, datum, teknik", + ] + }, + + "myndigheter": { + "beskrivning": "Myndighetsdokument och typgodkännanden", + "källor": { + "Sverige": ["https://www.transportstyrelsen.se/", "https://www.boverket.se/"], + "EU": ["https://eur-lex.europa.eu/", "https://ec.europa.eu/growth/single-market/european-standards/harmonised-standards"], + "USA": ["https://www.fhwa.dot.gov/", "https://www.osha.gov/"], + }, + "regler": [ + "Amerikanska federala dokument ofta public domain", + "Extrahera fakta, länka källan", + ] + }, + + "gis": { + "beskrivning": "Öppna GIS-data och anläggningsregister", + "källor": [ + "https://www.openstreetmap.org/ (ODbL — separat spår)", + "https://www.geodata.se/", + "https://www.lantmateriet.se/", + "https://www.usgs.gov/", + ], + "regler": [ + "OSM: ODbL kräver attribution och delningsvillkor", + "Håll i separat spår tills licensstrategi beslutad", + ] + }, + + "forum": { + "beskrivning": "Branschforum och installatörscommunities", + "exempel": [ + "https://www.elektrikerna.se/", + "https://www.byggnads.se/", + "https://www.reddit.com/r/electricians/", + "https://www.reddit.com/r/civilengineering/", + ], + "regler": [ + "Endast verifierade fältobservationer", + "Källhänvisning obligatorisk", + "AI-moderering för kvalitet", + ] + } +} + +def validera_url(url: str, kategori: str = None) -> bool: + """Kontrollera om URL finns på vitlistan.""" + for kat, data in KALLVITLISTA.items(): + if kategori and kat != kategori: + continue + sources = data.get("källor", {}) + if isinstance(sources, dict): + for land, urls in sources.items(): + if any(url.startswith(u) for u in urls): + return True + elif isinstance(sources, list): + if any(url.startswith(u) for u in sources): + return True + for ex in data.get("exempel", []): + if url.startswith(ex): + return True + return False + +def hamta_kallor_for_doman(doman: str) -> dict: + """Hämta relevanta källor för en domän.""" + mapping = { + "TRP": ["tillverkare", "upphandling", "standarder", "myndigheter", "gis"], + "ELN": ["tillverkare", "upphandling", "standarder", "myndigheter"], + "VAT": ["tillverkare", "upphandling", "standarder", "myndigheter", "gis"], + "TEL": ["tillverkare", "upphandling", "standarder", "myndigheter"], + "BYG": ["tillverkare", "upphandling", "standarder", "myndigheter"], + "PRK": ["tillverkare", "upphandling", "standarder", "myndigheter"], + "JVG": ["tillverkare", "upphandling", "standarder", "myndigheter"], + "HMN": ["tillverkare", "upphandling", "standarder", "myndigheter"], + "FLG": ["tillverkare", "upphandling", "standarder", "myndigheter"], + } + kategorier = mapping.get(doman, []) + return {k: KALLVITLISTA[k] for k in kategorier if k in KALLVITLISTA} + +if __name__ == "__main__": + print("Landvex Källvitlista") + print("=" * 50) + for kat, data in KALLVITLISTA.items(): + print(f"\n{kat.upper()}: {data['beskrivning']}") + print(f" Regler: {len(data['regler'])}") diff --git a/projects/landvex/datafabrik/skordare/upphandlings_skordare.py b/projects/landvex/datafabrik/skordare/upphandlings_skordare.py new file mode 100644 index 000000000..d8b595e84 --- /dev/null +++ b/projects/landvex/datafabrik/skordare/upphandlings_skordare.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +""" +Landvex Skördare — Offentliga upphandlingar +Guld för installerad-bas-data: produkt, tillverkare, plats, tidpunkt, volym. +""" +import asyncio +import json +from datetime import datetime +from pathlib import Path +from typing import List, Dict, Optional +from dataclasses import dataclass + +import aiohttp +from bs4 import BeautifulSoup + +@dataclass +class UpphandlingsPost: + dokument_id: str + kalla: str # URL till upphandlingen + land: str + myndighet: str + produkt: str # Vad som upphandlades + tillverkare: str # Angiven eller vinnande tillverkare + plats: str # Geografisk plats + tidpunkt: str # Upphandlingsdatum + volym: Optional[str] # Antal, värde, etc. + konfidens: float + +class UpphandlingsSkordare: + """Skördar offentliga upphandlingar för Landvex.""" + + KALLOR = { + "Sverige": [ + "https://www.mercell.com/", + "https://www.opic.com/", + ], + "EU": [ + "https://ted.europa.eu/", + ], + "Norge": [ + "https://doffin.no/", + ], + "Danmark": [ + "https://www.udbudsportalen.dk/", + ], + "Tyskland": [ + "https://www.vergabe24.de/", + ], + "USA": [ + "https://sam.gov/", + "https://www.usaspending.gov/", + ], + } + + def __init__(self, output_dir: Path): + self.output_dir = output_dir + self.output_dir.mkdir(parents=True, exist_ok=True) + self.session: Optional[aiohttp.ClientSession] = None + + async def __aenter__(self): + self.session = aiohttp.ClientSession( + headers={ + "User-Agent": "LandvexBot/1.0 (Research; https://landvex.io/bot)" + } + ) + return self + + async def __aexit__(self, *args): + if self.session: + await self.session.close() + + async def hamta_sida(self, url: str) -> Optional[str]: + """Hämta HTML från URL.""" + try: + async with self.session.get(url, timeout=30) as resp: + if resp.status == 200: + return await resp.text() + print(f" ⚠️ HTTP {resp.status}: {url}") + return None + except Exception as e: + print(f" ⚠️ Fel: {e}") + return None + + def extrahera_upphandlingar(self, html: str, kalla: str, land: str) -> List[UpphandlingsPost]: + """Extrahera upphandlingsposter från HTML.""" + soup = BeautifulSoup(html, 'html.parser') + poster = [] + + # Hitta upphandlingslistningar (anpassas per källa) + for item in soup.find_all(['tr', 'div', 'article'], class_=lambda x: x and ('tender' in x.lower() or 'contract' in x.lower() or 'upphandling' in x.lower())): + try: + # Extrahera fält + produkt = self._extrahera_text(item, ['title', 'description', 'objekt']) + myndighet = self._extrahera_text(item, ['authority', 'buyer', 'upphandlande']) + plats = self._extrahera_text(item, ['place', 'location', 'ort']) + datum = self._extrahera_text(item, ['date', 'deadline', 'publicerad']) + + if produkt: # Minst produkt måste finnas + post = UpphandlingsPost( + dokument_id=self._generera_id(produkt + datum), + kalla=kalla, + land=land, + myndighet=myndighet or "Okänd", + produkt=produkt, + tillverkare="", # Kräver djupanalys + plats=plats or "", + tidpunkt=datum or "", + volym=None, + konfidens=0.7 if myndighet else 0.5, + ) + poster.append(post) + except Exception as e: + continue + + return poster + + def _extrahera_text(self, element, klasser: List[str]) -> Optional[str]: + """Hjälp: extrahera text från element med matchande klass.""" + for klass in klasser: + found = element.find(class_=lambda x: x and klass in x.lower()) + if found: + return found.get_text(strip=True) + return None + + def _generera_id(self, text: str) -> str: + """Generera unikt ID från text.""" + import hashlib + return hashlib.sha256(text.encode()).hexdigest()[:16] + + def spara_poster(self, poster: List[UpphandlingsPost], land: str): + """Spara upphandlingsposter.""" + timestamp = datetime.utcnow().strftime('%Y%m%d_%H%M%S') + fil_path = self.output_dir / f"upphandlingar_{land}_{timestamp}.json" + + data = [ + { + "dokument_id": p.dokument_id, + "kalla": p.kalla, + "land": p.land, + "myndighet": p.myndighet, + "produkt": p.produkt, + "tillverkare": p.tillverkare, + "plats": p.plats, + "tidpunkt": p.tidpunkt, + "volym": p.volym, + "konfidens": p.konfidens, + "skordad": datetime.utcnow().isoformat(), + } + for p in poster + ] + + with open(fil_path, 'w', encoding='utf-8') as f: + json.dump(data, f, ensure_ascii=False, indent=2) + + print(f" 💾 {len(poster)} poster sparade: {fil_path}") + + async def skorda_land(self, land: str, max_sidor: int = 5) -> dict: + """Skörda upphandlingar för ett land.""" + print(f"\n🌾 Skördar upphandlingar: {land}") + + kallor = self.KALLOR.get(land, []) + alla_poster = [] + + for kalla in kallor[:2]: # Begränsa initialt + print(f" 🔍 {kalla}") + html = await self.hamta_sida(kalla) + if html: + poster = self.extrahera_upphandlingar(html, kalla, land) + alla_poster.extend(poster) + print(f" ✅ {len(poster)} poster") + + # Spara + if alla_poster: + self.spara_poster(alla_poster, land) + + return { + "land": land, + "poster": len(alla_poster), + "kallor": len(kallor), + } + +async def main(): + """Demo: skörda svenska upphandlingar.""" + output = Path("/tmp/landvex-upphandlingar") + + async with UpphandlingsSkordare(output) as skordare: + resultat = await skordare.skorda_land("Sverige", max_sidor=2) + print(f"\n📊 Resultat: {resultat}") + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/projects/landvex/datafabrik/upplosare/entitetsupplosare.py b/projects/landvex/datafabrik/upplosare/entitetsupplosare.py new file mode 100644 index 000000000..5455bdae2 --- /dev/null +++ b/projects/landvex/datafabrik/upplosare/entitetsupplosare.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +""" +Landvex Upplösare — Entitetsupplösning +Slår ihop dubbletter till kanoniska Landvex-ID:n. +""" +import json +import difflib +from pathlib import Path +from typing import List, Dict, Tuple, Optional +from dataclasses import dataclass + +import numpy as np +from sklearn.feature_extraction.text import TfidfVectorizer +from sklearn.metrics.pairwise import cosine_similarity + +@dataclass +class EntitetsMatch: + kandidat_id: str + befintlig_id: str + likhet: float + match_typ: str # "namn", "spec", "bild", "kombinerad" + rekommendation: str # "sammanfoga", "granska", "separat" + +class EntitetsUpplosare: + """Upplöser entiteter och föreslår sammanslagningar.""" + + def __init__( + self, + namn_tröskel: float = 0.85, + spec_tröskel: float = 0.90, + bild_tröskel: float = 0.95, + ): + self.namn_tröskel = namn_tröskel + self.spec_tröskel = spec_tröskel + self.bild_tröskel = bild_tröskel + + self.vectorizer = TfidfVectorizer( + analyzer='word', + ngram_range=(1, 3), + min_df=1, + ) + + def namn_likhet(self, namn1: str, namn2: str) -> float: + """Beräkna likhet mellan två namn.""" + # Exakt match + if namn1.lower() == namn2.lower(): + return 1.0 + + # Fuzzy match + return difflib.SequenceMatcher(None, namn1.lower(), namn2.lower()).ratio() + + def spec_likhet(self, spec1: Dict, spec2: Dict) -> float: + """Beräkna likhet mellan tekniska specifikationer.""" + # Jämför nyckelfält + nyckel_falt = ["material", "ip_klass", "belastningsklass", "dimensioner"] + + matchande = 0 + totala = 0 + + for falt in nyckel_falt: + v1 = spec1.get(falt, "") + v2 = spec2.get(falt, "") + + if v1 and v2: + totala += 1 + if isinstance(v1, str) and isinstance(v2, str): + if v1.lower() == v2.lower(): + matchande += 1 + elif v1 == v2: + matchande += 1 + + return matchande / totala if totala > 0 else 0 + + def bild_likhet(self, emb1: np.ndarray, emb2: np.ndarray) -> float: + """Beräkna likhet mellan bildembeddings.""" + # Cosine similarity + return float(np.dot(emb1, emb2) / (np.linalg.norm(emb1) * np.linalg.norm(emb2))) + + def hitta_dubbletter( + self, + kandidat: Dict, + befintliga: List[Dict], + ) -> List[EntitetsMatch]: + """Hitta potentiella dubbletter för en kandidat.""" + matcher = [] + + for bef in befintliga: + likheter = [] + + # Namnlikhet + namn1 = kandidat.get("namn", {}).get("sv", "") + namn2 = bef.get("namn", {}).get("sv", "") + if namn1 and namn2: + nl = self.namn_likhet(namn1, namn2) + if nl >= self.namn_tröskel: + likheter.append(("namn", nl)) + + # Speclikhet + spec1 = kandidat.get("teknik", {}) + spec2 = bef.get("teknik", {}) + if spec1 and spec2: + sl = self.spec_likhet(spec1, spec2) + if sl >= self.spec_tröskel: + likheter.append(("spec", sl)) + + # Kombinerad likhet + if likheter: + medel = np.mean([l[1] for l in likheter]) + + # Rekommendation + if medel > 0.95: + rekommendation = "sammanfoga" + elif medel > 0.85: + rekommendation = "granska" + else: + rekommendation = "separat" + + matcher.append(EntitetsMatch( + kandidat_id=kandidat.get("lvx_id", "?"), + befintlig_id=bef.get("lvx_id", "?"), + likhet=medel, + match_typ=" + ".join([l[0] for l in likheter]), + rekommendation=rekommendation, + )) + + # Sortera efter likhet + matcher.sort(key=lambda x: x.likhet, reverse=True) + return matcher[:5] # Top 5 + + def sammanfoga(self, kandidat: Dict, befintlig: Dict) -> Dict: + """Sammanfoga två poster till en kanonisk post.""" + resultat = befintlig.copy() + + # Merge källor + kallor1 = set(json.dumps(k, sort_keys=True) for k in kandidat.get("proveniens", {}).get("kallor", [])) + kallor2 = set(json.dumps(k, sort_keys=True) for k in befintlig.get("proveniens", {}).get("kallor", [])) + alla_kallor = [json.loads(k) for k in kallor1 | kallor2] + + resultat.setdefault("proveniens", {})["kallor"] = alla_kallor + + # Höj verifieringsnivå om möjligt + verif_rank = {"obekraftad": 0, "kallbelagd": 1, "faltverifierad": 2, "tillverkarbekraftad": 3} + v1 = verif_rank.get(kandidat.get("proveniens", {}).get("verifieringsniva", "obekraftad"), 0) + v2 = verif_rank.get(befintlig.get("proveniens", {}).get("verifieringsniva", "obekraftad"), 0) + resultat["proveniens"]["verifieringsniva"] = max([v1, v2], key=lambda x: verif_rank.get(x, 0)) + + return resultat + +def main(): + """Demo: entitetsupplösning.""" + print("🔗 Landvex Entitetsupplösare") + + upplosare = EntitetsUpplosare() + + # Testdata + kandidat = { + "lvx_id": "KAND-001", + "namn": {"sv": "ABB Kabeldon CDC-LT", "en": "ABB Kabeldon CDC-LT"}, + "teknik": {"material": "stål", "ip_klass": "IP34D"}, + "proveniens": {"verifieringsniva": "kallbelagd", "kallor": [{"titel": "Test"}]}, + } + + befintliga = [ + { + "lvx_id": "LVX-ELN-0101", + "namn": {"sv": "Kabelskåp, lågspänning", "en": "LV distribution cabinet"}, + "teknik": {"material": "stål", "ip_klass": "IP34D"}, + "proveniens": {"verifieringsniva": "kallbelagd", "kallor": []}, + }, + { + "lvx_id": "LVX-ELN-0102", + "namn": {"sv": "Elstolpe", "en": "Utility pole"}, + "teknik": {"material": "trä", "ip_klass": ""}, + "proveniens": {"verifieringsniva": "kallbelagd", "kallor": []}, + }, + ] + + matcher = upplosare.hitta_dubbletter(kandidat, befintliga) + + print("\n🎯 Matchningar:") + for m in matcher: + print(f" {m.kandidat_id} ↔ {m.befintlig_id}") + print(f" Likhet: {m.likhet:.3f} ({m.match_typ})") + print(f" Rekommendation: {m.rekommendation}") + + if matcher and matcher[0].rekommendation == "sammanfoga": + sammanfogad = upplosare.sammanfoga(kandidat, befintliga[0]) + print(f"\n✅ Sammanfogad: {sammanfogad['lvx_id']}") + +if __name__ == "__main__": + main() diff --git a/projects/landvex/docker-compose.prod.yml b/projects/landvex/docker-compose.prod.yml new file mode 100644 index 000000000..458400632 --- /dev/null +++ b/projects/landvex/docker-compose.prod.yml @@ -0,0 +1,148 @@ +version: "3.9" + +services: + # ─── API ─────────────────────────────────────────────── + api: + build: ./api + ports: + - "8081:8080" + volumes: + - ./10-data:/app/data:ro + - ./vision/modeller:/models:ro + environment: + - DATA_DIR=/app/data + - NEO4J_URI=bolt://neo4j:7687 + - NEO4J_USER=neo4j + - NEO4J_PASSWORD=${NEO4J_PASSWORD} + - VISION_MODEL_PATH=/models/landvex-vision-model.pt + depends_on: + - neo4j + - vectordb + restart: unless-stopped + deploy: + resources: + limits: + memory: 2G + + # ─── Neo4j (Property Graph) ──────────────────────────── + neo4j: + image: neo4j:5.19-enterprise + ports: + - "7474:7474" + - "7687:7687" + environment: + - NEO4J_AUTH=neo4j/${NEO4J_PASSWORD} + - NEO4J_PLUGINS=["apoc"] + - NEO4J_dbms_memory_heap_max__size=8G + - NEO4J_dbms_memory_pagecache_size=4G + volumes: + - neo4j_data:/data + - neo4j_logs:/logs + restart: unless-stopped + deploy: + resources: + limits: + memory: 12G + + # ─── Vektordatabas (FAISS) ───────────────────────────── + vectordb: + build: ./vision/embeddings + ports: + - "8082:8080" + volumes: + - vectordb_data:/data + environment: + - FAISS_INDEX_DIR=/data + restart: unless-stopped + + # ─── Datafabrik (Skördning + Extraktion) ─────────────── + datafabrik: + build: ./datafabrik + volumes: + - datafabrik_output:/data/output + - datafabrik_koer:/data/koer + environment: + - OPENAI_API_KEY=${OPENAI_API_KEY} + - DATAFABRIK_KO_DIR=/data/koer + - DATAFABRIK_OUTPUT=/data/output + restart: unless-stopped + deploy: + resources: + limits: + memory: 4G + + # ─── Vision-träning (GPU) ────────────────────────────── + vision-trainer: + build: ./vision + volumes: + - vision_models:/models + - vision_dataset:/data/dataset + environment: + - CUDA_VISIBLE_DEVICES=0 + - VISION_MODEL_DIR=/models + - VISION_DATASET_DIR=/data/dataset + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: 1 + capabilities: [gpu] + profiles: ["training"] + + # ─── MinIO (Objektlagring för bilder/dokument) ───────── + minio: + image: minio/minio:latest + ports: + - "9000:9000" + - "9001:9001" + environment: + - MINIO_ROOT_USER=${MINIO_USER:-landvex} + - MINIO_ROOT_PASSWORD=${MINIO_PASSWORD:-landvex2024} + volumes: + - minio_data:/data + command: server /data --console-address ":9001" + restart: unless-stopped + + # ─── Redis (Köer och cache) ──────────────────────────── + redis: + image: redis:7-alpine + ports: + - "6379:6379" + volumes: + - redis_data:/data + restart: unless-stopped + + # ─── Prometheus (Metriker) ───────────────────────────── + prometheus: + image: prom/prometheus:latest + ports: + - "9090:9090" + volumes: + - ./infrastruktur/monitorering/prometheus.yml:/etc/prometheus/prometheus.yml:ro + - prometheus_data:/prometheus + restart: unless-stopped + + # ─── Grafana (Dashboards) ────────────────────────────── + grafana: + image: grafana/grafana:latest + ports: + - "3000:3000" + environment: + - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-landvex} + volumes: + - grafana_data:/var/lib/grafana + restart: unless-stopped + +volumes: + neo4j_data: + neo4j_logs: + vectordb_data: + datafabrik_output: + datafabrik_koer: + vision_models: + vision_dataset: + minio_data: + redis_data: + prometheus_data: + grafana_data: diff --git a/projects/landvex/infrastruktur/monitorering/dashboards/landvex.json b/projects/landvex/infrastruktur/monitorering/dashboards/landvex.json new file mode 100644 index 000000000..1ddba0303 --- /dev/null +++ b/projects/landvex/infrastruktur/monitorering/dashboards/landvex.json @@ -0,0 +1,72 @@ +{ + "dashboard": { + "title": "Landvex IOL", + "tags": ["landvex", "iol"], + "timezone": "UTC", + "panels": [ + { + "title": "API Requests/min", + "type": "graph", + "targets": [ + { + "expr": "rate(landvex_api_requests_total[5m])", + "legendFormat": "{{method}} {{endpoint}}" + } + ], + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0} + }, + { + "title": "Identify Konfidens", + "type": "graph", + "targets": [ + { + "expr": "histogram_quantile(0.95, rate(landvex_identify_confidence_bucket[5m]))", + "legendFormat": "p95" + }, + { + "expr": "histogram_quantile(0.50, rate(landvex_identify_confidence_bucket[5m]))", + "legendFormat": "p50" + } + ], + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0} + }, + { + "title": "Graf-storlek", + "type": "stat", + "targets": [ + { + "expr": "landvex_graph_nodes_total", + "legendFormat": "Noder" + }, + { + "expr": "landvex_graph_edges_total", + "legendFormat": "Kanter" + } + ], + "gridPos": {"h": 4, "w": 6, "x": 0, "y": 8} + }, + { + "title": "Kö-storlek", + "type": "stat", + "targets": [ + { + "expr": "landvex_queue_size", + "legendFormat": "{{queue}}" + } + ], + "gridPos": {"h": 4, "w": 6, "x": 6, "y": 8} + }, + { + "title": "Kostnad per faktum", + "type": "graph", + "targets": [ + { + "expr": "landvex_cost_per_fact_usd", + "legendFormat": "USD/faktum" + } + ], + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8} + } + ] + } +} diff --git a/projects/landvex/infrastruktur/monitorering/prometheus.yml b/projects/landvex/infrastruktur/monitorering/prometheus.yml new file mode 100644 index 000000000..552072277 --- /dev/null +++ b/projects/landvex/infrastruktur/monitorering/prometheus.yml @@ -0,0 +1,23 @@ +global: + scrape_interval: 15s + evaluation_interval: 15s + +scrape_configs: + - job_name: 'landvex-api' + static_configs: + - targets: ['api:8080'] + metrics_path: '/metrics' + + - job_name: 'landvex-datafabrik' + static_configs: + - targets: ['datafabrik:8080'] + metrics_path: '/metrics' + + - job_name: 'neo4j' + static_configs: + - targets: ['neo4j:7474'] + metrics_path: '/metrics' + + - job_name: 'prometheus' + static_configs: + - targets: ['localhost:9090'] diff --git a/projects/landvex/infrastruktur/terraform/main.tf b/projects/landvex/infrastruktur/terraform/main.tf new file mode 100644 index 000000000..85d92f074 --- /dev/null +++ b/projects/landvex/infrastruktur/terraform/main.tf @@ -0,0 +1,162 @@ +# Landvex Infrastruktur — AWS Terraform +# Skalar till 50GB+ agentkraft + +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 5.0" + } + } +} + +provider "aws" { + region = var.aws_region +} + +# ─── VPC och nätverk ───────────────────────────────────── +resource "aws_vpc" "landvex" { + cidr_block = "10.0.0.0/16" + enable_dns_hostnames = true + enable_dns_support = true + + tags = { + Name = "landvex-vpc" + } +} + +resource "aws_subnet" "public" { + count = 2 + vpc_id = aws_vpc.landvex.id + cidr_block = "10.0.${count.index + 1}.0/24" + availability_zone = data.aws_availability_zones.available.names[count.index] + map_public_ip_on_launch = true + + tags = { + Name = "landvex-public-${count.index + 1}" + } +} + +resource "aws_subnet" "private" { + count = 2 + vpc_id = aws_vpc.landvex.id + cidr_block = "10.0.${count.index + 10}.0/24" + availability_zone = data.aws_availability_zones.available.names[count.index] + + tags = { + Name = "landvex-private-${count.index + 1}" + } +} + +data "aws_availability_zones" "available" { + state = "available" +} + +# ─── ECR Repositories ──────────────────────────────────── +resource "aws_ecr_repository" "api" { + name = "landvex-api" + image_tag_mutability = "MUTABLE" + + image_scanning_configuration { + scan_on_push = true + } +} + +resource "aws_ecr_repository" "datafabrik" { + name = "landvex-datafabrik" + image_tag_mutability = "MUTABLE" + + image_scanning_configuration { + scan_on_push = true + } +} + +resource "aws_ecr_repository" "vision" { + name = "landvex-vision" + image_tag_mutability = "MUTABLE" + + image_scanning_configuration { + scan_on_push = true + } +} + +# ─── ECS Cluster ───────────────────────────────────────── +resource "aws_ecs_cluster" "landvex" { + name = "landvex-cluster" + + setting { + name = "containerInsights" + value = "enabled" + } +} + +resource "aws_ecs_cluster_capacity_providers" "landvex" { + cluster_name = aws_ecs_cluster.landvex.name + + capacity_providers = ["FARGATE", "FARGATE_SPOT"] + + default_capacity_provider_strategy { + base = 1 + weight = 1 + capacity_provider = "FARGATE" + } +} + +# ─── S3 Bucket för datalagring ─────────────────────────── +resource "aws_s3_bucket" "landvex_data" { + bucket = "landvex-data-${var.environment}" +} + +resource "aws_s3_bucket_versioning" "landvex_data" { + bucket = aws_s3_bucket.landvex_data.id + versioning_configuration { + status = "Enabled" + } +} + +# ─── RDS (PostgreSQL för metadata) ─────────────────────── +resource "aws_db_instance" "landvex_metadata" { + identifier = "landvex-metadata" + allocated_storage = 100 + max_allocated_storage = 1000 + engine = "postgres" + engine_version = "16" + instance_class = "db.r6g.xlarge" + db_name = "landvex" + username = "landvex" + password = var.db_password + skip_final_snapshot = true + + tags = { + Name = "landvex-metadata" + } +} + +# ─── Variables ─────────────────────────────────────────── +variable "aws_region" { + description = "AWS region" + default = "eu-north-1" +} + +variable "environment" { + description = "Environment (dev/staging/prod)" + default = "prod" +} + +variable "db_password" { + description = "Database password" + sensitive = true +} + +# ─── Outputs ───────────────────────────────────────────── +output "vpc_id" { + value = aws_vpc.landvex.id +} + +output "ecr_api_url" { + value = aws_ecr_repository.api.repository_url +} + +output "s3_bucket" { + value = aws_s3_bucket.landvex_data.bucket +} diff --git a/projects/landvex/vision/Dockerfile b/projects/landvex/vision/Dockerfile new file mode 100644 index 000000000..864e95b16 --- /dev/null +++ b/projects/landvex/vision/Dockerfile @@ -0,0 +1,32 @@ +FROM pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime + +WORKDIR /app + +# Systemberoenden +RUN apt-get update && apt-get install -y \ + tesseract-ocr \ + tesseract-ocr-swe \ + tesseract-ocr-eng \ + libgl1-mesa-glx \ + libglib2.0-0 \ + && rm -rf /var/lib/apt/lists/* + +# Python-paket +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +# Kopiera kod +COPY modeller/ ./modeller/ +COPY traening/ ./traening/ +COPY embeddings/ ./embeddings/ +COPY dataset/ ./dataset/ + +# Miljövariabler +ENV PYTHONPATH=/app +ENV VISION_MODEL_DIR=/models +ENV VISION_DATASET_DIR=/data/dataset + +VOLUME ["/models", "/data"] + +# Default: träna +CMD ["python3", "-m", "traening.trainer"] diff --git a/projects/landvex/vision/embeddings/ocr_pipeline.py b/projects/landvex/vision/embeddings/ocr_pipeline.py new file mode 100644 index 000000000..cddd7a986 --- /dev/null +++ b/projects/landvex/vision/embeddings/ocr_pipeline.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +""" +Landvex Vision — OCR-pipeline för typskyltar +Extraherar tillverkare, modell, serienummer från foton. +""" +import json +from pathlib import Path +from typing import List, Dict, Optional, Tuple +from dataclasses import dataclass + +import pytesseract +from PIL import Image +import cv2 +import numpy as np + +@dataclass +class OCRResultat: + text: str + konfidens: float + bbox: Tuple[int, int, int, int] # x, y, w, h + falt: Dict[str, str] # Extraherade fält + +class LandvexOCR: + """OCR-pipeline optimerad för infrastrukturtypskyltar.""" + + def __init__(self, tesseract_cmd: Optional[str] = None): + if tesseract_cmd: + pytesseract.pytesseract.tesseract_cmd = tesseract_cmd + + # Fältmönster per objektklass + self.falt_monster = { + "LVX-ELN-0101": { # Kabelskåp + "tillverkare": r"(ABB|Schneider|Siemens|GE)\s*[\w-]*", + "modell": r"(CDC|Prisma|Okken|IM\s*\w+)", + "ar": r"20\d{2}", + }, + "LVX-TRP-0102": { # Vägbelysningsarmatur + "tillverkare": r"(Philips|Thorn|Schreder|Acuity|Cree)", + "modell": r"(SL-\w+|ER\w+|Vista\w+)", + "effekt": r"(\d+)\s*W", + }, + "LVX-VAT-0101": { # Brunnsbetäckning + "tillverkare": r"(ULMA|ACO|Wrede|GDK)", + "klass": r"(A15|B125|C250|D400|E600|F900)", + "material": r"(gjutjärn|stål|komposit|betong)", + }, + } + + def forbehandla_bild(self, bild: np.ndarray) -> np.ndarray: + """Förbättra bildkvalitet för OCR.""" + # Konvertera till gråskala + if len(bild.shape) == 3: + gray = cv2.cvtColor(bild, cv2.COLOR_RGB2GRAY) + else: + gray = bild + + # Brusreducering + denoised = cv2.fastNlMeansDenoising(gray) + + # Kontrastförbättring (CLAHE) + clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)) + enhanced = clahe.apply(denoised) + + # Skärpa + kernel = np.array([[-1, -1, -1], [-1, 9, -1], [-1, -1, -1]]) + sharpened = cv2.filter2D(enhanced, -1, kernel) + + return sharpened + + def hitta_text_regioner(self, bild: np.ndarray) -> List[Tuple[int, int, int, int]]: + """Hitta regioner som troligen innehåller text.""" + # MSER (Maximally Stable Extremal Regions) + mser = cv2.MSER_create() + regions, _ = mser.detectRegions(bild) + + # Filtrera små regioner + h, w = bild.shape + min_area = (h * w) * 0.001 # Minst 0.1% av bilden + + bboxes = [] + for region in regions: + x, y, w, h = cv2.boundingRect(region) + if w * h > min_area and w > h * 2: # Text är oftast bredare än hög + bboxes.append((x, y, w, h)) + + return bboxes + + def kora_ocr(self, bild: Image.Image, lvx_id: Optional[str] = None) -> List[OCRResultat]: + """Kör OCR på bild och extrahera fält.""" + # Konvertera till numpy + img_array = np.array(bild) + + # Förbehandla + processed = self.forbehandla_bild(img_array) + + # Hitta textregioner + regioner = self.hitta_text_regioner(processed) + + resultat = [] + for x, y, w, h in regioner[:5]: # Max 5 regioner + # Beskär region + roi = processed[y:y+h, x:x+w] + + # Kör Tesseract + text = pytesseract.image_to_string(roi, lang='eng+swe') + conf_data = pytesseract.image_to_data(roi, output_type=pytesseract.Output.DICT) + + # Beräkna medelkonfidens + konfidenser = [c for c in conf_data['conf'] if c > 0] + medel_konf = np.mean(konfidenser) if konfidenser else 0 + + # Extrahera fält om vi vet objektklassen + falt = {} + if lvx_id and lvx_id in self.falt_monster: + import re + for falt_namn, monster in self.falt_monster[lvx_id].items(): + match = re.search(monster, text, re.IGNORECASE) + if match: + falt[falt_namn] = match.group(1) + + resultat.append(OCRResultat( + text=text.strip(), + konfidens=medel_konf / 100.0, # Normalisera till 0-1 + bbox=(x, y, w, h), + falt=falt + )) + + # Sortera efter konfidens + resultat.sort(key=lambda x: x.konfidens, reverse=True) + return resultat + + def extrahera_falt(self, text: str, lvx_id: str) -> Dict[str, str]: + """Extrahera strukturerade fält från OCR-text.""" + import re + + falt = {} + monster = self.falt_monster.get(lvx_id, {}) + + for falt_namn, pattern in monster.items(): + matches = re.findall(pattern, text, re.IGNORECASE) + if matches: + falt[falt_namn] = matches[0] + + return falt + +def main(): + """Demo: OCR på syntetisk bild.""" + print("🔤 Landvex OCR Pipeline") + + ocr = LandvexOCR() + + # Skapa syntetisk testbild med text + from PIL import ImageDraw, ImageFont + img = Image.new('RGB', (400, 200), color='white') + draw = ImageDraw.Draw(img) + + try: + font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", 24) + except: + font = ImageFont.load_default() + + draw.text((20, 20), "ABB Kabeldon", fill='black', font=font) + draw.text((20, 60), "CDC-LT 400", fill='black', font=font) + draw.text((20, 100), "2023", fill='black', font=font) + + # Kör OCR + resultat = ocr.kora_ocr(img, lvx_id="LVX-ELN-0101") + + print("\n🎯 OCR-resultat:") + for i, r in enumerate(resultat[:3]): + print(f" Region {i+1}:") + print(f" Text: {r.text[:100]}") + print(f" Konfidens: {r.konfidens:.2f}") + print(f" Fält: {r.falt}") + + print("\n✅ OCR klar!") + +if __name__ == "__main__": + main() diff --git a/projects/landvex/vision/embeddings/vektor_databas.py b/projects/landvex/vision/embeddings/vektor_databas.py new file mode 100644 index 000000000..39052319c --- /dev/null +++ b/projects/landvex/vision/embeddings/vektor_databas.py @@ -0,0 +1,140 @@ +#!/usr/bin/env python3 +""" +Landvex Vision — Vektordatabas +Lagrar och söker bild- och textembeddings. +""" +import json +import numpy as np +from pathlib import Path +from typing import List, Dict, Tuple, Optional +from dataclasses import dataclass + +import faiss # Facebook AI Similarity Search + +@dataclass +class VektorsokResultat: + lvx_id: str + distans: float + metadata: dict + +class LandvexVektordatabas: + """Vektordatabas för bild- och textembeddings.""" + + def __init__(self, dimension: int = 2048, index_type: str = "FlatIP"): + self.dimension = dimension + self.index_type = index_type + + # FAISS index för snabb sökning + if index_type == "FlatIP": + self.index = faiss.IndexFlatIP(dimension) # Inner product (cosine om normaliserad) + elif index_type == "IVF":: + nlist = 100 # Antal kluster + quantizer = faiss.IndexFlatIP(dimension) + self.index = faiss.IndexIVFFlat(quantizer, dimension, nlist) + else: + raise ValueError(f"Okänd index-typ: {index_type}") + + # Mappning index_id → lvx_id + self.id_map: List[str] = [] + self.metadata: Dict[str, dict] = {} + + def lagg_till(self, lvx_id: str, embedding: np.ndarray, metadata: dict): + """Lägg till embedding i databasen.""" + # Normalisera för cosine similarity + embedding = embedding / np.linalg.norm(embedding) + embedding = embedding.reshape(1, -1).astype('float32') + + self.index.add(embedding) + self.id_map.append(lvx_id) + self.metadata[lvx_id] = metadata + + def sok(self, query: np.ndarray, top_k: int = 5) -> List[VektorsokResultat]: + """Sök närmaste grannar.""" + query = query / np.linalg.norm(query) + query = query.reshape(1, -1).astype('float32') + + distanser, indices = self.index.search(query, top_k) + + resultat = [] + for dist, idx in zip(distanser[0], indices[0]): + if idx < 0 or idx >= len(self.id_map): + continue + + lvx_id = self.id_map[idx] + resultat.append(VektorsokResultat( + lvx_id=lvx_id, + distans=float(dist), + metadata=self.metadata.get(lvx_id, {}) + )) + + return resultat + + def spara(self, path: Path): + """Spara index och metadata.""" + path.parent.mkdir(parents=True, exist_ok=True) + + # Spara FAISS-index + faiss.write_index(self.index, str(path / "index.faiss")) + + # Spara metadata + with open(path / "metadata.json", 'w') as f: + json.dump({ + "id_map": self.id_map, + "metadata": self.metadata, + "dimension": self.dimension, + "index_type": self.index_type, + }, f, indent=2) + + print(f" 💾 Vektordatabas sparad: {path}") + + def ladda(self, path: Path): + """Ladda index och metadata.""" + self.index = faiss.read_index(str(path / "index.faiss")) + + with open(path / "metadata.json", 'r') as f: + data = json.load(f) + + self.id_map = data["id_map"] + self.metadata = data["metadata"] + self.dimension = data["dimension"] + self.index_type = data["index_type"] + + print(f" 📂 Vektordatabas laddad: {len(self.id_map)} vektorer") + +def main(): + """Demo: vektordatabas.""" + print("🔍 Landvex Vektordatabas") + + db = LandvexVektordatabas(dimension=128) # Låg dimension för demo + + # Lägg till exempel-embeddings + np.random.seed(42) + for i in range(100): + emb = np.random.randn(128) + db.lagg_till( + lvx_id=f"LVX-TRP-{1000+i:04d}", + embedding=emb, + metadata={ + "namn_sv": f"Testobjekt {i}", + "verifieringsniva": "kallbelagd", + } + ) + + # Sök + query = np.random.randn(128) + resultat = db.sok(query, top_k=5) + + print("\n🎯 Sökresultat:") + for r in resultat: + print(f" {r.lvx_id}: distans={r.distans:.3f}, {r.metadata['namn_sv']}") + + # Spara och ladda + db.spara(Path("/tmp/landvex-vectordb")) + + db2 = LandvexVektordatabas() + db2.ladda(Path("/tmp/landvex-vectordb")) + + print(f"\n✅ Databas: {len(db2.id_map)} vektorer") + +if __name__ == "__main__": + main() diff --git a/projects/landvex/vision/modeller/identify_model.py b/projects/landvex/vision/modeller/identify_model.py new file mode 100644 index 000000000..e9dad2ef0 --- /dev/null +++ b/projects/landvex/vision/modeller/identify_model.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +""" +Landvex Vision — Identify-modell +Foto in → rankade kandidater med konfidens +""" +import json +import hashlib +from pathlib import Path +from typing import List, Dict, Tuple, Optional +from dataclasses import dataclass + +import torch +import torch.nn as nn +from torchvision import models, transforms +from PIL import Image +import numpy as np + +@dataclass +class IdentifyKandidat: + lvx_id: str + posttyp: str + namn_sv: str + namn_en: str + konfidens: float + verifieringsniva: str + kannetecken_match: List[str] + embedding_distans: float + +class LandvexIdentifyModel: + """Vision-modell för infrastrukturidentifiering.""" + + def __init__(self, model_path: Optional[Path] = None): + self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + print(f" 🖥️ Enhet: {self.device}") + + # Ladda förtränad ResNet som backbone + self.backbone = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2) + self.backbone = nn.Sequential(*list(self.backbone.children())[:-1]) # Ta bort sista FC + self.backbone = self.backbone.to(self.device) + self.backbone.eval() + + # Transform för bilder + self.transform = transforms.Compose([ + transforms.Resize(256), + transforms.CenterCrop(224), + transforms.ToTensor(), + transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), + ]) + + # Embedding-databas (lvx_id → embedding) + self.embeddings: Dict[str, np.ndarray] = {} + self.metadata: Dict[str, dict] = {} + + if model_path and model_path.exists(): + self.ladda(model_path) + + def bild_till_embedding(self, bild: Image.Image) -> np.ndarray: + """Konvertera bild till embedding-vektor.""" + tensor = self.transform(bild).unsqueeze(0).to(self.device) + + with torch.no_grad(): + embedding = self.backbone(tensor) + + return embedding.cpu().numpy().flatten() + + def lagg_till_klass(self, lvx_id: str, bilder: List[Image.Image], metadata: dict): + """Lägg till en objektklass med träningsbilder.""" + embeddings = [self.bild_till_embedding(b) for b in bilder] + medel_embedding = np.mean(embeddings, axis=0) + + self.embeddings[lvx_id] = medel_embedding / np.linalg.norm(medel_embedding) + self.metadata[lvx_id] = metadata + + def identifiera(self, bild: Image.Image, top_k: int = 5) -> List[IdentifyKandidat]: + """Identifiera objekt i bild. Returnera top-k kandidater.""" + if not self.embeddings: + return [] + + query_embedding = self.bild_till_embedding(bild) + query_embedding = query_embedding / np.linalg.norm(query_embedding) + + # Beräkna kosinuslikhet + resultat = [] + for lvx_id, emb in self.embeddings.items(): + distans = np.dot(query_embedding, emb) + meta = self.metadata[lvx_id] + + # Konfidens = likhet * verifieringsnivå-faktor + verif_faktor = { + "obekraftad": 0.5, + "kallbelagd": 0.75, + "faltverifierad": 0.9, + "tillverkarbekraftad": 0.95, + }.get(meta.get("verifieringsniva", "obekraftad"), 0.5) + + konfidens = float(distans * verif_faktor) + + resultat.append(IdentifyKandidat( + lvx_id=lvx_id, + posttyp=meta.get("posttyp", "objektklass"), + namn_sv=meta.get("namn_sv", ""), + namn_en=meta.get("namn_en", ""), + konfidens=konfidens, + verifieringsniva=meta.get("verifieringsniva", "obekraftad"), + kannetecken_match=meta.get("kannetecken", [])[:3], + embedding_distans=float(distans), + )) + + # Sortera efter konfidens + resultat.sort(key=lambda x: x.konfidens, reverse=True) + return resultat[:top_k] + + def spara(self, path: Path): + """Spara modell och embeddings.""" + path.parent.mkdir(parents=True, exist_ok=True) + data = { + "embeddings": {k: v.tolist() for k, v in self.embeddings.items()}, + "metadata": self.metadata, + } + torch.save(data, path) + print(f" 💾 Modell sparad: {path}") + + def ladda(self, path: Path): + """Ladda modell och embeddings.""" + data = torch.load(path, map_location=self.device) + self.embeddings = {k: np.array(v) for k, v in data["embeddings"].items()} + self.metadata = data["metadata"] + print(f" 📂 Modell laddad: {len(self.embeddings)} klasser") + +def main(): + """Demo: träna på syntetiska data och identifiera.""" + from PIL import ImageDraw + + model = LandvexIdentifyModel() + + # Skapa syntetiska träningsbilder (i verkligheten: riktiga foton) + def skapa_testbild(farg, storlek=(224, 224)): + img = Image.new('RGB', storlek, farg) + draw = ImageDraw.Draw(img) + draw.rectangle([50, 50, 174, 174], outline="white", width=3) + return img + + # Lägg till två klasser + model.lagg_till_klass("LVX-TRP-0101", [ + skapa_testbild("gray"), + skapa_testbild("lightgray"), + ], { + "posttyp": "objektklass", + "namn_sv": "Belysningsstolpe", + "namn_en": "Lighting column", + "verifieringsniva": "kallbelagd", + "kannetecken": ["Grå stolpe", "Ljustopp"], + }) + + model.lagg_till_klass("LVX-TRP-0102", [ + skapa_testbild("blue"), + skapa_testbild("darkblue"), + ], { + "posttyp": "objektklass", + "namn_sv": "Vägbelysningsarmatur", + "namn_en": "Road lighting luminaire", + "verifieringsniva": "kallbelagd", + "kannetecken": ["Blå armatur", "LED-ljus"], + }) + + # Testa identifiering + test_bild = skapa_testbild("gray") + kandidater = model.identifiera(test_bild, top_k=2) + + print("\n🎯 Identifieringsresultat:") + for k in kandidater: + print(f" {k.lvx_id}: {k.namn_sv} (konfidens: {k.konfidens:.3f})") + + # Spara modell + model.spara(Path("/tmp/landvex-vision-model.pt")) + +if __name__ == "__main__": + main() diff --git a/projects/landvex/vision/traening/trainer.py b/projects/landvex/vision/traening/trainer.py new file mode 100644 index 000000000..246cfb8e9 --- /dev/null +++ b/projects/landvex/vision/traening/trainer.py @@ -0,0 +1,232 @@ +#!/usr/bin/env python3 +""" +Landvex Vision — Träningspipeline +Kontrastivt lärande från Zoomer-foton och tillverkarbilder. +""" +import json +from pathlib import Path +from typing import List, Dict, Tuple +from dataclasses import dataclass + +import torch +import torch.nn as nn +import torch.optim as optim +from torch.utils.data import Dataset, DataLoader +from torchvision import models, transforms +from PIL import Image +import numpy as np +from tqdm import tqdm + +@dataclass +class Traeningsexempel: + bild_path: Path + lvx_id: str + positiv: bool # True = matchar lvx_id, False = negativt exempel + kalla: str # "zoomer", "tillverkare", "syntetisk" + +class LandvexDataset(Dataset): + """Dataset för kontrastivt lärande.""" + + def __init__(self, exempel: List[Traeningsexempel], transform=None): + self.exempel = exempel + self.transform = transform or transforms.Compose([ + transforms.Resize(256), + transforms.CenterCrop(224), + transforms.ToTensor(), + transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), + ]) + + def __len__(self): + return len(self.exempel) + + def __getitem__(self, idx): + ex = self.exempel[idx] + bild = Image.open(ex.bild_path).convert('RGB') + + if self.transform: + bild = self.transform(bild) + + return bild, ex.lvx_id, ex.positiv, ex.kalla + +class KontrastivtVerlust(nn.Module): + """NT-Xent loss (Normalized Temperature-scaled Cross Entropy).""" + + def __init__(self, temperatur: float = 0.5): + super().__init__() + self.temperatur = temperatur + self.cos_sim = nn.CosineSimilarity(dim=-1) + + def forward(self, z_i: torch.Tensor, z_j: torch.Tensor) -> torch.Tensor: + """ + z_i, z_j: normaliserade embeddings [batch_size, dim] + """ + # Cosine similarity + sim = self.cos_sim(z_i.unsqueeze(1), z_j.unsqueeze(0)) / self.temperatur + + # Positiva par är på diagonalen + etiketter = torch.arange(len(z_i)).to(z_i.device) + + # Cross entropy + return nn.functional.cross_entropy(sim, etiketter) + +class LandvexTrainer: + """Träningspipeline för Identify-modellen.""" + + def __init__( + self, + model: nn.Module, + device: torch.device, + learning_rate: float = 1e-4, + temperatur: float = 0.5 + ): + self.model = model.to(device) + self.device = device + self.optimizer = optim.Adam(model.parameters(), lr=learning_rate) + self.criterion = KontrastivtVerlust(temperatur) + self.epoch = 0 + + def trana_epok(self, dataloader: DataLoader) -> dict: + """Träna en epok.""" + self.model.train() + total_loss = 0 + antal_batch = 0 + + for bilder, lvx_ids, positiva, kallor in tqdm(dataloader, desc=f"Epok {self.epoch}"): + bilder = bilder.to(self.device) + + # Forward pass + embeddings = self.model(bilder) + + # Kontrastivt förlust + # Dela i två vyer (augmentation) + batch_size = len(bilder) // 2 + z_i = embeddings[:batch_size] + z_j = embeddings[batch_size:] + + loss = self.criterion(z_i, z_j) + + # Backward pass + self.optimizer.zero_grad() + loss.backward() + self.optimizer.step() + + total_loss += loss.item() + antal_batch += 1 + + self.epoch += 1 + + return { + "epok": self.epoch, + "medel_loss": total_loss / antal_batch, + } + + def utvardera(self, dataloader: DataLoader) -> dict: + """Utvärdera modellen.""" + self.model.eval() + korrekta = 0 + totala = 0 + + with torch.no_grad(): + for bilder, lvx_ids, positiva, kallor in dataloader: + bilder = bilder.to(self.device) + embeddings = self.model(bilder) + + # TODO: Implementera top-k utvärdering + totala += len(bilder) + + return { + "noggrannhet": korrekta / totala if totala > 0 else 0, + "antal": totala, + } + + def spara(self, path: Path): + """Spara träningsstatus.""" + path.parent.mkdir(parents=True, exist_ok=True) + torch.save({ + "epoch": self.epoch, + "model_state": self.model.state_dict(), + "optimizer_state": self.optimizer.state_dict(), + }, path) + print(f" 💾 Träningsstatus sparad: {path}") + + def ladda(self, path: Path): + """Ladda träningsstatus.""" + checkpoint = torch.load(path, map_location=self.device) + self.model.load_state_dict(checkpoint["model_state"]) + self.optimizer.load_state_dict(checkpoint["optimizer_state"]) + self.epoch = checkpoint["epoch"] + print(f" 📂 Träningsstatus laddad: epok {self.epoch}") + +def skapa_syntetiskt_dataset(output_dir: Path, antal_klasser: int = 10, antal_bilder_per_klass: int = 20): + """Skapa syntetiskt dataset för testning.""" + from PIL import ImageDraw + + output_dir.mkdir(parents=True, exist_ok=True) + exempel = [] + + for klass_idx in range(antal_klasser): + lvx_id = f"LVX-TRP-{1000 + klass_idx:04d}" + klass_dir = output_dir / lvx_id + klass_dir.mkdir(exist_ok=True) + + for bild_idx in range(antal_bilder_per_klass): + # Skapa syntetisk bild + img = Image.new('RGB', (224, 224), color=(klass_idx * 20, 100, 150)) + draw = ImageDraw.Draw(img) + draw.rectangle([50, 50, 174, 174], outline="white", width=3) + + # Spara + bild_path = klass_dir / f"{bild_idx:03d}.jpg" + img.save(bild_path) + + exempel.append(Traeningsexempel( + bild_path=bild_path, + lvx_id=lvx_id, + positiv=True, + kalla="syntetisk" + )) + + # Spara metadata + with open(output_dir / "dataset.json", 'w') as f: + json.dump([{ + "bild_path": str(e.bild_path), + "lvx_id": e.lvx_id, + "positiv": e.positiv, + "kalla": e.kalla, + } for e in exempel], f, indent=2) + + return exempel + +def main(): + """Demo: träna på syntetiskt dataset.""" + print("🚀 Landvex Vision Trainer") + + # Skapa dataset + dataset_dir = Path("/tmp/landvex-vision-dataset") + exempel = skapa_syntetiskt_dataset(dataset_dir, antal_klasser=5, antal_bilder_per_klass=10) + print(f"📊 Dataset: {len(exempel)} exempel") + + # Skapa modell + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + backbone = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2) + backbone.fc = nn.Identity() # Ta bort klassificeringslager + + # Dataset och dataloader + dataset = LandvexDataset(exempel) + dataloader = DataLoader(dataset, batch_size=8, shuffle=True) + + # Tränare + trainer = LandvexTrainer(backbone, device) + + # Träna + for epok in range(3): + resultat = trainer.trana_epok(dataloader) + print(f" Epok {resultat['epok']}: loss = {resultat['medel_loss']:.4f}") + + # Spara + trainer.spara(Path("/tmp/landvex-vision-checkpoint.pt")) + + print("\n✅ Träning klar!") + +if __name__ == "__main__": + main()