# ETL-Pipeline Migration: CSV → KI Workflow → Frontend ## System-Architektur (Klargestellt) ### Aktueller Zustand (Nicht optimal) ``` ┌─────────────────────────────────────────────────────┐ │ Laravel Frontend │ │ │ │ CSV Upload → Manual Processing │ │ ↓ │ │ public.companies (11 Felder) │ │ ↓ │ │ public.transactions (49 Felder, komplex) │ │ - Core-Felder (14) │ │ - Enrichment-Felder (35) │ │ │ └─────────────────────────────────────────────────────┘ ``` **Probleme**: - ❌ Companies und Transactions vermischt - ❌ Enrichment-Logik direkt in Laravel - ❌ Keine klare Trennung: Raw Data vs Processed Data - ❌ Skalierbarkeit begrenzt --- ### Ziel-Zustand (Backend-Pipeline) ``` ┌──────────────────────────────────────────────────────────────┐ │ CSV Upload │ │ ↓ │ │ backend.transactions (14 Felder) │ │ - corporate_entity │ │ - corporate_counterparty │ │ - tx_date, tx_amount, tx_currency │ │ - raw_payload (Original-CSV) │ │ - status: 'pending' │ └─────────────────────────┬────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────────┐ │ KI Workflow (Python/MCP) │ │ │ │ Für jede Transaction: │ │ 1. Analyse: Risk Assessment │ │ 2. Enrichment: External APIs │ │ 3. Output: Strukturierte Ergebnisse │ │ │ │ Nutzt: backend.prompt_templates │ │ backend.prompt_runs │ └─────────────────────────┬────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────────┐ │ backend.transaction_outputs (5 Felder) │ │ │ │ - transaction_id (FK → backend.transactions.id) │ │ - prompt_id (FK → backend.prompt_templates.id) │ │ - output_key (z.B. 'risk_assessment', 'sanctions_check') │ │ - content (JSON mit Ergebnissen) │ │ - run_id (Batch-Tracking) │ │ │ │ Beispiel-Outputs: │ │ • 'risk_assessment' → {score: 85, level: 'high'} │ │ • 'sanctions_check' → {found: true, matches: [...]} │ │ • 'pep_check' → {is_pep: false} │ │ • 'company_info' → {name, sector, country} │ └─────────────────────────┬────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────────┐ │ Laravel Frontend │ │ │ │ Liest von: │ │ - backend.transactions (Raw Data) │ │ - backend.transaction_outputs (Processed Results) │ │ │ │ Zeigt an: │ │ - Transaction Details │ │ - Risk Score & Assessment │ │ - Enrichment Results (Sanctions, PEP, etc.) │ │ - Review Queue │ └──────────────────────────────────────────────────────────────┘ ``` --- ## Mapping: Alt → Neu ### 1. Companies Table → ENTFÄLLT **Warum?** - Companies sind **keine separate Entität** mehr - Firmeninformationen kommen aus: 1. `backend.transactions.corporate_entity` (Name aus CSV) 2. `backend.transaction_outputs` mit `output_key='company_info'` (KI-Enrichment) **Migration**: ```sql -- Bestehende Companies werden zu Transaction Outputs INSERT INTO backend.transaction_outputs ( transaction_id, prompt_id, output_key, content ) SELECT bt.id as transaction_id, 1 as prompt_id, -- Default prompt für Company Info 'company_info' as output_key, jsonb_build_object( 'name', c.name, 'legal_name', c.legal_name, 'ticker', c.ticker, 'sector', c.sector, 'country', c.country, 'headquarters', c.headquarters, 'kyc_risk_level', c.kyc_risk_level, 'summary', c.summary ) as content FROM public.companies c JOIN public.transactions t ON t.company_id = c.id JOIN backend.transactions bt ON bt.corporate_entity = c.name; ``` ### 2. Transactions Table → transaction_outputs **public.transactions (49 Felder)**: ``` Core-Felder (14): - company_id, reference, amount, currency - counterparty, executed_at, status - risk_score, requires_review → Gespeichert in backend.transactions Enrichment-Felder (35): - registry_data, sanctions_data, pep_data, ... → Jedes wird zu einem Output in transaction_outputs ``` **Mapping-Strategie**: | public.transactions Feld | Ziel | output_key | |--------------------------|------|------------| | amount, currency, counterparty | backend.transactions | - | | risk_score, requires_review | transaction_outputs | 'risk_assessment' | | registry_data | transaction_outputs | 'registry' | | sanctions_data | transaction_outputs | 'sanctions' | | pep_data | transaction_outputs | 'pep' | | gleif_data | transaction_outputs | 'gleif' | | ... | ... | ... | --- ## Phase 1: Backend Models & Relationships (Tag 1-2) ### 1.1 Database Config **config/database.php**: ```php 'backend' => [ 'driver' => 'pgsql', 'host' => env('DB_HOST', '127.0.0.1'), 'port' => env('DB_PORT', '5432'), 'database' => env('DB_DATABASE', 'forge'), 'username' => env('DB_USERNAME', 'forge'), 'password' => env('DB_PASSWORD', ''), 'charset' => 'utf8', 'prefix' => '', 'search_path' => 'backend', 'sslmode' => 'prefer', ], ``` ### 1.2 Backend\Transaction Model **app/Models/Backend/Transaction.php**: ```php 'decimal:2', // tx_date ist noch text - später zu timestamp migrieren ]; /** * KI-generierte Outputs für diese Transaktion */ public function outputs(): HasMany { return $this->hasMany(TransactionOutput::class, 'transaction_id'); } /** * Hole spezifischen Output-Typ */ public function getOutput(string $key): ?array { return $this->outputs() ->where('output_key', $key) ->first() ?->content; } /** * Hole alle Outputs als Key-Value Array */ public function getOutputsArray(): array { return $this->outputs() ->get() ->pluck('content', 'output_key') ->toArray(); } /** * Company Info aus Outputs */ public function getCompanyInfo(): ?array { return $this->getOutput('company_info'); } /** * Risk Assessment aus Outputs */ public function getRiskAssessment(): ?array { return $this->getOutput('risk_assessment'); } /** * Helper: Hat diese Transaction einen bestimmten Output? */ public function hasOutput(string $key): bool { return $this->outputs()->where('output_key', $key)->exists(); } /** * Ist KI-Verarbeitung abgeschlossen? */ public function isProcessed(): bool { return $this->status === self::STATUS_COMPLETED; } /** * Benötigt Review? */ public function requiresReview(): bool { $risk = $this->getRiskAssessment(); if (!$risk) { return true; // Keine Risk-Assessment → Review } return ($risk['score'] ?? 0) >= 70 || ($risk['requires_review'] ?? false); } } ``` ### 1.3 Backend\TransactionOutput Model **app/Models/Backend/TransactionOutput.php**: ```php 'array', // Automatisch JSON encode/decode ]; /** * Transaction zu der dieser Output gehört */ public function transaction(): BelongsTo { return $this->belongsTo(Transaction::class, 'transaction_id'); } /** * Prompt Template das diesen Output erzeugt hat */ public function promptTemplate(): BelongsTo { return $this->belongsTo(PromptTemplate::class, 'prompt_id'); } /** * Output-Key Konstanten für Type Safety */ public const KEY_COMPANY_INFO = 'company_info'; public const KEY_RISK_ASSESSMENT = 'risk_assessment'; public const KEY_SANCTIONS = 'sanctions'; public const KEY_PEP = 'pep'; public const KEY_REGISTRY = 'registry'; public const KEY_GLEIF = 'gleif'; public const KEY_INSOLVENCY = 'insolvency'; public const KEY_BUNDESANZEIGER = 'bundesanzeiger'; public const KEY_RSS = 'rss'; public const KEY_EU_SANCTIONS = 'eu_sanctions'; public const KEY_HANDELSREGISTER = 'handelsregister'; /** * Alle verfügbaren Output-Keys */ public static function availableKeys(): array { return [ self::KEY_COMPANY_INFO, self::KEY_RISK_ASSESSMENT, self::KEY_SANCTIONS, self::KEY_PEP, self::KEY_REGISTRY, self::KEY_GLEIF, self::KEY_INSOLVENCY, self::KEY_BUNDESANZEIGER, self::KEY_RSS, self::KEY_EU_SANCTIONS, self::KEY_HANDELSREGISTER, ]; } } ``` ### 1.4 Backend\PromptTemplate Model (optional) **app/Models/Backend/PromptTemplate.php**: ```php hasMany(TransactionOutput::class, 'prompt_id'); } } ``` --- ## Phase 2: CSV Upload Service (Tag 3-4) ### 2.1 CSV Upload Controller **app/Http/Controllers/TransactionUploadController.php**: ```php validate([ 'csv_file' => 'required|file|mimes:csv,txt|max:10240', // 10MB ]); try { $result = $this->uploadService->process($validated['csv_file']); return redirect() ->route('transactions.index') ->with('success', "Imported {$result['count']} transactions. Processing started."); } catch (\Exception $e) { return back() ->withErrors(['csv_file' => $e->getMessage()]) ->withInput(); } } } ``` ### 2.2 Upload Service **app/Services/TransactionUploadService.php**: ```php storeFile($file); // 2. Parse CSV $rows = $this->parseCsv($file); // 3. Validiere Daten $this->validate($rows); // 4. Importiere zu backend.transactions $transactions = $this->import($rows, $fileName); // 5. Starte KI-Verarbeitung (async) $this->queueAiProcessing($transactions); return [ 'count' => count($transactions), 'file' => $fileName, ]; } /** * Speichere Original-CSV für Audit */ private function storeFile(UploadedFile $file): string { return Storage::disk('local')->putFileAs( 'uploads/transactions', $file, date('Y-m-d_His') . '_' . $file->getClientOriginalName() ); } /** * Parse CSV-Datei */ private function parseCsv(UploadedFile $file): array { $handle = fopen($file->getRealPath(), 'r'); $headers = fgetcsv($handle); // Erste Zeile = Header $rows = []; while (($data = fgetcsv($handle)) !== false) { $rows[] = array_combine($headers, $data); } fclose($handle); return $rows; } /** * Validiere CSV-Daten */ private function validate(array $rows): void { if (empty($rows)) { throw new \InvalidArgumentException('CSV file is empty'); } $requiredColumns = [ 'corporate_entity', 'corporate_counterparty', 'tx_date', 'tx_amount', ]; $headers = array_keys($rows[0]); $missing = array_diff($requiredColumns, $headers); if (!empty($missing)) { throw new \InvalidArgumentException( 'Missing required columns: ' . implode(', ', $missing) ); } } /** * Importiere Transaktionen */ private function import(array $rows, string $fileName): array { $transactions = []; DB::connection('backend')->transaction(function () use ($rows, $fileName, &$transactions) { foreach ($rows as $row) { $transactions[] = Transaction::create([ 'corporate_entity' => $row['corporate_entity'], 'corporate_counterparty' => $row['corporate_counterparty'], 'tx_date' => $row['tx_date'], 'tx_amount' => (float) $row['tx_amount'], 'tx_currency' => $row['tx_currency'] ?? 'EUR', 'tx_purpose' => $row['tx_purpose'] ?? null, 'tx_country_outgoing' => $row['tx_country_outgoing'] ?? null, 'tx_country_incoming' => $row['tx_country_incoming'] ?? null, 'source_file' => $fileName, 'raw_payload' => json_encode($row), 'status' => Transaction::STATUS_PENDING, ]); } }); return $transactions; } /** * Starte KI-Verarbeitung für alle Transaktionen */ private function queueAiProcessing(array $transactions): void { foreach ($transactions as $transaction) { ProcessTransactionWithAI::dispatch($transaction); } } } ``` --- ## Phase 3: KI Workflow Integration (Tag 5-7) ### 3.1 AI Processing Job **app/Jobs/ProcessTransactionWithAI.php**: ```php transaction->update(['status' => Transaction::STATUS_PROCESSING]); try { // 1. Company Info Enrichment $companyInfo = $aiService->enrichCompanyInfo($this->transaction); $this->storeOutput('company_info', $companyInfo); // 2. Risk Assessment $riskAssessment = $aiService->assessRisk($this->transaction); $this->storeOutput('risk_assessment', $riskAssessment); // 3. Sanctions Check $sanctionsCheck = $aiService->checkSanctions($this->transaction); $this->storeOutput('sanctions', $sanctionsCheck); // 4. PEP Check $pepCheck = $aiService->checkPep($this->transaction); $this->storeOutput('pep', $pepCheck); // 5. Weitere Checks (conditional) if ($riskAssessment['score'] >= 50) { $registryData = $aiService->checkRegistry($this->transaction); $this->storeOutput('registry', $registryData); } // Fertig $this->transaction->update(['status' => Transaction::STATUS_COMPLETED]); } catch (\Exception $e) { $this->transaction->update([ 'status' => Transaction::STATUS_FAILED, ]); // Store Error Output $this->storeOutput('error', [ 'message' => $e->getMessage(), 'trace' => $e->getTraceAsString(), ]); throw $e; } } private function storeOutput(string $key, array $data): void { TransactionOutput::create([ 'transaction_id' => $this->transaction->id, 'prompt_id' => 1, // TODO: Map zu echtem Prompt Template 'output_key' => $key, 'content' => $data, 'run_id' => null, // TODO: Track Batch-Runs ]); } } ``` ### 3.2 AI Workflow Service (Stub) **app/Services/AiWorkflowService.php**: ```php $transaction->corporate_entity, 'legal_name' => null, 'country' => $transaction->tx_country_outgoing, 'sector' => 'Unknown', 'kyc_risk_level' => 'medium', 'enriched_at' => now()->toIso8601String(), ]; } /** * Risk Assessment */ public function assessRisk(Transaction $transaction): array { // TODO: Echte Risk-Logik via KI $amount = (float) $transaction->tx_amount; $score = 0; // Einfache Heuristik if ($amount > 100000) { $score += 30; } if ($transaction->tx_country_incoming !== 'DE') { $score += 20; } return [ 'score' => $score, 'level' => $score >= 70 ? 'high' : ($score >= 40 ? 'medium' : 'low'), 'requires_review' => $score >= 70, 'factors' => [ 'high_amount' => $amount > 100000, 'foreign_country' => $transaction->tx_country_incoming !== 'DE', ], 'assessed_at' => now()->toIso8601String(), ]; } /** * Sanctions Check */ public function checkSanctions(Transaction $transaction): array { // TODO: API zu Sanktionslisten return [ 'checked' => true, 'found' => false, 'matches' => [], 'sources' => ['EU', 'OFAC'], 'checked_at' => now()->toIso8601String(), ]; } /** * PEP Check */ public function checkPep(Transaction $transaction): array { // TODO: PEP Database Check return [ 'is_pep' => false, 'confidence' => 0.0, 'matches' => [], 'checked_at' => now()->toIso8601String(), ]; } /** * Registry Check */ public function checkRegistry(Transaction $transaction): array { // TODO: Handelsregister API return [ 'found' => false, 'company_number' => null, 'match_score' => 0.0, 'data' => null, 'checked_at' => now()->toIso8601String(), ]; } } ``` --- ## Phase 4: Frontend Anpassung (Tag 8-10) ### 4.1 Transaction Repository **app/Repositories/TransactionRepository.php**: ```php orderBy('created_at', 'desc') ->get(); } /** * Transaktionen die Review benötigen */ public function requiresReview(): Collection { return Transaction::with('outputs') ->where('status', Transaction::STATUS_COMPLETED) ->get() ->filter(fn($t) => $t->requiresReview()); } /** * High-Risk Transaktionen */ public function highRisk(int $threshold = 70): Collection { return $this->allWithOutputs() ->filter(function ($transaction) use ($threshold) { $risk = $transaction->getRiskAssessment(); return ($risk['score'] ?? 0) >= $threshold; }); } /** * Transaktionen nach Firma */ public function byCompany(string $companyName): Collection { return Transaction::with('outputs') ->where('corporate_entity', $companyName) ->orderBy('tx_date', 'desc') ->get(); } /** * Statistiken */ public function stats(): array { $total = Transaction::count(); $pending = Transaction::where('status', Transaction::STATUS_PENDING)->count(); $processing = Transaction::where('status', Transaction::STATUS_PROCESSING)->count(); $completed = Transaction::where('status', Transaction::STATUS_COMPLETED)->count(); $failed = Transaction::where('status', Transaction::STATUS_FAILED)->count(); return compact('total', 'pending', 'processing', 'completed', 'failed'); } } ``` ### 4.2 Transaction Controller **app/Http/Controllers/TransactionController.php**: ```php transactions->allWithOutputs(); $stats = $this->transactions->stats(); return view('transactions.index', compact('transactions', 'stats')); } public function show(int $id) { $transaction = $this->transactions->find($id); if (!$transaction) { abort(404); } return view('transactions.show', compact('transaction')); } public function review() { $transactions = $this->transactions->requiresReview(); return view('transactions.review', compact('transactions')); } } ``` ### 4.3 Volt Component für Transaction List **resources/views/pages/transactions/index.blade.php**: ```php $transactionRepo->allWithOutputs()); $stats = computed(fn() => $transactionRepo->stats()); ?>
{{ json_encode($output->content, JSON_PRETTY_PRINT) }}