Migration & Datapool created with job and scheduler backend->frontend(public) schema

SCHEDULING_SETUP.md

@@ -0,0 +1,445 @@
+# Backend Data Pool Sync - Scheduling Setup
+
+## Übersicht
+
+Der Backend Data Pool Sync wurde mit automatischem Scheduling konfiguriert:
+
+### 🔄 Incremental Sync
+- **Frequenz:** Alle 6 Stunden
+- **Zweck:** Synchronisiert nur neue oder geänderte Transaktionen
+- **Max. Laufzeit:** 30 Minuten
+- **Batch Size:** 1000 Records
+
+### 🔃 Full Sync
+- **Frequenz:** Jeden Sonntag um 3:00 Uhr morgens
+- **Zweck:** Kompletter Rebuild des Data Pools (Truncate + Rebuild)
+- **Max. Laufzeit:** 60 Minuten
+- **Batch Size:** 1000 Records
+
+---
+
+## Setup & Aktivierung
+
+### Option 1: Laravel Scheduler (Empfohlen für Produktion)
+
+Der Laravel Scheduler benötigt einen Cron Job, der jede Minute läuft.
+
+#### 1. Cron Job einrichten
+
+Öffne die Crontab:
+```bash
+crontab -e
+```
+
+Füge folgende Zeile hinzu:
+```cron
+* * * * * cd /Users/sebastianfrohlich/Herd/frontend && ~/Library/Application\ Support/Herd/bin/php artisan schedule:run >> /dev/null 2>&1
+```
+
+**Oder** für besseres Logging:
+```cron
+* * * * * cd /Users/sebastianfrohlich/Herd/frontend && ~/Library/Application\ Support/Herd/bin/php artisan schedule:run >> /Users/sebastianfrohlich/Herd/frontend/storage/logs/scheduler.log 2>&1
+```
+
+#### 2. Cron Job verifizieren
+
+```bash
+# Prüfe ob Cron Job aktiv ist
+crontab -l
+
+# Teste den Schedule manuell
+cd /Users/sebastianfrohlich/Herd/frontend
+~/Library/Application\ Support/Herd/bin/php artisan schedule:run
+```
+
+#### 3. Schedule List prüfen
+
+```bash
+php artisan schedule:list
+```
+
+**Expected Output:**
+```
+┌─────────────────────────────────────────────┬─────────────┬─────────────────────────────────┬───────────────┐
+│ Command                                     │ Interval    │ Description                     │ Next Due      │
+├─────────────────────────────────────────────┼─────────────┼─────────────────────────────────┼───────────────┤
+│ App\Jobs\SyncBackendDataPool               │ 0 */6 * * * │ Sync new/updated backend trans… │ in 5 hours    │
+│ App\Jobs\SyncBackendDataPool               │ 0 3 * * 0   │ Full sync and rebuild of backe… │ in 6 days     │
+└─────────────────────────────────────────────┴─────────────┴─────────────────────────────────┴───────────────┘
+```
+
+---
+
+### Option 2: Schedule Worker (Empfohlen für Development)
+
+Für lokales Development kannst du den Schedule Worker nutzen:
+
+```bash
+php artisan schedule:work
+```
+
+Dieser Befehl läuft dauerhaft und führt die Schedules automatisch aus.
+
+**Vorteile:**
+- ✅ Kein Cron Job notwendig
+- ✅ Echtzeit-Ausgabe in der Console
+- ✅ Einfaches Debugging
+
+**Nachteile:**
+- ❌ Muss manuell gestartet werden
+- ❌ Stoppt wenn Terminal geschlossen wird
+
+**Lösung:** Nutze einen Process Manager wie Supervisor oder Screen:
+
+```bash
+# Mit screen
+screen -S scheduler
+php artisan schedule:work
+# Ctrl+A, dann D zum Detachen
+
+# Später wieder attachen
+screen -r scheduler
+```
+
+---
+
+## Monitoring & Logs
+
+### 1. Laravel Logs prüfen
+
+```bash
+# Live-Logs anzeigen
+tail -f storage/logs/laravel.log
+
+# Nur Sync-Logs filtern
+tail -f storage/logs/laravel.log | grep "SyncBackendDataPool"
+
+# Scheduler-Logs (falls Cron Log aktiviert)
+tail -f storage/logs/scheduler.log
+```
+
+### 2. Schedule-Status prüfen
+
+```bash
+# Nächste geplante Ausführungen
+php artisan schedule:list
+
+# Alle Schedules testen (ohne Ausführung)
+php artisan schedule:test
+```
+
+### 3. Manuelle Sync-Ausführung
+
+```bash
+# Incremental Sync manuell ausführen
+php artisan backend:sync-data-pool --incremental
+
+# Full Sync manuell ausführen
+php artisan backend:sync-data-pool --full
+
+# Mit Queue (asynchron)
+php artisan backend:sync-data-pool --incremental --queue
+```
+
+### 4. Queue Worker (falls Jobs in Queue laufen)
+
+Wenn du `--queue` nutzt, muss ein Queue Worker laufen:
+
+```bash
+# Queue Worker starten
+php artisan queue:work --queue=default --tries=3
+
+# Oder mit Supervisor für Produktion
+```
+
+---
+
+## Zeitplan Übersicht
+
+### Incremental Sync (alle 6 Stunden)
+
+| Zeit      | Aktion           |
+|-----------|------------------|
+| 00:00 Uhr | Incremental Sync |
+| 06:00 Uhr | Incremental Sync |
+| 12:00 Uhr | Incremental Sync |
+| 18:00 Uhr | Incremental Sync |
+
+### Full Sync (Sonntags)
+
+| Tag     | Zeit      | Aktion    |
+|---------|-----------|-----------|
+| Sonntag | 03:00 Uhr | Full Sync |
+
+**Wichtig:** Am Sonntagmorgen um 3 Uhr läuft nur der Full Sync (nicht zusätzlich Incremental).
+
+---
+
+## Anpassungen
+
+### Schedule-Zeiten ändern
+
+Editiere [routes/console.php](routes/console.php):
+
+#### Beispiele für andere Frequenzen:
+
+```php
+// Incremental Sync: Stündlich
+Schedule::job(new SyncBackendDataPool(fullSync: false))
+    ->hourly();
+
+// Incremental Sync: Alle 2 Stunden
+Schedule::job(new SyncBackendDataPool(fullSync: false))
+    ->everyTwoHours();
+
+// Incremental Sync: Täglich um 2 Uhr
+Schedule::job(new SyncBackendDataPool(fullSync: false))
+    ->dailyAt('02:00');
+
+// Full Sync: Täglich um 3 Uhr
+Schedule::job(new SyncBackendDataPool(fullSync: true))
+    ->dailyAt('03:00');
+
+// Full Sync: Monatlich am 1. um 4 Uhr
+Schedule::job(new SyncBackendDataPool(fullSync: true))
+    ->monthlyOn(1, '04:00');
+```
+
+### Batch Size ändern
+
+```php
+// Kleinere Batches für weniger Speicherverbrauch
+Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 500))
+    ->everySixHours();
+
+// Größere Batches für schnellere Verarbeitung
+Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 5000))
+    ->everySixHours();
+```
+
+### Overlap Protection anpassen
+
+```php
+// Längere Lock-Zeit (z.B. für große Datenmengen)
+Schedule::job(new SyncBackendDataPool(fullSync: true))
+    ->weeklyOn(0, '03:00')
+    ->withoutOverlapping(maxLockTime: 7200); // 2 Stunden
+```
+
+---
+
+## Troubleshooting
+
+### Problem: "No scheduled commands are ready to run"
+
+**Ursache:** Es ist noch nicht Zeit für die nächste Ausführung.
+
+**Lösung:** Prüfe `php artisan schedule:list` für nächste Ausführungszeit.
+
+---
+
+### Problem: Schedule läuft nicht
+
+**Ursache:** Cron Job nicht aktiv oder falsch konfiguriert.
+
+**Lösung:**
+```bash
+# Prüfe Cron Job
+crontab -l
+
+# Teste Schedule manuell
+php artisan schedule:run
+
+# Prüfe Logs
+tail -f storage/logs/laravel.log
+```
+
+---
+
+### Problem: "Class SyncBackendDataPool not found"
+
+**Ursache:** Autoload-Cache ist veraltet.
+
+**Lösung:**
+```bash
+composer dump-autoload
+php artisan optimize:clear
+```
+
+---
+
+### Problem: Jobs laufen mehrfach parallel
+
+**Ursache:** `withoutOverlapping()` funktioniert nicht.
+
+**Lösung:** Stelle sicher, dass ein Cache-Driver konfiguriert ist:
+```bash
+# In .env
+CACHE_STORE=database
+```
+
+Dann Cache-Tabellen migrieren:
+```bash
+php artisan cache:table
+php artisan migrate
+```
+
+---
+
+### Problem: Sync dauert zu lange
+
+**Lösungen:**
+
+1. **Batch Size erhöhen:**
+```php
+Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 5000))
+```
+
+2. **Queue nutzen:**
+```php
+Schedule::job(new SyncBackendDataPool(fullSync: false))
+    ->everySixHours()
+    ->runInBackground(); // Bereits aktiviert
+```
+
+3. **Max Lock Time erhöhen:**
+```php
+->withoutOverlapping(maxLockTime: 3600) // 1 Stunde
+```
+
+---
+
+## Notifications (Optional)
+
+Du kannst Notifications hinzufügen um bei Erfolg/Fehler benachrichtigt zu werden:
+
+### Slack Notification
+
+```php
+use Illuminate\Support\Facades\Notification;
+use App\Notifications\SyncCompletedNotification;
+use App\Notifications\SyncFailedNotification;
+
+Schedule::job(new SyncBackendDataPool(fullSync: true))
+    ->weeklyOn(0, '03:00')
+    ->onSuccess(function () {
+        // Notification::route('slack', env('SLACK_WEBHOOK'))
+        //     ->notify(new SyncCompletedNotification());
+    })
+    ->onFailure(function () {
+        // Notification::route('slack', env('SLACK_WEBHOOK'))
+        //     ->notify(new SyncFailedNotification());
+    });
+```
+
+### Email Notification
+
+```php
+->onSuccess(function () {
+    Mail::to('admin@example.com')
+        ->send(new SyncCompletedMail());
+})
+->onFailure(function () {
+    Mail::to('admin@example.com')
+        ->send(new SyncFailedMail());
+});
+```
+
+---
+
+## Performance Monitoring
+
+### Database Queries überwachen
+
+```sql
+-- Anzahl Syncs heute
+SELECT DATE(synced_at) as date, COUNT(*) as syncs
+FROM public.backend_data_pool
+WHERE synced_at >= CURRENT_DATE
+GROUP BY DATE(synced_at);
+
+-- Letzte Sync-Zeiten
+SELECT MAX(synced_at) as last_sync,
+       MIN(synced_at) as first_sync,
+       COUNT(*) as total_records
+FROM public.backend_data_pool;
+
+-- Sync-Performance (Records pro Minute)
+SELECT
+    DATE_TRUNC('minute', synced_at) as minute,
+    COUNT(*) as records_synced
+FROM public.backend_data_pool
+WHERE synced_at >= NOW() - INTERVAL '1 hour'
+GROUP BY DATE_TRUNC('minute', synced_at)
+ORDER BY minute DESC;
+```
+
+### Laravel Telescope (Optional)
+
+Installiere Telescope für besseres Monitoring:
+
+```bash
+composer require laravel/telescope --dev
+php artisan telescope:install
+php artisan migrate
+```
+
+---
+
+## Produktion Deployment
+
+### Supervisor Configuration (Empfohlen)
+
+Erstelle `/etc/supervisor/conf.d/laravel-scheduler.conf`:
+
+```ini
+[program:laravel-scheduler]
+process_name=%(program_name)s
+command=php /Users/sebastianfrohlich/Herd/frontend/artisan schedule:work
+autostart=true
+autorestart=true
+user=sebastianfrohlich
+redirect_stderr=true
+stdout_logfile=/Users/sebastianfrohlich/Herd/frontend/storage/logs/scheduler.log
+stopwaitsecs=3600
+```
+
+Dann:
+```bash
+sudo supervisorctl reread
+sudo supervisorctl update
+sudo supervisorctl start laravel-scheduler
+```
+
+---
+
+## Testing
+
+### Schedule testen (ohne Ausführung)
+
+```bash
+php artisan schedule:test
+```
+
+### Nächste Ausführung simulieren
+
+```bash
+# Teste Incremental Sync
+php artisan backend:sync-data-pool --incremental --no-interaction
+
+# Teste Full Sync
+php artisan backend:sync-data-pool --full --no-interaction
+```
+
+---
+
+## Zusammenfassung
+
+✅ **Scheduling konfiguriert** in [routes/console.php](routes/console.php)
+✅ **Incremental Sync:** Alle 6 Stunden
+✅ **Full Sync:** Sonntags 3:00 Uhr
+✅ **Overlap Protection:** Aktiviert
+✅ **Background Execution:** Aktiviert
+✅ **Success/Failure Callbacks:** Implementiert
+
+**Nächster Schritt:** Cron Job einrichten oder `php artisan schedule:work` starten!

SYNC_MIGRATION_INSTRUCTIONS.md

@@ -0,0 +1,305 @@
+# Backend Data Pool Sync - Migration & Testing Instructions
+
+## Übersicht
+
+Dieser Guide führt dich durch das Testen und Ausführen der Backend Data Pool Migration.
+
+## Schritt 1: Migration ausführen
+
+```bash
+php artisan migrate
+```
+
+**Expected Output:**
+```
+Migrating: 2025_11_16_150000_create_backend_data_pool_table
+Migrated:  2025_11_16_150000_create_backend_data_pool_table
+```
+
+### Bei Problemen:
+
+Wenn die Migration fehlschlägt, prüfe:
+
+```bash
+# Datenbankverbindung testen
+php artisan db:show
+
+# Oder spezifisch für pgsql
+php artisan db:show --database=pgsql
+```
+
+## Schritt 2: Struktur der Tabelle prüfen
+
+```sql
+-- In psql oder einem DB-Tool
+\d public.backend_data_pool
+
+-- Oder mit artisan tinker
+php artisan tinker
+>>> DB::select("SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'backend_data_pool' ORDER BY ordinal_position");
+```
+
+## Schritt 3: Statistiken vor dem Sync prüfen
+
+```bash
+php artisan backend:sync-data-pool --stats
+```
+
+**Expected Output:**
+```
+┌─────────────────────────────────┬────────┐
+│ Metric                          │ Value  │
+├─────────────────────────────────┼────────┤
+│ Backend Transactions (done)     │ XXX    │
+│ Backend Transaction Outputs     │ XXX    │
+│ Data Pool Records               │ 0      │
+│ Last Synced At                  │ Never  │
+└─────────────────────────────────┴────────┘
+```
+
+## Schritt 4: Ersten Full Sync durchführen
+
+```bash
+php artisan backend:sync-data-pool --full
+```
+
+**Expected Output:**
+```
+Preparing for FULL SYNC
+
+⚠ This will truncate and rebuild the entire backend_data_pool table.
+
+┌ Do you want to continue? ─────────────────────────────┐
+│ Yes / No                                              │
+└───────────────────────────────────────────────────────┘
+
+Stats BEFORE sync:
+┌─────────────────────────────────┬────────┐
+│ Metric                          │ Value  │
+├─────────────────────────────────┼────────┤
+│ Backend Transactions (done)     │ XXX    │
+│ Data Pool Records               │ 0      │
+│ Last Synced At                  │ Never  │
+└─────────────────────────────────┴────────┘
+
+⠙ Syncing data...
+
+✔ Sync completed successfully!
+
+Stats AFTER sync:
+┌─────────────────────────────────┬────────┐
+│ Metric                          │ Value  │
+├─────────────────────────────────┼────────┤
+│ Backend Transactions (done)     │ XXX    │
+│ Data Pool Records               │ XXX    │
+│ Last Synced At                  │ 2025-11-16 15:30:45 │
+└─────────────────────────────────┴────────┘
+```
+
+## Schritt 5: Daten verifizieren
+
+### SQL Query zum Vergleichen:
+
+```sql
+-- Anzahl Datensätze in backend.transactions mit status='done'
+SELECT COUNT(*)
+FROM backend.transactions
+WHERE status = 'done';
+
+-- Anzahl Datensätze in backend.transaction_outputs (für done transactions)
+SELECT COUNT(*)
+FROM backend.transaction_outputs tout
+JOIN backend.transactions t ON t.id = tout.transaction_id
+WHERE t.status = 'done';
+
+-- Anzahl Datensätze in public.backend_data_pool
+SELECT COUNT(*)
+FROM public.backend_data_pool;
+
+-- Die zweite und dritte Zahl sollten gleich sein!
+```
+
+### Beispiel-Datensätze prüfen:
+
+```sql
+-- Ersten 5 Datensätze anzeigen
+SELECT
+    transaction_id,
+    corporate_entity,
+    tx_amount,
+    output_key,
+    LEFT(content, 50) as content_preview,
+    synced_at
+FROM public.backend_data_pool
+ORDER BY transaction_id, prompt_id
+LIMIT 5;
+```
+
+### Prüfe ob alle output_keys vorhanden sind:
+
+```sql
+-- Welche output_keys sind im Data Pool?
+SELECT
+    output_key,
+    COUNT(*) as count
+FROM public.backend_data_pool
+GROUP BY output_key
+ORDER BY count DESC;
+```
+
+## Schritt 6: Tests ausführen
+
+```bash
+php artisan test --filter=SyncBackendDataPool
+```
+
+**Expected Output:**
+```
+PASS  Tests\Feature\Jobs\SyncBackendDataPoolTest
+✓ full sync truncates and rebuilds backend_data_pool
+✓ incremental sync only adds new records
+✓ sync correctly maps backend.transactions and backend.transaction_outputs
+✓ sync only includes transactions with status done
+✓ sync processes records in batches
+✓ getStats returns correct statistics
+✓ sync handles empty backend gracefully
+✓ sync orders records by transaction_id and prompt_id
+
+Tests:    8 passed (X assertions)
+Duration: XXs
+```
+
+## Schritt 7: Incremental Sync testen
+
+```bash
+# Warte kurz oder füge neue Daten im backend hinzu
+# Dann führe incremental sync aus:
+php artisan backend:sync-data-pool --incremental
+```
+
+## Troubleshooting
+
+### Problem: "SQLSTATE[42P01]: Undefined table"
+
+**Lösung:** Migration wurde nicht ausgeführt oder fehlgeschlagen.
+```bash
+php artisan migrate:status
+php artisan migrate --force
+```
+
+### Problem: "Connection refused" oder Connection Error
+
+**Lösung:** Prüfe .env Datei:
+```bash
+DB_CONNECTION2=pgsql
+DB_HOST2=127.0.0.1
+DB_PORT2=5433
+DB_DATABASE2=risk_ingest_db
+DB_USERNAME2=risk_ingest_user
+DB_PASSWORD2=S0prast3r1a
+```
+
+Teste Verbindung:
+```bash
+psql -h 127.0.0.1 -p 5433 -U risk_ingest_user -d risk_ingest_db
+```
+
+### Problem: "schema backend does not exist"
+
+**Lösung:** Backend Schema fehlt. Erstelle es:
+```sql
+CREATE SCHEMA IF NOT EXISTS backend;
+```
+
+### Problem: Job läuft sehr lange
+
+**Lösung:** Nutze kleinere Batch Size oder Queue:
+```bash
+# Kleinere Batches
+php artisan backend:sync-data-pool --full --batch-size=100
+
+# Oder mit Queue (asynchron)
+php artisan backend:sync-data-pool --full --queue
+```
+
+## Performance Optimization
+
+### Für große Datenmengen:
+
+1. **Nutze Queue:**
+```bash
+php artisan backend:sync-data-pool --full --queue
+php artisan queue:work
+```
+
+2. **Optimiere Batch Size:**
+```bash
+# Für viele Datensätze
+php artisan backend:sync-data-pool --full --batch-size=5000
+
+# Für wenig RAM
+php artisan backend:sync-data-pool --full --batch-size=100
+```
+
+3. **Schedule für regelmäßige Syncs:**
+
+Füge in `routes/console.php` hinzu:
+```php
+use App\Jobs\SyncBackendDataPool;
+use Illuminate\Support\Facades\Schedule;
+
+// Täglich um 2 Uhr incremental
+Schedule::job(new SyncBackendDataPool(fullSync: false))
+    ->dailyAt('02:00');
+
+// Sonntags um 3 Uhr full sync
+Schedule::job(new SyncBackendDataPool(fullSync: true))
+    ->weeklyOn(0, '03:00');
+```
+
+Dann starte den Scheduler:
+```bash
+php artisan schedule:work
+```
+
+## Monitoring
+
+### Logs prüfen:
+
+```bash
+tail -f storage/logs/laravel.log
+```
+
+### SQL Queries für Monitoring:
+
+```sql
+-- Letzte Sync Zeit
+SELECT MAX(synced_at) FROM public.backend_data_pool;
+
+-- Anzahl Datensätze pro Status
+SELECT status, COUNT(*)
+FROM public.backend_data_pool
+GROUP BY status;
+
+-- Anzahl Datensätze pro corporate_entity (Top 10)
+SELECT corporate_entity, COUNT(*) as transaction_count
+FROM public.backend_data_pool
+GROUP BY corporate_entity
+ORDER BY transaction_count DESC
+LIMIT 10;
+
+-- Durchschnittlicher Transaktionsbetrag
+SELECT AVG(tx_amount) as avg_amount
+FROM public.backend_data_pool;
+```
+
+## Nächste Schritte
+
+Nach erfolgreichem Sync kannst du mit **Stufe 2** weitermachen:
+- Transformation der Daten in `public.companies` und `public.transactions`
+- Mit KYC Risk Level Berechnung
+- Automatisches Enrichment der Companies
+
+---
+
+**Bei Fragen oder Problemen, prüfe die Logs in `storage/logs/laravel.log`**

app/Console/Commands/SyncBackendDataPoolCommand.php

@@ -0,0 +1,156 @@
+<?php
+
+declare(strict_types=1);
+
+namespace App\Console\Commands;
+
+use App\Jobs\SyncBackendDataPool;
+use Illuminate\Console\Command;
+
+use function Laravel\Prompts\confirm;
+use function Laravel\Prompts\info;
+use function Laravel\Prompts\spin;
+use function Laravel\Prompts\table;
+use function Laravel\Prompts\warning;
+
+class SyncBackendDataPoolCommand extends Command
+{
+    /**
+     * The name and signature of the console command.
+     */
+    protected $signature = 'backend:sync-data-pool
+                            {--full : Perform a full sync (truncate and rebuild)}
+                            {--incremental : Perform an incremental sync (only new/updated records)}
+                            {--batch-size=1000 : Number of records to process per batch}
+                            {--queue : Dispatch the job to the queue instead of running synchronously}
+                            {--stats : Show statistics only, do not perform sync}';
+
+    /**
+     * The console command description.
+     */
+    protected $description = 'Sync backend.transactions + backend.transaction_outputs to public.backend_data_pool';
+
+    /**
+     * Execute the console command.
+     */
+    public function handle(): int
+    {
+        // Show stats only
+        if ($this->option('stats')) {
+            return $this->showStats();
+        }
+
+        // Determine sync type
+        $fullSync = $this->option('full') ?? ! $this->option('incremental');
+
+        if ($fullSync) {
+            info('Preparing for FULL SYNC');
+            warning('This will truncate and rebuild the entire backend_data_pool table.');
+
+            if (! $this->option('no-interaction') && ! confirm('Do you want to continue?', true)) {
+                info('Sync cancelled.');
+
+                return self::SUCCESS;
+            }
+        } else {
+            info('Preparing for INCREMENTAL SYNC');
+        }
+
+        $batchSize = (int) $this->option('batch-size');
+
+        // Show current stats before sync
+        $this->showStatsBefore();
+
+        // Create the job
+        $job = new SyncBackendDataPool(
+            fullSync: $fullSync,
+            batchSize: $batchSize
+        );
+
+        // Execute the job
+        if ($this->option('queue')) {
+            info('Dispatching job to queue...');
+            dispatch($job);
+            info('Job dispatched successfully!');
+        } else {
+            info('Running sync synchronously...');
+
+            spin(
+                fn () => $job->handle(),
+                'Syncing data...'
+            );
+
+            info('Sync completed successfully!');
+        }
+
+        // Show stats after sync
+        $this->showStatsAfter();
+
+        return self::SUCCESS;
+    }
+
+    /**
+     * Show statistics only
+     */
+    private function showStats(): int
+    {
+        $job = new SyncBackendDataPool();
+        $stats = $job->getStats();
+
+        info('Backend Data Pool Statistics');
+
+        table(
+            ['Metric', 'Value'],
+            [
+                ['Backend Transactions (done)', number_format($stats['backend_transactions_done'])],
+                ['Backend Transaction Outputs', number_format($stats['backend_transaction_outputs'])],
+                ['Data Pool Records', number_format($stats['backend_data_pool_records'])],
+                ['Last Synced At', $stats['last_synced_at'] ?? 'Never'],
+            ]
+        );
+
+        return self::SUCCESS;
+    }
+
+    /**
+     * Show stats before sync
+     */
+    private function showStatsBefore(): void
+    {
+        $job = new SyncBackendDataPool();
+        $stats = $job->getStats();
+
+        $this->newLine();
+        info('Stats BEFORE sync:');
+        table(
+            ['Metric', 'Value'],
+            [
+                ['Backend Transactions (done)', number_format($stats['backend_transactions_done'])],
+                ['Data Pool Records', number_format($stats['backend_data_pool_records'])],
+                ['Last Synced At', $stats['last_synced_at'] ?? 'Never'],
+            ]
+        );
+        $this->newLine();
+    }
+
+    /**
+     * Show stats after sync
+     */
+    private function showStatsAfter(): void
+    {
+        $job = new SyncBackendDataPool();
+        $stats = $job->getStats();
+
+        $this->newLine();
+        info('Stats AFTER sync:');
+        table(
+            ['Metric', 'Value'],
+            [
+                ['Backend Transactions (done)', number_format($stats['backend_transactions_done'])],
+                ['Data Pool Records', number_format($stats['backend_data_pool_records'])],
+                ['Last Synced At', $stats['last_synced_at']],
+            ]
+        );
+        $this->newLine();
+    }
+}

app/Jobs/SyncBackendDataPool.php

@@ -0,0 +1,228 @@
+<?php
+
+declare(strict_types=1);
+
+namespace App\Jobs;
+
+use Illuminate\Bus\Queueable;
+use Illuminate\Contracts\Queue\ShouldQueue;
+use Illuminate\Foundation\Bus\Dispatchable;
+use Illuminate\Queue\InteractsWithQueue;
+use Illuminate\Queue\SerializesModels;
+use Illuminate\Support\Facades\DB;
+use Illuminate\Support\Facades\Log;
+
+class SyncBackendDataPool implements ShouldQueue
+{
+    use Dispatchable;
+    use InteractsWithQueue;
+    use Queueable;
+    use SerializesModels;
+
+    public int $timeout = 3600; // 1 hour timeout
+
+    public int $tries = 3;
+
+    /**
+     * Create a new job instance.
+     */
+    public function __construct(
+        public bool $fullSync = true,
+        public ?int $batchSize = 1000,
+    ) {
+    }
+
+    /**
+     * Execute the job.
+     */
+    public function handle(): void
+    {
+        $startTime = now();
+
+        Log::info('SyncBackendDataPool: Starting sync', [
+            'full_sync' => $this->fullSync,
+            'batch_size' => $this->batchSize,
+        ]);
+
+        try {
+            if ($this->fullSync) {
+                $this->performFullSync();
+            } else {
+                $this->performIncrementalSync();
+            }
+
+            $duration = now()->diffInSeconds($startTime);
+
+            Log::info('SyncBackendDataPool: Sync completed successfully', [
+                'duration_seconds' => $duration,
+            ]);
+        } catch (\Exception $e) {
+            Log::error('SyncBackendDataPool: Sync failed', [
+                'error' => $e->getMessage(),
+                'trace' => $e->getTraceAsString(),
+            ]);
+
+            throw $e;
+        }
+    }
+
+    /**
+     * Perform full synchronization - truncates and rebuilds the data pool
+     */
+    private function performFullSync(): void
+    {
+        DB::transaction(function () {
+            // Truncate existing data pool
+            DB::table('backend_data_pool')->truncate();
+
+            Log::info('SyncBackendDataPool: Truncated backend_data_pool');
+
+            // Insert all data in batches
+            $this->insertDataInBatches();
+        });
+    }
+
+    /**
+     * Perform incremental synchronization - only syncs new/updated records
+     */
+    private function performIncrementalSync(): void
+    {
+        DB::transaction(function () {
+            // Get the last synced timestamp
+            $lastSyncedAt = DB::table('backend_data_pool')
+                ->max('synced_at');
+
+            Log::info('SyncBackendDataPool: Incremental sync', [
+                'last_synced_at' => $lastSyncedAt,
+            ]);
+
+            // Insert new records
+            $this->insertDataInBatches($lastSyncedAt);
+        });
+    }
+
+    /**
+     * Insert data from backend schema into public.backend_data_pool in batches
+     */
+    private function insertDataInBatches(?string $lastSyncedAt = null): void
+    {
+        $totalInserted = 0;
+        $offset = 0;
+
+        while (true) {
+            // Build the query
+            $query = $this->buildSyncQuery($lastSyncedAt)
+                ->limit($this->batchSize)
+                ->offset($offset);
+
+            $records = $query->get();
+
+            if ($records->isEmpty()) {
+                break;
+            }
+
+            // Transform records for insertion
+            $dataToInsert = $records->map(function ($record) {
+                return [
+                    'transaction_id' => $record->transaction_id,
+                    'corporate_entity' => $record->corporate_entity,
+                    'corporate_counterparty' => $record->corporate_counterparty,
+                    'tx_date' => $record->tx_date,
+                    'tx_amount' => $record->tx_amount,
+                    'tx_currency' => $record->tx_currency,
+                    'tx_purpose' => $record->tx_purpose,
+                    'tx_country_outgoing' => $record->tx_country_outgoing,
+                    'tx_country_incoming' => $record->tx_country_incoming,
+                    'source_file' => $record->source_file,
+                    'raw_payload' => $record->raw_payload,
+                    'status' => $record->status,
+                    'created_at' => $record->created_at,
+                    'last_modified_at' => $record->last_modified_at,
+                    'prompt_id' => $record->prompt_id,
+                    'output_key' => $record->output_key,
+                    'content' => $record->content,
+                    'run_id' => $record->run_id,
+                    'synced_at' => now(),
+                ];
+            })->toArray();
+
+            // Insert batch
+            DB::table('backend_data_pool')->insert($dataToInsert);
+
+            $totalInserted += count($dataToInsert);
+
+            Log::info('SyncBackendDataPool: Batch inserted', [
+                'batch_size' => count($dataToInsert),
+                'total_inserted' => $totalInserted,
+            ]);
+
+            $offset += $this->batchSize;
+        }
+
+        Log::info('SyncBackendDataPool: All batches inserted', [
+            'total_records' => $totalInserted,
+        ]);
+    }
+
+    /**
+     * Build the sync query that replicates the SQL:
+     * SELECT * FROM backend.transactions t
+     * LEFT JOIN backend.transaction_outputs tout ON t.id = tout.transaction_id
+     * WHERE t.status = 'done'
+     * ORDER BY transaction_id, prompt_id
+     */
+    private function buildSyncQuery(?string $lastSyncedAt = null): \Illuminate\Database\Query\Builder
+    {
+        $query = DB::connection('pgsql')
+            ->table('backend.transactions as t')
+            ->leftJoin('backend.transaction_outputs as tout', 't.id', '=', 'tout.transaction_id')
+            ->select([
+                't.id as transaction_id',
+                't.corporate_entity',
+                't.corporate_counterparty',
+                't.tx_date',
+                't.tx_amount',
+                't.tx_currency',
+                't.tx_purpose',
+                't.tx_country_outgoing',
+                't.tx_country_incoming',
+                't.source_file',
+                't.raw_payload',
+                't.status',
+                't.created_at',
+                't.last_modified_at',
+                'tout.prompt_id',
+                'tout.output_key',
+                'tout.content',
+                'tout.run_id',
+            ])
+            ->where('t.status', '=', 'done')
+            ->orderBy('t.id')
+            ->orderBy('tout.prompt_id');
+
+        // For incremental sync, only get records modified after last sync
+        if ($lastSyncedAt !== null) {
+            $query->where('t.last_modified_at', '>', $lastSyncedAt);
+        }
+
+        return $query;
+    }
+
+    /**
+     * Get statistics about the sync
+     */
+    public function getStats(): array
+    {
+        return [
+            'backend_transactions_done' => DB::connection('pgsql')
+                ->table('backend.transactions')
+                ->where('status', 'done')
+                ->count(),
+            'backend_transaction_outputs' => DB::connection('pgsql')
+                ->table('backend.transaction_outputs')
+                ->count(),
+            'backend_data_pool_records' => DB::table('backend_data_pool')->count(),
+            'last_synced_at' => DB::table('backend_data_pool')->max('synced_at'),
+        ];
+    }
+}

database/migrations/2025_11_16_150000_create_backend_data_pool_table.php

@@ -0,0 +1,70 @@
+<?php
+
+use Illuminate\Database\Migrations\Migration;
+use Illuminate\Database\Schema\Blueprint;
+use Illuminate\Support\Facades\Schema;
+
+return new class extends Migration
+{
+    /**
+     * Run the migrations.
+     *
+     * Creates a denormalized data pool table that contains the result of:
+     * SELECT * FROM backend.transactions t
+     * LEFT JOIN backend.transaction_outputs tout ON t.id = tout.transaction_id
+     * WHERE t.status = 'done'
+     */
+    public function up(): void
+    {
+        Schema::create('backend_data_pool', function (Blueprint $table) {
+            $table->id();
+
+            // ===================================
+            // FROM backend.transactions (t.*)
+            // ===================================
+            $table->integer('transaction_id')->index();
+            $table->text('corporate_entity');
+            $table->text('corporate_counterparty');
+            $table->text('tx_date');
+            $table->double('tx_amount');
+            $table->text('tx_currency')->nullable();
+            $table->text('tx_purpose')->nullable();
+            $table->text('tx_country_outgoing')->nullable();
+            $table->text('tx_country_incoming')->nullable();
+            $table->text('source_file')->nullable();
+            $table->text('raw_payload')->nullable();
+            $table->text('status')->index();
+            $table->text('created_at');
+            $table->text('last_modified_at');
+
+            // ===================================
+            // FROM backend.transaction_outputs (tout.*)
+            // ===================================
+            $table->integer('prompt_id');
+            $table->text('output_key')->index();
+            $table->text('content');
+            $table->integer('run_id')->nullable();
+
+            // ===================================
+            // Metadata for tracking sync
+            // ===================================
+            $table->timestamp('synced_at')->useCurrent();
+            $table->timestamp('updated_at')->nullable();
+
+            // ===================================
+            // Composite Indexes for Performance
+            // ===================================
+            $table->index(['transaction_id', 'output_key']);
+            $table->index(['transaction_id', 'prompt_id']);
+            $table->index('synced_at');
+        });
+    }
+
+    /**
+     * Reverse the migrations.
+     */
+    public function down(): void
+    {
+        Schema::dropIfExists('backend_data_pool');
+    }
+};

routes/console.php

@@ -1,8 +1,32 @@
 <?php
 
+use App\Jobs\SyncBackendDataPool;
 use Illuminate\Foundation\Inspiring;
 use Illuminate\Support\Facades\Artisan;
+use Illuminate\Support\Facades\Log;
+use Illuminate\Support\Facades\Schedule;
 
 Artisan::command('inspire', function () {
     $this->comment(Inspiring::quote());
 })->purpose('Display an inspiring quote');
+
+/**
+ * Backend Data Pool Sync Scheduling
+ *
+ * - Incremental Sync: Läuft alle 6 Stunden (nur neue/geänderte Daten)
+ * - Full Sync: Läuft jeden Sonntag um 3 Uhr morgens (kompletter Rebuild)
+ */
+
+// Incremental Sync: Alle 6 Stunden
+Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 1000))
+    ->everySixHours()
+    ->name('sync-backend-data-pool-incremental')
+    ->description('Sync new/updated backend transactions to data pool')
+    ->withoutOverlapping(1800); // Max 30 Minuten Lock (1800 Sekunden)
+
+// Full Sync: Jeden Sonntag um 3 Uhr morgens
+Schedule::job(new SyncBackendDataPool(fullSync: true, batchSize: 1000))
+    ->weeklyOn(0, '03:00') // Sonntag, 3:00 Uhr
+    ->name('sync-backend-data-pool-full')
+    ->description('Full sync and rebuild of backend data pool')
+    ->withoutOverlapping(3600); // Max 1 Stunde Lock (3600 Sekunden)

test_backend_query.sql

@@ -0,0 +1,76 @@
+-- Test Query: Simuliert den Sync Query
+-- Führe diese Query aus um zu sehen, welche Daten synchronisiert werden
+
+-- 1. Zeige Anzahl der Transaktionen mit status='done'
+SELECT
+    'backend.transactions (done)' as table_name,
+    COUNT(*) as record_count
+FROM backend.transactions
+WHERE status = 'done'
+
+UNION ALL
+
+-- 2. Zeige Anzahl der transaction_outputs für 'done' Transaktionen
+SELECT
+    'backend.transaction_outputs (for done transactions)' as table_name,
+    COUNT(*) as record_count
+FROM backend.transaction_outputs tout
+JOIN backend.transactions t ON t.id = tout.transaction_id
+WHERE t.status = 'done'
+
+UNION ALL
+
+-- 3. Zeige JOIN Resultat (das wird synchronisiert)
+SELECT
+    'JOIN Result (will be synced)' as table_name,
+    COUNT(*) as record_count
+FROM backend.transactions t
+LEFT JOIN backend.transaction_outputs tout ON t.id = tout.transaction_id
+WHERE t.status = 'done';
+
+-- ==========================================
+
+-- 4. Zeige erste 5 Datensätze die synchronisiert werden
+SELECT
+    t.id as transaction_id,
+    t.corporate_entity,
+    t.corporate_counterparty,
+    t.tx_date,
+    t.tx_amount,
+    t.tx_currency,
+    t.status,
+    tout.prompt_id,
+    tout.output_key,
+    LEFT(tout.content, 100) as content_preview,
+    tout.run_id
+FROM backend.transactions t
+LEFT JOIN backend.transaction_outputs tout ON t.id = tout.transaction_id
+WHERE t.status = 'done'
+ORDER BY t.id, tout.prompt_id
+LIMIT 5;
+
+-- ==========================================
+
+-- 5. Zeige Verteilung der output_keys
+SELECT
+    tout.output_key,
+    COUNT(*) as count
+FROM backend.transaction_outputs tout
+JOIN backend.transactions t ON t.id = tout.transaction_id
+WHERE t.status = 'done'
+GROUP BY tout.output_key
+ORDER BY count DESC
+LIMIT 20;
+
+-- ==========================================
+
+-- 6. Zeige Transactions ohne Outputs (LEFT JOIN NULL)
+SELECT
+    t.id,
+    t.corporate_entity,
+    t.status
+FROM backend.transactions t
+LEFT JOIN backend.transaction_outputs tout ON t.id = tout.transaction_id
+WHERE t.status = 'done'
+    AND tout.transaction_id IS NULL
+LIMIT 10;

tests/Feature/Jobs/SyncBackendDataPoolTest.php

@@ -0,0 +1,170 @@
+<?php
+
+declare(strict_types=1);
+
+use App\Jobs\SyncBackendDataPool;
+use Illuminate\Support\Facades\DB;
+
+beforeEach(function () {
+    // Clean up tables before each test
+    DB::table('backend_data_pool')->truncate();
+});
+
+test('full sync truncates and rebuilds backend_data_pool', function () {
+    // Arrange: Insert some old data
+    DB::table('backend_data_pool')->insert([
+        'transaction_id' => 999,
+        'corporate_entity' => 'Old Company',
+        'corporate_counterparty' => 'Old Counterparty',
+        'tx_date' => '2024-01-01',
+        'tx_amount' => 1000.00,
+        'status' => 'done',
+        'created_at' => '2024-01-01',
+        'last_modified_at' => '2024-01-01',
+        'prompt_id' => 1,
+        'output_key' => 'test_key',
+        'content' => 'test content',
+        'synced_at' => now(),
+    ]);
+
+    expect(DB::table('backend_data_pool')->count())->toBe(1);
+
+    // Act: Perform full sync
+    $job = new SyncBackendDataPool(fullSync: true, batchSize: 100);
+    $job->handle();
+
+    // Assert: Old data should be removed, new data from backend should be present
+    $recordCount = DB::table('backend_data_pool')->count();
+    expect($recordCount)->toBeGreaterThanOrEqual(0); // Should have backend data
+
+    // Should not contain the old test record
+    $oldRecord = DB::table('backend_data_pool')
+        ->where('transaction_id', 999)
+        ->first();
+
+    expect($oldRecord)->toBeNull();
+});
+
+test('incremental sync only adds new records', function () {
+    // Arrange: Perform initial sync
+    $job = new SyncBackendDataPool(fullSync: true, batchSize: 100);
+    $job->handle();
+
+    $initialCount = DB::table('backend_data_pool')->count();
+
+    // Wait a moment to ensure timestamp difference
+    sleep(1);
+
+    // Act: Perform incremental sync (should not add duplicates)
+    $incrementalJob = new SyncBackendDataPool(fullSync: false, batchSize: 100);
+    $incrementalJob->handle();
+
+    // Assert: Count should not increase significantly (only if new records in backend)
+    $newCount = DB::table('backend_data_pool')->count();
+    expect($newCount)->toBeGreaterThanOrEqual($initialCount);
+});
+
+test('sync correctly maps backend.transactions and backend.transaction_outputs', function () {
+    // Act: Perform sync
+    $job = new SyncBackendDataPool(fullSync: true, batchSize: 100);
+    $job->handle();
+
+    // Assert: Check if records have correct structure
+    $record = DB::table('backend_data_pool')->first();
+
+    if ($record) {
+        expect($record)->toHaveProperty('transaction_id');
+        expect($record)->toHaveProperty('corporate_entity');
+        expect($record)->toHaveProperty('corporate_counterparty');
+        expect($record)->toHaveProperty('tx_date');
+        expect($record)->toHaveProperty('tx_amount');
+        expect($record)->toHaveProperty('status');
+        expect($record)->toHaveProperty('prompt_id');
+        expect($record)->toHaveProperty('output_key');
+        expect($record)->toHaveProperty('content');
+        expect($record)->toHaveProperty('synced_at');
+    }
+});
+
+test('sync only includes transactions with status done', function () {
+    // Act: Perform sync
+    $job = new SyncBackendDataPool(fullSync: true, batchSize: 100);
+    $job->handle();
+
+    // Assert: All records should have status = 'done'
+    $nonDoneRecords = DB::table('backend_data_pool')
+        ->where('status', '!=', 'done')
+        ->count();
+
+    expect($nonDoneRecords)->toBe(0);
+});
+
+test('sync processes records in batches', function () {
+    // Act: Perform sync with small batch size
+    $job = new SyncBackendDataPool(fullSync: true, batchSize: 10);
+    $job->handle();
+
+    // Assert: Should complete without errors
+    $recordCount = DB::table('backend_data_pool')->count();
+    expect($recordCount)->toBeGreaterThanOrEqual(0);
+});
+
+test('getStats returns correct statistics', function () {
+    // Arrange: Perform sync
+    $job = new SyncBackendDataPool(fullSync: true, batchSize: 100);
+    $job->handle();
+
+    // Act: Get stats
+    $stats = $job->getStats();
+
+    // Assert: Stats should have correct structure
+    expect($stats)->toBeArray();
+    expect($stats)->toHaveKey('backend_transactions_done');
+    expect($stats)->toHaveKey('backend_transaction_outputs');
+    expect($stats)->toHaveKey('backend_data_pool_records');
+    expect($stats)->toHaveKey('last_synced_at');
+
+    expect($stats['backend_data_pool_records'])->toBeGreaterThanOrEqual(0);
+});
+
+test('sync handles empty backend gracefully', function () {
+    // This test assumes backend might be empty or have no 'done' transactions
+
+    // Act: Perform sync
+    $job = new SyncBackendDataPool(fullSync: true, batchSize: 100);
+    $job->handle();
+
+    // Assert: Should complete without errors
+    $recordCount = DB::table('backend_data_pool')->count();
+    expect($recordCount)->toBeGreaterThanOrEqual(0);
+});
+
+test('sync orders records by transaction_id and prompt_id', function () {
+    // Act: Perform sync
+    $job = new SyncBackendDataPool(fullSync: true, batchSize: 100);
+    $job->handle();
+
+    // Assert: Check ordering
+    $records = DB::table('backend_data_pool')
+        ->limit(10)
+        ->get();
+
+    if ($records->count() > 1) {
+        $previousTransactionId = null;
+        $previousPromptId = null;
+
+        foreach ($records as $record) {
+            if ($previousTransactionId !== null) {
+                // Either transaction_id should increase or stay same (with prompt_id increasing)
+                expect($record->transaction_id)->toBeGreaterThanOrEqual($previousTransactionId);
+
+                if ($record->transaction_id === $previousTransactionId && $previousPromptId !== null) {
+                    expect($record->prompt_id)->toBeGreaterThanOrEqual($previousPromptId);
+                }
+            }
+
+            $previousTransactionId = $record->transaction_id;
+            $previousPromptId = $record->prompt_id;
+        }
+    }
+});