chore: major project cleanup and restructuring
## Dokumentation - Entfernt AGENTS.md (redundant zu CLAUDE.md) - Verschoben: 3 Dokumentationen von root → docs/ - Gelöscht: 8 historische Migrations-Pläne aus docs/archive/ - Entfernt: misc/ Ordner komplett (43 Dateien) ## Struktur - Erstellt: database/queries/ für SQL Test-Queries (4 Dateien) - Erstellt: database/schemas/ für Schema-Dokumentation (17 Dateien) - output_keys_mapping/ (8 CSV-Mappings) - migration-diagrams/ (4 Diagramme) ## Code-Qualität - Formatiert: 7 Style-Issues in 74 Dateien mit Laravel Pint - Gefixed: Migration für PostgreSQL/SQLite Kompatibilität - Gefixed: 2 fehlgeschlagene Tests (CSRF + Text-Assertion) ## Tests - Alle 73 Tests bestehen jetzt (100% Success Rate) - AuthenticationTest: CSRF-Token Fix für Logout-Test - CompanySearchTest: Text-Assertion aktualisiert ## Ergebnis - Root ist sauber (nur CLAUDE.md) - Dokumentation strukturiert in docs/ - Database-Dateien organisiert in database/ - Code entspricht Style-Guide - Alle Tests bestehen
This commit is contained in:
@@ -0,0 +1,445 @@
|
||||
# Backend Data Pool Sync - Scheduling Setup
|
||||
|
||||
## Übersicht
|
||||
|
||||
Der Backend Data Pool Sync wurde mit automatischem Scheduling konfiguriert:
|
||||
|
||||
### 🔄 Incremental Sync
|
||||
- **Frequenz:** Alle 6 Stunden
|
||||
- **Zweck:** Synchronisiert nur neue oder geänderte Transaktionen
|
||||
- **Max. Laufzeit:** 30 Minuten
|
||||
- **Batch Size:** 1000 Records
|
||||
|
||||
### 🔃 Full Sync
|
||||
- **Frequenz:** Jeden Sonntag um 3:00 Uhr morgens
|
||||
- **Zweck:** Kompletter Rebuild des Data Pools (Truncate + Rebuild)
|
||||
- **Max. Laufzeit:** 60 Minuten
|
||||
- **Batch Size:** 1000 Records
|
||||
|
||||
---
|
||||
|
||||
## Setup & Aktivierung
|
||||
|
||||
### Option 1: Laravel Scheduler (Empfohlen für Produktion)
|
||||
|
||||
Der Laravel Scheduler benötigt einen Cron Job, der jede Minute läuft.
|
||||
|
||||
#### 1. Cron Job einrichten
|
||||
|
||||
Öffne die Crontab:
|
||||
```bash
|
||||
crontab -e
|
||||
```
|
||||
|
||||
Füge folgende Zeile hinzu:
|
||||
```cron
|
||||
* * * * * cd /Users/sebastianfrohlich/Herd/frontend && ~/Library/Application\ Support/Herd/bin/php artisan schedule:run >> /dev/null 2>&1
|
||||
```
|
||||
|
||||
**Oder** für besseres Logging:
|
||||
```cron
|
||||
* * * * * cd /Users/sebastianfrohlich/Herd/frontend && ~/Library/Application\ Support/Herd/bin/php artisan schedule:run >> /Users/sebastianfrohlich/Herd/frontend/storage/logs/scheduler.log 2>&1
|
||||
```
|
||||
|
||||
#### 2. Cron Job verifizieren
|
||||
|
||||
```bash
|
||||
# Prüfe ob Cron Job aktiv ist
|
||||
crontab -l
|
||||
|
||||
# Teste den Schedule manuell
|
||||
cd /Users/sebastianfrohlich/Herd/frontend
|
||||
~/Library/Application\ Support/Herd/bin/php artisan schedule:run
|
||||
```
|
||||
|
||||
#### 3. Schedule List prüfen
|
||||
|
||||
```bash
|
||||
php artisan schedule:list
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
┌─────────────────────────────────────────────┬─────────────┬─────────────────────────────────┬───────────────┐
|
||||
│ Command │ Interval │ Description │ Next Due │
|
||||
├─────────────────────────────────────────────┼─────────────┼─────────────────────────────────┼───────────────┤
|
||||
│ App\Jobs\SyncBackendDataPool │ 0 */6 * * * │ Sync new/updated backend trans… │ in 5 hours │
|
||||
│ App\Jobs\SyncBackendDataPool │ 0 3 * * 0 │ Full sync and rebuild of backe… │ in 6 days │
|
||||
└─────────────────────────────────────────────┴─────────────┴─────────────────────────────────┴───────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Option 2: Schedule Worker (Empfohlen für Development)
|
||||
|
||||
Für lokales Development kannst du den Schedule Worker nutzen:
|
||||
|
||||
```bash
|
||||
php artisan schedule:work
|
||||
```
|
||||
|
||||
Dieser Befehl läuft dauerhaft und führt die Schedules automatisch aus.
|
||||
|
||||
**Vorteile:**
|
||||
- ✅ Kein Cron Job notwendig
|
||||
- ✅ Echtzeit-Ausgabe in der Console
|
||||
- ✅ Einfaches Debugging
|
||||
|
||||
**Nachteile:**
|
||||
- ❌ Muss manuell gestartet werden
|
||||
- ❌ Stoppt wenn Terminal geschlossen wird
|
||||
|
||||
**Lösung:** Nutze einen Process Manager wie Supervisor oder Screen:
|
||||
|
||||
```bash
|
||||
# Mit screen
|
||||
screen -S scheduler
|
||||
php artisan schedule:work
|
||||
# Ctrl+A, dann D zum Detachen
|
||||
|
||||
# Später wieder attachen
|
||||
screen -r scheduler
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Logs
|
||||
|
||||
### 1. Laravel Logs prüfen
|
||||
|
||||
```bash
|
||||
# Live-Logs anzeigen
|
||||
tail -f storage/logs/laravel.log
|
||||
|
||||
# Nur Sync-Logs filtern
|
||||
tail -f storage/logs/laravel.log | grep "SyncBackendDataPool"
|
||||
|
||||
# Scheduler-Logs (falls Cron Log aktiviert)
|
||||
tail -f storage/logs/scheduler.log
|
||||
```
|
||||
|
||||
### 2. Schedule-Status prüfen
|
||||
|
||||
```bash
|
||||
# Nächste geplante Ausführungen
|
||||
php artisan schedule:list
|
||||
|
||||
# Alle Schedules testen (ohne Ausführung)
|
||||
php artisan schedule:test
|
||||
```
|
||||
|
||||
### 3. Manuelle Sync-Ausführung
|
||||
|
||||
```bash
|
||||
# Incremental Sync manuell ausführen
|
||||
php artisan backend:sync-data-pool --incremental
|
||||
|
||||
# Full Sync manuell ausführen
|
||||
php artisan backend:sync-data-pool --full
|
||||
|
||||
# Mit Queue (asynchron)
|
||||
php artisan backend:sync-data-pool --incremental --queue
|
||||
```
|
||||
|
||||
### 4. Queue Worker (falls Jobs in Queue laufen)
|
||||
|
||||
Wenn du `--queue` nutzt, muss ein Queue Worker laufen:
|
||||
|
||||
```bash
|
||||
# Queue Worker starten
|
||||
php artisan queue:work --queue=default --tries=3
|
||||
|
||||
# Oder mit Supervisor für Produktion
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Zeitplan Übersicht
|
||||
|
||||
### Incremental Sync (alle 6 Stunden)
|
||||
|
||||
| Zeit | Aktion |
|
||||
|-----------|------------------|
|
||||
| 00:00 Uhr | Incremental Sync |
|
||||
| 06:00 Uhr | Incremental Sync |
|
||||
| 12:00 Uhr | Incremental Sync |
|
||||
| 18:00 Uhr | Incremental Sync |
|
||||
|
||||
### Full Sync (Sonntags)
|
||||
|
||||
| Tag | Zeit | Aktion |
|
||||
|---------|-----------|-----------|
|
||||
| Sonntag | 03:00 Uhr | Full Sync |
|
||||
|
||||
**Wichtig:** Am Sonntagmorgen um 3 Uhr läuft nur der Full Sync (nicht zusätzlich Incremental).
|
||||
|
||||
---
|
||||
|
||||
## Anpassungen
|
||||
|
||||
### Schedule-Zeiten ändern
|
||||
|
||||
Editiere [routes/console.php](routes/console.php):
|
||||
|
||||
#### Beispiele für andere Frequenzen:
|
||||
|
||||
```php
|
||||
// Incremental Sync: Stündlich
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->hourly();
|
||||
|
||||
// Incremental Sync: Alle 2 Stunden
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->everyTwoHours();
|
||||
|
||||
// Incremental Sync: Täglich um 2 Uhr
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->dailyAt('02:00');
|
||||
|
||||
// Full Sync: Täglich um 3 Uhr
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->dailyAt('03:00');
|
||||
|
||||
// Full Sync: Monatlich am 1. um 4 Uhr
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->monthlyOn(1, '04:00');
|
||||
```
|
||||
|
||||
### Batch Size ändern
|
||||
|
||||
```php
|
||||
// Kleinere Batches für weniger Speicherverbrauch
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 500))
|
||||
->everySixHours();
|
||||
|
||||
// Größere Batches für schnellere Verarbeitung
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 5000))
|
||||
->everySixHours();
|
||||
```
|
||||
|
||||
### Overlap Protection anpassen
|
||||
|
||||
```php
|
||||
// Längere Lock-Zeit (z.B. für große Datenmengen)
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->weeklyOn(0, '03:00')
|
||||
->withoutOverlapping(maxLockTime: 7200); // 2 Stunden
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Problem: "No scheduled commands are ready to run"
|
||||
|
||||
**Ursache:** Es ist noch nicht Zeit für die nächste Ausführung.
|
||||
|
||||
**Lösung:** Prüfe `php artisan schedule:list` für nächste Ausführungszeit.
|
||||
|
||||
---
|
||||
|
||||
### Problem: Schedule läuft nicht
|
||||
|
||||
**Ursache:** Cron Job nicht aktiv oder falsch konfiguriert.
|
||||
|
||||
**Lösung:**
|
||||
```bash
|
||||
# Prüfe Cron Job
|
||||
crontab -l
|
||||
|
||||
# Teste Schedule manuell
|
||||
php artisan schedule:run
|
||||
|
||||
# Prüfe Logs
|
||||
tail -f storage/logs/laravel.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem: "Class SyncBackendDataPool not found"
|
||||
|
||||
**Ursache:** Autoload-Cache ist veraltet.
|
||||
|
||||
**Lösung:**
|
||||
```bash
|
||||
composer dump-autoload
|
||||
php artisan optimize:clear
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem: Jobs laufen mehrfach parallel
|
||||
|
||||
**Ursache:** `withoutOverlapping()` funktioniert nicht.
|
||||
|
||||
**Lösung:** Stelle sicher, dass ein Cache-Driver konfiguriert ist:
|
||||
```bash
|
||||
# In .env
|
||||
CACHE_STORE=database
|
||||
```
|
||||
|
||||
Dann Cache-Tabellen migrieren:
|
||||
```bash
|
||||
php artisan cache:table
|
||||
php artisan migrate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem: Sync dauert zu lange
|
||||
|
||||
**Lösungen:**
|
||||
|
||||
1. **Batch Size erhöhen:**
|
||||
```php
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 5000))
|
||||
```
|
||||
|
||||
2. **Queue nutzen:**
|
||||
```php
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->everySixHours()
|
||||
->runInBackground(); // Bereits aktiviert
|
||||
```
|
||||
|
||||
3. **Max Lock Time erhöhen:**
|
||||
```php
|
||||
->withoutOverlapping(maxLockTime: 3600) // 1 Stunde
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notifications (Optional)
|
||||
|
||||
Du kannst Notifications hinzufügen um bei Erfolg/Fehler benachrichtigt zu werden:
|
||||
|
||||
### Slack Notification
|
||||
|
||||
```php
|
||||
use Illuminate\Support\Facades\Notification;
|
||||
use App\Notifications\SyncCompletedNotification;
|
||||
use App\Notifications\SyncFailedNotification;
|
||||
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->weeklyOn(0, '03:00')
|
||||
->onSuccess(function () {
|
||||
// Notification::route('slack', env('SLACK_WEBHOOK'))
|
||||
// ->notify(new SyncCompletedNotification());
|
||||
})
|
||||
->onFailure(function () {
|
||||
// Notification::route('slack', env('SLACK_WEBHOOK'))
|
||||
// ->notify(new SyncFailedNotification());
|
||||
});
|
||||
```
|
||||
|
||||
### Email Notification
|
||||
|
||||
```php
|
||||
->onSuccess(function () {
|
||||
Mail::to('admin@example.com')
|
||||
->send(new SyncCompletedMail());
|
||||
})
|
||||
->onFailure(function () {
|
||||
Mail::to('admin@example.com')
|
||||
->send(new SyncFailedMail());
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Monitoring
|
||||
|
||||
### Database Queries überwachen
|
||||
|
||||
```sql
|
||||
-- Anzahl Syncs heute
|
||||
SELECT DATE(synced_at) as date, COUNT(*) as syncs
|
||||
FROM public.backend_data_pool
|
||||
WHERE synced_at >= CURRENT_DATE
|
||||
GROUP BY DATE(synced_at);
|
||||
|
||||
-- Letzte Sync-Zeiten
|
||||
SELECT MAX(synced_at) as last_sync,
|
||||
MIN(synced_at) as first_sync,
|
||||
COUNT(*) as total_records
|
||||
FROM public.backend_data_pool;
|
||||
|
||||
-- Sync-Performance (Records pro Minute)
|
||||
SELECT
|
||||
DATE_TRUNC('minute', synced_at) as minute,
|
||||
COUNT(*) as records_synced
|
||||
FROM public.backend_data_pool
|
||||
WHERE synced_at >= NOW() - INTERVAL '1 hour'
|
||||
GROUP BY DATE_TRUNC('minute', synced_at)
|
||||
ORDER BY minute DESC;
|
||||
```
|
||||
|
||||
### Laravel Telescope (Optional)
|
||||
|
||||
Installiere Telescope für besseres Monitoring:
|
||||
|
||||
```bash
|
||||
composer require laravel/telescope --dev
|
||||
php artisan telescope:install
|
||||
php artisan migrate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Produktion Deployment
|
||||
|
||||
### Supervisor Configuration (Empfohlen)
|
||||
|
||||
Erstelle `/etc/supervisor/conf.d/laravel-scheduler.conf`:
|
||||
|
||||
```ini
|
||||
[program:laravel-scheduler]
|
||||
process_name=%(program_name)s
|
||||
command=php /Users/sebastianfrohlich/Herd/frontend/artisan schedule:work
|
||||
autostart=true
|
||||
autorestart=true
|
||||
user=sebastianfrohlich
|
||||
redirect_stderr=true
|
||||
stdout_logfile=/Users/sebastianfrohlich/Herd/frontend/storage/logs/scheduler.log
|
||||
stopwaitsecs=3600
|
||||
```
|
||||
|
||||
Dann:
|
||||
```bash
|
||||
sudo supervisorctl reread
|
||||
sudo supervisorctl update
|
||||
sudo supervisorctl start laravel-scheduler
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Schedule testen (ohne Ausführung)
|
||||
|
||||
```bash
|
||||
php artisan schedule:test
|
||||
```
|
||||
|
||||
### Nächste Ausführung simulieren
|
||||
|
||||
```bash
|
||||
# Teste Incremental Sync
|
||||
php artisan backend:sync-data-pool --incremental --no-interaction
|
||||
|
||||
# Teste Full Sync
|
||||
php artisan backend:sync-data-pool --full --no-interaction
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Zusammenfassung
|
||||
|
||||
✅ **Scheduling konfiguriert** in [routes/console.php](routes/console.php)
|
||||
✅ **Incremental Sync:** Alle 6 Stunden
|
||||
✅ **Full Sync:** Sonntags 3:00 Uhr
|
||||
✅ **Overlap Protection:** Aktiviert
|
||||
✅ **Background Execution:** Aktiviert
|
||||
✅ **Success/Failure Callbacks:** Implementiert
|
||||
|
||||
**Nächster Schritt:** Cron Job einrichten oder `php artisan schedule:work` starten!
|
||||
@@ -0,0 +1,447 @@
|
||||
# Stufe 2: Transformation vom Data Pool in produktive Tabellen
|
||||
|
||||
## Aktueller Stand (Stufe 1 ✅)
|
||||
|
||||
```
|
||||
backend.transactions (74 Transaktionen)
|
||||
+
|
||||
backend.transaction_outputs (7,471 Outputs)
|
||||
↓
|
||||
[SyncBackendDataPool Job]
|
||||
↓
|
||||
public.backend_data_pool (7,471 Datensätze)
|
||||
```
|
||||
|
||||
**Status:** ✅ Datenpool ist befüllt und wird alle 6h aktualisiert
|
||||
|
||||
---
|
||||
|
||||
## Ziel von Stufe 2
|
||||
|
||||
```
|
||||
public.backend_data_pool (7,471 Datensätze)
|
||||
↓
|
||||
[TransformDataPoolToProduction Job] ← NEU
|
||||
↓
|
||||
public.companies (Unique Companies mit KYC Risk Level)
|
||||
+
|
||||
public.transactions (Transaktionen mit allen 139 Spalten befüllt)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Was Stufe 2 macht
|
||||
|
||||
### 1. Companies erstellen/aktualisieren
|
||||
|
||||
Aus dem Data Pool werden unique Companies extrahiert:
|
||||
|
||||
**Input (Data Pool):**
|
||||
```
|
||||
transaction_id | corporate_entity | output_key | content
|
||||
---------------|----------------------------|-------------------|------------------
|
||||
1 | Mercedes-Benz Group AG | corporate_summary | {"answer": "..."}
|
||||
1 | Mercedes-Benz Group AG | corporate_sector | {"answer": "..."}
|
||||
2 | Samsung Electronics | corporate_summary | {"answer": "..."}
|
||||
```
|
||||
|
||||
**Output (public.companies):**
|
||||
```sql
|
||||
id | name | sector | country | kyc_risk_level | summary
|
||||
---|-------------------------|---------------|---------|----------------|----------
|
||||
1 | Mercedes-Benz Group AG | Automotive | DE | low | German...
|
||||
2 | Samsung Electronics | Electronics | KR | high | Korean...
|
||||
```
|
||||
|
||||
**Wie:**
|
||||
- Gruppiere nach `corporate_entity` (eindeutige Firmennamen)
|
||||
- Extrahiere Company-Daten aus `corporate_*` outputs
|
||||
- **Berechne KYC Risk Level** basierend auf:
|
||||
- `tranx_score`
|
||||
- `corporate_eusanctions`, `corporate_ofacsanctions`
|
||||
- `country_risk`
|
||||
- `corporate_pepexposure`, `corporate_AMLexposure`
|
||||
- `corruption_*` outputs
|
||||
|
||||
---
|
||||
|
||||
### 2. Transactions erstellen/aktualisieren
|
||||
|
||||
Für jede Transaction im Data Pool wird ein Datensatz in `public.transactions` erstellt:
|
||||
|
||||
**Input (Data Pool - gruppiert nach transaction_id):**
|
||||
```
|
||||
transaction_id: 1
|
||||
- corporate_entity: "Mercedes-Benz Group AG"
|
||||
- tx_amount: 54880.9
|
||||
- tx_date: "2025-10-28"
|
||||
- outputs:
|
||||
- corporate_summary: {...}
|
||||
- corporate_history: {...}
|
||||
- tranx_score: {"score": 45}
|
||||
- ... (101 output_keys total)
|
||||
```
|
||||
|
||||
**Output (public.transactions):**
|
||||
```sql
|
||||
id | company_id | reference | amount | executed_at | risk_score | corporate_summary | tranx_score | ... (139 Spalten)
|
||||
---|------------|------------------|----------|-------------|------------|-------------------|-------------|----
|
||||
1 | 1 | MIGRATED-1 | 54880.9 | 2025-10-28 | 45 | {"answer": "..."} | {"score":45}| ...
|
||||
```
|
||||
|
||||
**Mapping:**
|
||||
- **Core Felder:**
|
||||
- `company_id` ← Lookup/Create Company by `corporate_entity`
|
||||
- `reference` ← `'MIGRATED-' || transaction_id`
|
||||
- `amount` ← `tx_amount`
|
||||
- `currency` ← `tx_currency`
|
||||
- `executed_at` ← `tx_date::timestamp`
|
||||
- `counterparty` ← `corporate_counterparty`
|
||||
- `status` ← `status`
|
||||
|
||||
- **Output Felder (102 JSONB Spalten):**
|
||||
- `corporate_summary` ← content WHERE output_key='corporate_summary'
|
||||
- `corporate_history` ← content WHERE output_key='corporate_history'
|
||||
- `tranx_score` ← content WHERE output_key='tranx_score'
|
||||
- ... für alle 102 output_keys
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Technische Details
|
||||
|
||||
### Job-Struktur
|
||||
|
||||
```php
|
||||
class TransformDataPoolToProduction implements ShouldQueue
|
||||
{
|
||||
public function handle()
|
||||
{
|
||||
// 1. Hole alle unique transactions aus data pool
|
||||
$transactions = DB::table('backend_data_pool')
|
||||
->select('transaction_id')
|
||||
->distinct()
|
||||
->get();
|
||||
|
||||
foreach ($transactions as $transaction) {
|
||||
// 2. Hole alle Outputs für diese Transaction
|
||||
$outputs = $this->getOutputsForTransaction($transaction->transaction_id);
|
||||
|
||||
// 3. Erstelle/Update Company
|
||||
$company = $this->createOrUpdateCompany($outputs);
|
||||
|
||||
// 4. Erstelle/Update Transaction
|
||||
$this->createOrUpdateTransaction($company, $outputs);
|
||||
}
|
||||
}
|
||||
|
||||
private function createOrUpdateCompany($outputs)
|
||||
{
|
||||
$corporateEntity = $outputs->first()->corporate_entity;
|
||||
|
||||
// Berechne KYC Risk Level
|
||||
$kycRiskLevel = $this->calculateKycRiskLevel($outputs);
|
||||
|
||||
return Company::updateOrCreate(
|
||||
['name' => $corporateEntity],
|
||||
[
|
||||
'sector' => $this->extractSector($outputs),
|
||||
'country' => $this->extractCountry($outputs),
|
||||
'kyc_risk_level' => $kycRiskLevel,
|
||||
'summary' => $this->extractSummary($outputs),
|
||||
// ... weitere Felder
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
private function createOrUpdateTransaction($company, $outputs)
|
||||
{
|
||||
$first = $outputs->first();
|
||||
|
||||
// Core Felder
|
||||
$transactionData = [
|
||||
'company_id' => $company->id,
|
||||
'reference' => 'MIGRATED-' . $first->transaction_id,
|
||||
'amount' => $first->tx_amount,
|
||||
'currency' => $first->tx_currency,
|
||||
'executed_at' => Carbon::parse($first->tx_date),
|
||||
'status' => $first->status,
|
||||
// ...
|
||||
];
|
||||
|
||||
// Output Felder (102 JSONB columns)
|
||||
foreach ($outputs as $output) {
|
||||
$columnName = $output->output_key;
|
||||
$transactionData[$columnName] = json_decode($output->content, true);
|
||||
}
|
||||
|
||||
return Transaction::updateOrCreate(
|
||||
['reference' => $transactionData['reference']],
|
||||
$transactionData
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Beispiel-Transformation
|
||||
|
||||
### Input: Data Pool Datensätze für Transaction ID=1
|
||||
|
||||
```
|
||||
transaction_id=1, corporate_entity="Mercedes-Benz Group AG", tx_amount=54880.9
|
||||
output_key="corporate_summary" → content={"answer": "..."}
|
||||
output_key="corporate_sector" → content={"answer": "Automotive"}
|
||||
output_key="corporate_HQ" → content={"answer": "Stuttgart, Germany"}
|
||||
output_key="tranx_score" → content={"score": 45}
|
||||
output_key="corporate_eusanctions" → content={"found": false}
|
||||
... (101 outputs total)
|
||||
```
|
||||
|
||||
### Output 1: Companies Tabelle
|
||||
|
||||
```sql
|
||||
INSERT INTO public.companies (name, sector, headquarters, kyc_risk_level, summary)
|
||||
VALUES (
|
||||
'Mercedes-Benz Group AG',
|
||||
'Automotive',
|
||||
'Stuttgart, Germany',
|
||||
'low', -- Berechnet aus tranx_score=45, keine Sanctions, etc.
|
||||
'...'
|
||||
);
|
||||
```
|
||||
|
||||
### Output 2: Transactions Tabelle
|
||||
|
||||
```sql
|
||||
INSERT INTO public.transactions (
|
||||
company_id,
|
||||
reference,
|
||||
amount,
|
||||
currency,
|
||||
executed_at,
|
||||
status,
|
||||
corporate_summary,
|
||||
corporate_sector,
|
||||
tranx_score,
|
||||
corporate_eusanctions,
|
||||
... -- alle 102 output columns
|
||||
)
|
||||
VALUES (
|
||||
1, -- Company ID
|
||||
'MIGRATED-1',
|
||||
54880.9,
|
||||
'EUR',
|
||||
'2025-10-28',
|
||||
'done',
|
||||
'{"answer": "..."}',
|
||||
'{"answer": "Automotive"}',
|
||||
'{"score": 45}',
|
||||
'{"found": false}',
|
||||
...
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Features von Stufe 2
|
||||
|
||||
### ✅ Intelligente Company-Erstellung
|
||||
- **Deduplizierung:** Gleicher Name = gleiche Company
|
||||
- **Enrichment:** Automatische Extraktion von Sector, HQ, etc.
|
||||
- **KYC Risk Berechnung:** Automatische Risikobewertung
|
||||
|
||||
### ✅ Vollständige Transaction-Daten
|
||||
- Alle 139 Spalten werden befüllt
|
||||
- JSON-Daten aus Outputs werden korrekt gemappt
|
||||
- Referenz-Nummern für Tracking
|
||||
|
||||
### ✅ Idempotent
|
||||
- Mehrfaches Ausführen ist sicher
|
||||
- `updateOrCreate()` verhindert Duplikate
|
||||
- Bestehende Daten werden aktualisiert
|
||||
|
||||
### ✅ Batch Processing
|
||||
- Verarbeitet Daten in Batches
|
||||
- Kann in Queue laufen
|
||||
- Progress Tracking
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Workflow nach Implementation
|
||||
|
||||
```
|
||||
1. Backend Daten ändern sich
|
||||
↓
|
||||
2. SyncBackendDataPool läuft (alle 6h)
|
||||
→ backend_data_pool aktualisiert
|
||||
↓
|
||||
3. TransformDataPoolToProduction läuft (nach Sync)
|
||||
→ Companies aktualisiert
|
||||
→ Transactions aktualisiert
|
||||
↓
|
||||
4. Frontend zeigt aktuelle Daten
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ❓ Offene Fragen für Stufe 2
|
||||
|
||||
### 1. Update-Strategie
|
||||
- Sollen bestehende Transactions überschrieben werden?
|
||||
- Oder nur neue Transactions hinzufügen?
|
||||
|
||||
### 2. Company Matching
|
||||
- Nur exakter Name-Match?
|
||||
- Oder fuzzy matching (z.B. "BASF SE" vs "BASF")?
|
||||
|
||||
### 3. Risk Level Defaults
|
||||
- Was wenn keine Risk-Daten vorhanden?
|
||||
- Default zu "high" (vorsichtig) oder "low"?
|
||||
|
||||
### 4. Fehlende Felder
|
||||
- Was wenn ein output_key fehlt?
|
||||
- NULL speichern oder Default-Wert?
|
||||
|
||||
### 5. Scheduling
|
||||
- Soll Stufe 2 direkt nach Stufe 1 laufen?
|
||||
- Oder separater Schedule?
|
||||
|
||||
---
|
||||
|
||||
## 🔑 KYC Risk Level Berechnung
|
||||
|
||||
### Verfügbare Risk-Daten
|
||||
|
||||
**Aus transaction_outputs:**
|
||||
- `tranx_score` - Haupt-Risiko-Score (JSON mit numerischem Wert)
|
||||
- `tranx_reasoning` - Begründung für den Score
|
||||
- `country_risk` - Länder-Risiko
|
||||
- `sanctions_circumvention` - Sanktionsumgehung
|
||||
- `corruption_sector` - Korruption im Sektor
|
||||
- `corruption_country` - Korruption im Land
|
||||
- `corruption_relationship` - Korruption in Beziehungen
|
||||
- `corporate_eusanctions` - EU Sanktionen
|
||||
- `corporate_ofacsanctions` - OFAC Sanktionen
|
||||
- `corporate_uksanctions` - UK Sanktionen
|
||||
- `corporate_pepexposure` - PEP Exposure
|
||||
- `corporate_AMLexposure` - AML Exposure
|
||||
- `corporate_adverse` - Adverse Media
|
||||
|
||||
### Berechnungs-Logik (3 Risk Levels)
|
||||
|
||||
**Risk Levels:**
|
||||
1. **low** (Geringes Risiko): Score 0-40
|
||||
2. **high** (Hohes Risiko): Score 41-70
|
||||
3. **critical** (Kritisches Risiko): Score 71-100
|
||||
|
||||
**Gewichtung:**
|
||||
- Transaction Score: 40%
|
||||
- Sanctions: 25%
|
||||
- Country Risk: 15%
|
||||
- PEP/Adverse: 10%
|
||||
- Corruption: 10%
|
||||
|
||||
**Beispiel:**
|
||||
```
|
||||
Transaction Score: 45 × 0.40 = 18
|
||||
Sanctions: 0 × 0.25 = 0
|
||||
Country Risk: 30 × 0.15 = 4.5
|
||||
PEP/Adverse: 0 × 0.10 = 0
|
||||
Corruption: 0 × 0.10 = 0
|
||||
--------------------------------
|
||||
Total Score: 22.5 → "low"
|
||||
```
|
||||
|
||||
**Company-Level Risk:**
|
||||
- Aggregiert über alle Transaktionen einer Company
|
||||
- Worst-Case-Prinzip: Eine critical Transaction → Company ist critical
|
||||
- Wenn >30% high → Company ist critical
|
||||
- Wenn >10% high → Company ist high
|
||||
|
||||
---
|
||||
|
||||
## 📋 Implementierungs-Schritte
|
||||
|
||||
### Phase 1: Vorbereitung ✅ ABGESCHLOSSEN
|
||||
1. ✅ Data Pool ist befüllt
|
||||
2. ✅ KYC Risk Calculator Service erstellen
|
||||
3. ✅ Data Extraction Helpers erstellen
|
||||
4. ✅ Mapping-Logik definieren
|
||||
|
||||
### Phase 2: Job Implementation ✅ ABGESCHLOSSEN
|
||||
1. ✅ TransformDataPoolToProduction Job erstellen
|
||||
2. ✅ Company Creation Logic implementieren
|
||||
3. ✅ Transaction Creation Logic implementieren
|
||||
4. ✅ Error Handling & Logging
|
||||
|
||||
### Phase 3: Testing
|
||||
1. ⬜ Unit Tests für Risk Calculator
|
||||
2. ⬜ Feature Tests für Transformation Job
|
||||
3. ⬜ Datenintegritäts-Checks
|
||||
|
||||
### Phase 4: Scheduling ✅ ABGESCHLOSSEN
|
||||
1. ✅ Schedule konfigurieren (läuft alle 6h um :30)
|
||||
2. ✅ Queue Setup (optional)
|
||||
3. ⬜ Monitoring einrichten
|
||||
|
||||
### Phase 5: Deployment
|
||||
1. ✅ Produktions-Test mit echten Daten
|
||||
2. ⬜ Performance-Optimierung
|
||||
3. ⬜ Dokumentation finalisieren
|
||||
|
||||
---
|
||||
|
||||
## 📈 Erwartete Ergebnisse
|
||||
|
||||
Nach erfolgreicher Implementation von Stufe 2:
|
||||
|
||||
**Companies:**
|
||||
- ~60-70 unique Companies (geschätzt aus 74 Transaktionen)
|
||||
- Alle mit KYC Risk Level
|
||||
- Enriched mit Sector, Country, HQ, etc.
|
||||
|
||||
**Transactions:**
|
||||
- 74 Transaktionen
|
||||
- Alle 139 Spalten befüllt
|
||||
- Verlinkt mit Companies
|
||||
- Referenz-Nummern für Tracking
|
||||
|
||||
**Performance:**
|
||||
- Erste Transformation: ~30-60 Sekunden
|
||||
- Incremental Updates: ~5-10 Sekunden
|
||||
- Kann parallel zu Stufe 1 laufen
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Nächste Schritte
|
||||
|
||||
1. **Beantworten der offenen Fragen**
|
||||
2. **Implementation des TransformDataPoolToProduction Jobs**
|
||||
3. **Tests schreiben und ausführen**
|
||||
4. **Scheduling einrichten**
|
||||
5. **Monitoring & Alerts konfigurieren**
|
||||
|
||||
---
|
||||
|
||||
## 🎉 Status Update
|
||||
|
||||
**Erstellt am:** 2025-11-16
|
||||
**Aktualisiert am:** 2025-11-21
|
||||
**Status:** ✅ **PRODUKTIV** - Stufe 2 ist implementiert und läuft automatisch!
|
||||
|
||||
### Was funktioniert:
|
||||
|
||||
✅ **TransformDataPoolToProduction Job** - Vollständig implementiert
|
||||
✅ **Automatisches Scheduling** - Läuft alle 6 Stunden um :30 (0:30, 6:30, 12:30, 18:30)
|
||||
✅ **Company & Transaction Creation** - Automatische Erstellung und Updates
|
||||
✅ **Risk Score Mapping** - Verwendet Backend risk_score direkt
|
||||
✅ **Artisan Command** - `php artisan backend:transform-data-pool` verfügbar
|
||||
|
||||
### Offene TODOs:
|
||||
|
||||
⬜ Unit Tests für Transformation Job (Phase 3)
|
||||
⬜ Feature Tests für Transformation Job (Phase 3)
|
||||
⬜ Monitoring einrichten (Phase 4)
|
||||
⬜ Performance-Optimierung (Phase 5)
|
||||
⬜ KycRiskCalculator Service integrieren (optional, aktuell nicht genutzt)
|
||||
@@ -0,0 +1,305 @@
|
||||
# Backend Data Pool Sync - Migration & Testing Instructions
|
||||
|
||||
## Übersicht
|
||||
|
||||
Dieser Guide führt dich durch das Testen und Ausführen der Backend Data Pool Migration.
|
||||
|
||||
## Schritt 1: Migration ausführen
|
||||
|
||||
```bash
|
||||
php artisan migrate
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
Migrating: 2025_11_16_150000_create_backend_data_pool_table
|
||||
Migrated: 2025_11_16_150000_create_backend_data_pool_table
|
||||
```
|
||||
|
||||
### Bei Problemen:
|
||||
|
||||
Wenn die Migration fehlschlägt, prüfe:
|
||||
|
||||
```bash
|
||||
# Datenbankverbindung testen
|
||||
php artisan db:show
|
||||
|
||||
# Oder spezifisch für pgsql
|
||||
php artisan db:show --database=pgsql
|
||||
```
|
||||
|
||||
## Schritt 2: Struktur der Tabelle prüfen
|
||||
|
||||
```sql
|
||||
-- In psql oder einem DB-Tool
|
||||
\d public.backend_data_pool
|
||||
|
||||
-- Oder mit artisan tinker
|
||||
php artisan tinker
|
||||
>>> DB::select("SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'backend_data_pool' ORDER BY ordinal_position");
|
||||
```
|
||||
|
||||
## Schritt 3: Statistiken vor dem Sync prüfen
|
||||
|
||||
```bash
|
||||
php artisan backend:sync-data-pool --stats
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
┌─────────────────────────────────┬────────┐
|
||||
│ Metric │ Value │
|
||||
├─────────────────────────────────┼────────┤
|
||||
│ Backend Transactions (done) │ XXX │
|
||||
│ Backend Transaction Outputs │ XXX │
|
||||
│ Data Pool Records │ 0 │
|
||||
│ Last Synced At │ Never │
|
||||
└─────────────────────────────────┴────────┘
|
||||
```
|
||||
|
||||
## Schritt 4: Ersten Full Sync durchführen
|
||||
|
||||
```bash
|
||||
php artisan backend:sync-data-pool --full
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
Preparing for FULL SYNC
|
||||
|
||||
⚠ This will truncate and rebuild the entire backend_data_pool table.
|
||||
|
||||
┌ Do you want to continue? ─────────────────────────────┐
|
||||
│ Yes / No │
|
||||
└───────────────────────────────────────────────────────┘
|
||||
|
||||
Stats BEFORE sync:
|
||||
┌─────────────────────────────────┬────────┐
|
||||
│ Metric │ Value │
|
||||
├─────────────────────────────────┼────────┤
|
||||
│ Backend Transactions (done) │ XXX │
|
||||
│ Data Pool Records │ 0 │
|
||||
│ Last Synced At │ Never │
|
||||
└─────────────────────────────────┴────────┘
|
||||
|
||||
⠙ Syncing data...
|
||||
|
||||
✔ Sync completed successfully!
|
||||
|
||||
Stats AFTER sync:
|
||||
┌─────────────────────────────────┬────────┐
|
||||
│ Metric │ Value │
|
||||
├─────────────────────────────────┼────────┤
|
||||
│ Backend Transactions (done) │ XXX │
|
||||
│ Data Pool Records │ XXX │
|
||||
│ Last Synced At │ 2025-11-16 15:30:45 │
|
||||
└─────────────────────────────────┴────────┘
|
||||
```
|
||||
|
||||
## Schritt 5: Daten verifizieren
|
||||
|
||||
### SQL Query zum Vergleichen:
|
||||
|
||||
```sql
|
||||
-- Anzahl Datensätze in backend.transactions mit status='done'
|
||||
SELECT COUNT(*)
|
||||
FROM backend.transactions
|
||||
WHERE status = 'done';
|
||||
|
||||
-- Anzahl Datensätze in backend.transaction_outputs (für done transactions)
|
||||
SELECT COUNT(*)
|
||||
FROM backend.transaction_outputs tout
|
||||
JOIN backend.transactions t ON t.id = tout.transaction_id
|
||||
WHERE t.status = 'done';
|
||||
|
||||
-- Anzahl Datensätze in public.backend_data_pool
|
||||
SELECT COUNT(*)
|
||||
FROM public.backend_data_pool;
|
||||
|
||||
-- Die zweite und dritte Zahl sollten gleich sein!
|
||||
```
|
||||
|
||||
### Beispiel-Datensätze prüfen:
|
||||
|
||||
```sql
|
||||
-- Ersten 5 Datensätze anzeigen
|
||||
SELECT
|
||||
transaction_id,
|
||||
corporate_entity,
|
||||
tx_amount,
|
||||
output_key,
|
||||
LEFT(content, 50) as content_preview,
|
||||
synced_at
|
||||
FROM public.backend_data_pool
|
||||
ORDER BY transaction_id, prompt_id
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
### Prüfe ob alle output_keys vorhanden sind:
|
||||
|
||||
```sql
|
||||
-- Welche output_keys sind im Data Pool?
|
||||
SELECT
|
||||
output_key,
|
||||
COUNT(*) as count
|
||||
FROM public.backend_data_pool
|
||||
GROUP BY output_key
|
||||
ORDER BY count DESC;
|
||||
```
|
||||
|
||||
## Schritt 6: Tests ausführen
|
||||
|
||||
```bash
|
||||
php artisan test --filter=SyncBackendDataPool
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
PASS Tests\Feature\Jobs\SyncBackendDataPoolTest
|
||||
✓ full sync truncates and rebuilds backend_data_pool
|
||||
✓ incremental sync only adds new records
|
||||
✓ sync correctly maps backend.transactions and backend.transaction_outputs
|
||||
✓ sync only includes transactions with status done
|
||||
✓ sync processes records in batches
|
||||
✓ getStats returns correct statistics
|
||||
✓ sync handles empty backend gracefully
|
||||
✓ sync orders records by transaction_id and prompt_id
|
||||
|
||||
Tests: 8 passed (X assertions)
|
||||
Duration: XXs
|
||||
```
|
||||
|
||||
## Schritt 7: Incremental Sync testen
|
||||
|
||||
```bash
|
||||
# Warte kurz oder füge neue Daten im backend hinzu
|
||||
# Dann führe incremental sync aus:
|
||||
php artisan backend:sync-data-pool --incremental
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Problem: "SQLSTATE[42P01]: Undefined table"
|
||||
|
||||
**Lösung:** Migration wurde nicht ausgeführt oder fehlgeschlagen.
|
||||
```bash
|
||||
php artisan migrate:status
|
||||
php artisan migrate --force
|
||||
```
|
||||
|
||||
### Problem: "Connection refused" oder Connection Error
|
||||
|
||||
**Lösung:** Prüfe .env Datei:
|
||||
```bash
|
||||
DB_CONNECTION2=pgsql
|
||||
DB_HOST2=127.0.0.1
|
||||
DB_PORT2=5433
|
||||
DB_DATABASE2=risk_ingest_db
|
||||
DB_USERNAME2=risk_ingest_user
|
||||
DB_PASSWORD2=S0prast3r1a
|
||||
```
|
||||
|
||||
Teste Verbindung:
|
||||
```bash
|
||||
psql -h 127.0.0.1 -p 5433 -U risk_ingest_user -d risk_ingest_db
|
||||
```
|
||||
|
||||
### Problem: "schema backend does not exist"
|
||||
|
||||
**Lösung:** Backend Schema fehlt. Erstelle es:
|
||||
```sql
|
||||
CREATE SCHEMA IF NOT EXISTS backend;
|
||||
```
|
||||
|
||||
### Problem: Job läuft sehr lange
|
||||
|
||||
**Lösung:** Nutze kleinere Batch Size oder Queue:
|
||||
```bash
|
||||
# Kleinere Batches
|
||||
php artisan backend:sync-data-pool --full --batch-size=100
|
||||
|
||||
# Oder mit Queue (asynchron)
|
||||
php artisan backend:sync-data-pool --full --queue
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Für große Datenmengen:
|
||||
|
||||
1. **Nutze Queue:**
|
||||
```bash
|
||||
php artisan backend:sync-data-pool --full --queue
|
||||
php artisan queue:work
|
||||
```
|
||||
|
||||
2. **Optimiere Batch Size:**
|
||||
```bash
|
||||
# Für viele Datensätze
|
||||
php artisan backend:sync-data-pool --full --batch-size=5000
|
||||
|
||||
# Für wenig RAM
|
||||
php artisan backend:sync-data-pool --full --batch-size=100
|
||||
```
|
||||
|
||||
3. **Schedule für regelmäßige Syncs:**
|
||||
|
||||
Füge in `routes/console.php` hinzu:
|
||||
```php
|
||||
use App\Jobs\SyncBackendDataPool;
|
||||
use Illuminate\Support\Facades\Schedule;
|
||||
|
||||
// Täglich um 2 Uhr incremental
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->dailyAt('02:00');
|
||||
|
||||
// Sonntags um 3 Uhr full sync
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->weeklyOn(0, '03:00');
|
||||
```
|
||||
|
||||
Dann starte den Scheduler:
|
||||
```bash
|
||||
php artisan schedule:work
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Logs prüfen:
|
||||
|
||||
```bash
|
||||
tail -f storage/logs/laravel.log
|
||||
```
|
||||
|
||||
### SQL Queries für Monitoring:
|
||||
|
||||
```sql
|
||||
-- Letzte Sync Zeit
|
||||
SELECT MAX(synced_at) FROM public.backend_data_pool;
|
||||
|
||||
-- Anzahl Datensätze pro Status
|
||||
SELECT status, COUNT(*)
|
||||
FROM public.backend_data_pool
|
||||
GROUP BY status;
|
||||
|
||||
-- Anzahl Datensätze pro corporate_entity (Top 10)
|
||||
SELECT corporate_entity, COUNT(*) as transaction_count
|
||||
FROM public.backend_data_pool
|
||||
GROUP BY corporate_entity
|
||||
ORDER BY transaction_count DESC
|
||||
LIMIT 10;
|
||||
|
||||
-- Durchschnittlicher Transaktionsbetrag
|
||||
SELECT AVG(tx_amount) as avg_amount
|
||||
FROM public.backend_data_pool;
|
||||
```
|
||||
|
||||
## Nächste Schritte
|
||||
|
||||
Nach erfolgreichem Sync kannst du mit **Stufe 2** weitermachen:
|
||||
- Transformation der Daten in `public.companies` und `public.transactions`
|
||||
- Mit KYC Risk Level Berechnung
|
||||
- Automatisches Enrichment der Companies
|
||||
|
||||
---
|
||||
|
||||
**Bei Fragen oder Problemen, prüfe die Logs in `storage/logs/laravel.log`**
|
||||
Reference in New Issue
Block a user