chore: major project cleanup and restructuring
## Dokumentation - Entfernt AGENTS.md (redundant zu CLAUDE.md) - Verschoben: 3 Dokumentationen von root → docs/ - Gelöscht: 8 historische Migrations-Pläne aus docs/archive/ - Entfernt: misc/ Ordner komplett (43 Dateien) ## Struktur - Erstellt: database/queries/ für SQL Test-Queries (4 Dateien) - Erstellt: database/schemas/ für Schema-Dokumentation (17 Dateien) - output_keys_mapping/ (8 CSV-Mappings) - migration-diagrams/ (4 Diagramme) ## Code-Qualität - Formatiert: 7 Style-Issues in 74 Dateien mit Laravel Pint - Gefixed: Migration für PostgreSQL/SQLite Kompatibilität - Gefixed: 2 fehlgeschlagene Tests (CSRF + Text-Assertion) ## Tests - Alle 73 Tests bestehen jetzt (100% Success Rate) - AuthenticationTest: CSRF-Token Fix für Logout-Test - CompanySearchTest: Text-Assertion aktualisiert ## Ergebnis - Root ist sauber (nur CLAUDE.md) - Dokumentation strukturiert in docs/ - Database-Dateien organisiert in database/ - Code entspricht Style-Guide - Alle Tests bestehen
This commit is contained in:
@@ -0,0 +1,305 @@
|
||||
# Backend Data Pool Sync - Migration & Testing Instructions
|
||||
|
||||
## Übersicht
|
||||
|
||||
Dieser Guide führt dich durch das Testen und Ausführen der Backend Data Pool Migration.
|
||||
|
||||
## Schritt 1: Migration ausführen
|
||||
|
||||
```bash
|
||||
php artisan migrate
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
Migrating: 2025_11_16_150000_create_backend_data_pool_table
|
||||
Migrated: 2025_11_16_150000_create_backend_data_pool_table
|
||||
```
|
||||
|
||||
### Bei Problemen:
|
||||
|
||||
Wenn die Migration fehlschlägt, prüfe:
|
||||
|
||||
```bash
|
||||
# Datenbankverbindung testen
|
||||
php artisan db:show
|
||||
|
||||
# Oder spezifisch für pgsql
|
||||
php artisan db:show --database=pgsql
|
||||
```
|
||||
|
||||
## Schritt 2: Struktur der Tabelle prüfen
|
||||
|
||||
```sql
|
||||
-- In psql oder einem DB-Tool
|
||||
\d public.backend_data_pool
|
||||
|
||||
-- Oder mit artisan tinker
|
||||
php artisan tinker
|
||||
>>> DB::select("SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'backend_data_pool' ORDER BY ordinal_position");
|
||||
```
|
||||
|
||||
## Schritt 3: Statistiken vor dem Sync prüfen
|
||||
|
||||
```bash
|
||||
php artisan backend:sync-data-pool --stats
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
┌─────────────────────────────────┬────────┐
|
||||
│ Metric │ Value │
|
||||
├─────────────────────────────────┼────────┤
|
||||
│ Backend Transactions (done) │ XXX │
|
||||
│ Backend Transaction Outputs │ XXX │
|
||||
│ Data Pool Records │ 0 │
|
||||
│ Last Synced At │ Never │
|
||||
└─────────────────────────────────┴────────┘
|
||||
```
|
||||
|
||||
## Schritt 4: Ersten Full Sync durchführen
|
||||
|
||||
```bash
|
||||
php artisan backend:sync-data-pool --full
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
Preparing for FULL SYNC
|
||||
|
||||
⚠ This will truncate and rebuild the entire backend_data_pool table.
|
||||
|
||||
┌ Do you want to continue? ─────────────────────────────┐
|
||||
│ Yes / No │
|
||||
└───────────────────────────────────────────────────────┘
|
||||
|
||||
Stats BEFORE sync:
|
||||
┌─────────────────────────────────┬────────┐
|
||||
│ Metric │ Value │
|
||||
├─────────────────────────────────┼────────┤
|
||||
│ Backend Transactions (done) │ XXX │
|
||||
│ Data Pool Records │ 0 │
|
||||
│ Last Synced At │ Never │
|
||||
└─────────────────────────────────┴────────┘
|
||||
|
||||
⠙ Syncing data...
|
||||
|
||||
✔ Sync completed successfully!
|
||||
|
||||
Stats AFTER sync:
|
||||
┌─────────────────────────────────┬────────┐
|
||||
│ Metric │ Value │
|
||||
├─────────────────────────────────┼────────┤
|
||||
│ Backend Transactions (done) │ XXX │
|
||||
│ Data Pool Records │ XXX │
|
||||
│ Last Synced At │ 2025-11-16 15:30:45 │
|
||||
└─────────────────────────────────┴────────┘
|
||||
```
|
||||
|
||||
## Schritt 5: Daten verifizieren
|
||||
|
||||
### SQL Query zum Vergleichen:
|
||||
|
||||
```sql
|
||||
-- Anzahl Datensätze in backend.transactions mit status='done'
|
||||
SELECT COUNT(*)
|
||||
FROM backend.transactions
|
||||
WHERE status = 'done';
|
||||
|
||||
-- Anzahl Datensätze in backend.transaction_outputs (für done transactions)
|
||||
SELECT COUNT(*)
|
||||
FROM backend.transaction_outputs tout
|
||||
JOIN backend.transactions t ON t.id = tout.transaction_id
|
||||
WHERE t.status = 'done';
|
||||
|
||||
-- Anzahl Datensätze in public.backend_data_pool
|
||||
SELECT COUNT(*)
|
||||
FROM public.backend_data_pool;
|
||||
|
||||
-- Die zweite und dritte Zahl sollten gleich sein!
|
||||
```
|
||||
|
||||
### Beispiel-Datensätze prüfen:
|
||||
|
||||
```sql
|
||||
-- Ersten 5 Datensätze anzeigen
|
||||
SELECT
|
||||
transaction_id,
|
||||
corporate_entity,
|
||||
tx_amount,
|
||||
output_key,
|
||||
LEFT(content, 50) as content_preview,
|
||||
synced_at
|
||||
FROM public.backend_data_pool
|
||||
ORDER BY transaction_id, prompt_id
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
### Prüfe ob alle output_keys vorhanden sind:
|
||||
|
||||
```sql
|
||||
-- Welche output_keys sind im Data Pool?
|
||||
SELECT
|
||||
output_key,
|
||||
COUNT(*) as count
|
||||
FROM public.backend_data_pool
|
||||
GROUP BY output_key
|
||||
ORDER BY count DESC;
|
||||
```
|
||||
|
||||
## Schritt 6: Tests ausführen
|
||||
|
||||
```bash
|
||||
php artisan test --filter=SyncBackendDataPool
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
PASS Tests\Feature\Jobs\SyncBackendDataPoolTest
|
||||
✓ full sync truncates and rebuilds backend_data_pool
|
||||
✓ incremental sync only adds new records
|
||||
✓ sync correctly maps backend.transactions and backend.transaction_outputs
|
||||
✓ sync only includes transactions with status done
|
||||
✓ sync processes records in batches
|
||||
✓ getStats returns correct statistics
|
||||
✓ sync handles empty backend gracefully
|
||||
✓ sync orders records by transaction_id and prompt_id
|
||||
|
||||
Tests: 8 passed (X assertions)
|
||||
Duration: XXs
|
||||
```
|
||||
|
||||
## Schritt 7: Incremental Sync testen
|
||||
|
||||
```bash
|
||||
# Warte kurz oder füge neue Daten im backend hinzu
|
||||
# Dann führe incremental sync aus:
|
||||
php artisan backend:sync-data-pool --incremental
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Problem: "SQLSTATE[42P01]: Undefined table"
|
||||
|
||||
**Lösung:** Migration wurde nicht ausgeführt oder fehlgeschlagen.
|
||||
```bash
|
||||
php artisan migrate:status
|
||||
php artisan migrate --force
|
||||
```
|
||||
|
||||
### Problem: "Connection refused" oder Connection Error
|
||||
|
||||
**Lösung:** Prüfe .env Datei:
|
||||
```bash
|
||||
DB_CONNECTION2=pgsql
|
||||
DB_HOST2=127.0.0.1
|
||||
DB_PORT2=5433
|
||||
DB_DATABASE2=risk_ingest_db
|
||||
DB_USERNAME2=risk_ingest_user
|
||||
DB_PASSWORD2=S0prast3r1a
|
||||
```
|
||||
|
||||
Teste Verbindung:
|
||||
```bash
|
||||
psql -h 127.0.0.1 -p 5433 -U risk_ingest_user -d risk_ingest_db
|
||||
```
|
||||
|
||||
### Problem: "schema backend does not exist"
|
||||
|
||||
**Lösung:** Backend Schema fehlt. Erstelle es:
|
||||
```sql
|
||||
CREATE SCHEMA IF NOT EXISTS backend;
|
||||
```
|
||||
|
||||
### Problem: Job läuft sehr lange
|
||||
|
||||
**Lösung:** Nutze kleinere Batch Size oder Queue:
|
||||
```bash
|
||||
# Kleinere Batches
|
||||
php artisan backend:sync-data-pool --full --batch-size=100
|
||||
|
||||
# Oder mit Queue (asynchron)
|
||||
php artisan backend:sync-data-pool --full --queue
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Für große Datenmengen:
|
||||
|
||||
1. **Nutze Queue:**
|
||||
```bash
|
||||
php artisan backend:sync-data-pool --full --queue
|
||||
php artisan queue:work
|
||||
```
|
||||
|
||||
2. **Optimiere Batch Size:**
|
||||
```bash
|
||||
# Für viele Datensätze
|
||||
php artisan backend:sync-data-pool --full --batch-size=5000
|
||||
|
||||
# Für wenig RAM
|
||||
php artisan backend:sync-data-pool --full --batch-size=100
|
||||
```
|
||||
|
||||
3. **Schedule für regelmäßige Syncs:**
|
||||
|
||||
Füge in `routes/console.php` hinzu:
|
||||
```php
|
||||
use App\Jobs\SyncBackendDataPool;
|
||||
use Illuminate\Support\Facades\Schedule;
|
||||
|
||||
// Täglich um 2 Uhr incremental
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->dailyAt('02:00');
|
||||
|
||||
// Sonntags um 3 Uhr full sync
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->weeklyOn(0, '03:00');
|
||||
```
|
||||
|
||||
Dann starte den Scheduler:
|
||||
```bash
|
||||
php artisan schedule:work
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Logs prüfen:
|
||||
|
||||
```bash
|
||||
tail -f storage/logs/laravel.log
|
||||
```
|
||||
|
||||
### SQL Queries für Monitoring:
|
||||
|
||||
```sql
|
||||
-- Letzte Sync Zeit
|
||||
SELECT MAX(synced_at) FROM public.backend_data_pool;
|
||||
|
||||
-- Anzahl Datensätze pro Status
|
||||
SELECT status, COUNT(*)
|
||||
FROM public.backend_data_pool
|
||||
GROUP BY status;
|
||||
|
||||
-- Anzahl Datensätze pro corporate_entity (Top 10)
|
||||
SELECT corporate_entity, COUNT(*) as transaction_count
|
||||
FROM public.backend_data_pool
|
||||
GROUP BY corporate_entity
|
||||
ORDER BY transaction_count DESC
|
||||
LIMIT 10;
|
||||
|
||||
-- Durchschnittlicher Transaktionsbetrag
|
||||
SELECT AVG(tx_amount) as avg_amount
|
||||
FROM public.backend_data_pool;
|
||||
```
|
||||
|
||||
## Nächste Schritte
|
||||
|
||||
Nach erfolgreichem Sync kannst du mit **Stufe 2** weitermachen:
|
||||
- Transformation der Daten in `public.companies` und `public.transactions`
|
||||
- Mit KYC Risk Level Berechnung
|
||||
- Automatisches Enrichment der Companies
|
||||
|
||||
---
|
||||
|
||||
**Bei Fragen oder Problemen, prüfe die Logs in `storage/logs/laravel.log`**
|
||||
Reference in New Issue
Block a user