AFC-Demo/misc/MIGRATION_PLAN.md

Migrationsplan: Schema-Restrukturierung

Zielsetzung

Aktueller Zustand

public.companies (11 Spalten)
    ↓ 1:N
public.transactions (49 Spalten) - Angereicherte Produktionsdaten

backend.transactions (14 Spalten) - Rohdaten
    ↓ N:M
backend.transaction_outputs (5 Spalten) - AI-generierte Outputs

Ziel-Zustand

backend.transactions (14 Spalten) - Ersetzt public.companies
    ↓ 1:N
public.transactions (5 Spalten, ähnlich backend.transaction_outputs) - Vereinfacht

---

⚠️ Klärungsfragen (KRITISCH)

Bevor wir fortfahren, müssen folgende Fragen geklärt werden:

1. Companies → backend.transactions Mapping

Problem: Die Strukturen sind sehr unterschiedlich

public.companies	backend.transactions	Kompatibilität
id (bigint)	id (integer)	⚠️ Typ-Unterschied
name	corporate_entity	✅ Ähnlich
legal_name	-	❌ Fehlt
ticker	-	❌ Fehlt
sector	-	❌ Fehlt
country	tx_country_outgoing?	⚠️ Unklar
headquarters	-	❌ Fehlt
kyc_risk_level	-	❌ Fehlt
summary	-	❌ Fehlt
-	corporate_counterparty	❌ Neu
-	tx_date, tx_amount, tx_currency	❌ Neu
-	tx_purpose	❌ Neu
-	source_file, raw_payload	❌ Neu
-	status	❌ Neu

Frage:

• Soll backend.transactions erweitert werden, um Company-Felder aufzunehmen?
• Oder sollen Companies als einzelne Zeilen ohne Transaktionsdaten gespeichert werden?
• Wie wird corporate_entity zu companies.name gemappt?

2. public.transactions → transaction_outputs Mapping

Problem: Drastischer Datenverlust bei Vereinfachung

public.transactions (49 Felder)	backend.transaction_outputs (5 Felder)
Alle Core-Felder (14)	❌ Verloren
Alle Enrichment-Felder (34)	❌ Verloren
-	transaction_id (Foreign Key)
-	prompt_id (Welcher?)
-	output_key (Welcher Typ?)
-	content (Wie strukturiert?)
-	run_id (Optional)

Frage:

• Welche Daten aus den 49 Feldern sollen in content serialisiert werden?
• Welchen output_key verwenden wir? (z.B. "transaction_data", "risk_assessment"?)
• Welchen prompt_id verwenden wir? (Muss in backend.prompt_templates existieren)
• Was passiert mit den 11 Enrichment-Datenquellen?

3. Relationship & Foreign Keys

Problem: Beziehungen ändern sich fundamental

Aktuell:

companies.id → transactions.company_id (1:N)

Ziel (unklar):

backend.transactions.id → public.transactions.transaction_id (1:N)?

Frage:

• Bleibt die 1:N Beziehung erhalten?
• Wie wird transaction_id in der neuen public.transactions gemappt?

---

Vorgeschlagene Alternative: Erweiterte Migration

Ich schlage eine modifizierte Zielstruktur vor, die Datenverlust minimiert:

Option A: Erweitere backend.transactions (Empfohlen)

backend.companies (neue Tabelle)
    - id (integer)
    - name (text)
    - legal_name (text, nullable)
    - country (text)
    - kyc_risk_level (text)
    - ... weitere Company-Felder

backend.transactions (erweitert, bleibt)
    - id (integer)
    - company_id (integer FK → backend.companies)
    - corporate_counterparty (text)
    - tx_date (text → sollte timestamp werden)
    - tx_amount (double precision)
    - tx_currency (text)
    - ... bestehende Felder

public.transaction_enrichments (neue Tabelle)
    - id (bigint)
    - transaction_id (integer FK → backend.transactions)
    - enrichment_type (varchar) -- 'registry', 'sanctions', etc.
    - data (jsonb)
    - last_refreshed_at (timestamp)
    - created_at, updated_at

Vorteile:

• ✅ Kein Datenverlust
• ✅ Klare Trennung: Companies, Transactions, Enrichments
• ✅ Backend-Schema behält Rohdaten
• ✅ Public-Schema behält angereicherte Daten
• ✅ Laravel-Logik kann schrittweise migriert werden

Option B: Vollständiger Umzug zu backend Schema

backend.companies (neu)
    - Alle Felder von public.companies

backend.transactions (bleibt)
    - Bestehende Struktur

backend.transaction_enrichments (neu)
    - Alle Enrichment-Daten aus public.transactions

public.* (deprecated, später löschen)

Vorteile:

• ✅ Alles im backend Schema
• ✅ Klare Schema-Trennung
• ❌ Laravel-App muss komplett umgeschrieben werden
• ❌ Größere Breaking Changes

---

Migrationsplan (nach Klärung)

Phase 1: Vorbereitung (1-2 Tage)

1.1 Backup erstellen

# Vollständiges Backup
pg_dump -h localhost -U username -d database_name > backup_$(date +%Y%m%d_%H%M%S).sql

# Schema-spezifische Backups
pg_dump -h localhost -U username -d database_name -n public > backup_public_$(date +%Y%m%d).sql
pg_dump -h localhost -U username -d database_name -n backend > backup_backend_$(date +%Y%m%d).sql

1.2 Datenanalyse

-- Anzahl Companies
SELECT COUNT(*) FROM public.companies;

-- Anzahl Transactions
SELECT COUNT(*) FROM public.transactions;

-- Datenintegrität prüfen
SELECT
    COUNT(*) as total_transactions,
    COUNT(DISTINCT company_id) as unique_companies,
    COUNT(*) FILTER (WHERE company_id NOT IN (SELECT id FROM public.companies)) as orphaned_transactions
FROM public.transactions;

-- Enrichment-Daten Analyse
SELECT
    COUNT(*) FILTER (WHERE registry_data IS NOT NULL) as has_registry,
    COUNT(*) FILTER (WHERE sanctions_data IS NOT NULL) as has_sanctions,
    COUNT(*) FILTER (WHERE pep_data IS NOT NULL) as has_pep
FROM public.transactions;

1.3 Test-Umgebung aufsetzen

# Kopie der Datenbank für Tests
createdb -T production_db test_migration_db

Phase 2: Schema-Erweiterung (2-3 Tage)

2.1 backend.companies erstellen

Laravel Migration:

<?php
// database/migrations/2025_11_12_create_backend_companies_table.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::connection('backend')->create('companies', function (Blueprint $table) {
            $table->id();
            $table->string('name');
            $table->string('legal_name')->nullable();
            $table->string('ticker')->nullable();
            $table->string('sector')->nullable();
            $table->string('country', 2)->default('DE');
            $table->string('headquarters')->nullable();
            $table->string('kyc_risk_level')->default('medium');
            $table->text('summary')->nullable();
            $table->timestamps();

            $table->index('name');
        });
    }

    public function down(): void
    {
        Schema::connection('backend')->dropIfExists('companies');
    }
};

2.2 backend.transactions erweitern

SQL Migration (wenn Laravel Multi-Schema-Support limitiert ist):

-- Füge company_id zu backend.transactions hinzu
ALTER TABLE backend.transactions
ADD COLUMN company_id INTEGER REFERENCES backend.companies(id) ON DELETE CASCADE;

-- Index für Performance
CREATE INDEX idx_backend_transactions_company_id ON backend.transactions(company_id);

2.3 public.transaction_enrichments erstellen

Laravel Migration:

<?php
// database/migrations/2025_11_12_create_transaction_enrichments_table.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::create('transaction_enrichments', function (Blueprint $table) {
            $table->id();
            $table->integer('backend_transaction_id'); // FK zu backend.transactions
            $table->string('enrichment_type', 50); // 'registry', 'sanctions', etc.
            $table->jsonb('data');
            $table->timestamp('last_refreshed_at')->nullable();
            $table->timestamps();

            $table->index(['backend_transaction_id', 'enrichment_type']);
            $table->unique(['backend_transaction_id', 'enrichment_type']);
        });
    }

    public function down(): void
    {
        Schema::dropIfExists('transaction_enrichments');
    }
};

Phase 3: Datenmigration (3-5 Tage)

3.1 Companies migrieren

<?php
// database/migrations/2025_11_12_migrate_companies_data.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;

return new class extends Migration
{
    public function up(): void
    {
        // Kopiere Companies von public → backend
        DB::statement("
            INSERT INTO backend.companies (
                id, name, legal_name, ticker, sector,
                country, headquarters, kyc_risk_level, summary,
                created_at, updated_at
            )
            SELECT
                id, name, legal_name, ticker, sector,
                country, headquarters, kyc_risk_level, summary,
                created_at, updated_at
            FROM public.companies
            ON CONFLICT (id) DO NOTHING
        ");

        // Aktualisiere Sequence
        DB::statement("
            SELECT setval('backend.companies_id_seq',
                (SELECT MAX(id) FROM backend.companies)
            )
        ");
    }

    public function down(): void
    {
        DB::statement("TRUNCATE backend.companies CASCADE");
    }
};

3.2 Transaction Core-Daten migrieren

<?php
// database/migrations/2025_11_12_migrate_transaction_core_data.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;

return new class extends Migration
{
    public function up(): void
    {
        // Füge company_id zu bestehenden backend.transactions hinzu
        // (falls diese Daten schon existieren und verknüpft werden müssen)

        // ODER: Migriere von public.transactions → backend.transactions
        DB::statement("
            INSERT INTO backend.transactions (
                corporate_entity,
                corporate_counterparty,
                tx_date,
                tx_amount,
                tx_currency,
                tx_purpose,
                tx_country_incoming,
                company_id,
                status,
                created_at,
                last_modified_at
            )
            SELECT
                c.name as corporate_entity,
                t.counterparty as corporate_counterparty,
                t.executed_at::text as tx_date,
                t.amount as tx_amount,
                t.currency as tx_currency,
                t.flagged_reason as tx_purpose,
                t.counterparty_country as tx_country_incoming,
                t.company_id,
                t.status,
                t.created_at::text,
                t.updated_at::text
            FROM public.transactions t
            JOIN public.companies c ON t.company_id = c.id
            WHERE NOT EXISTS (
                SELECT 1 FROM backend.transactions bt
                WHERE bt.corporate_entity = c.name
                AND bt.tx_date = t.executed_at::text
            )
        ");
    }

    public function down(): void
    {
        // Rollback nur für migrierte Daten
        DB::statement("
            DELETE FROM backend.transactions
            WHERE company_id IS NOT NULL
        ");
    }
};

3.3 Enrichment-Daten migrieren

<?php
// database/migrations/2025_11_12_migrate_enrichment_data.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;

return new class extends Migration
{
    private array $enrichmentSources = [
        'registry' => ['registry_data', 'registry_last_refreshed_at'],
        'genesis' => ['genesis_context', 'genesis_last_refreshed_at'],
        'govdata' => ['govdata_data', 'govdata_last_refreshed_at'],
        'bundesanzeiger' => ['bundesanzeiger_data', 'bundesanzeiger_last_refreshed_at'],
        'insolvency' => ['insolvency_data', 'insolvency_last_refreshed_at'],
        'rss' => ['rss_alerts', 'rss_last_refreshed_at'],
        'sanctions' => ['sanctions_data', 'sanctions_last_refreshed_at'],
        'pep' => ['pep_data', 'pep_last_refreshed_at'],
        'gleif' => ['gleif_data', 'gleif_last_refreshed_at'],
        'eu_sanctions' => ['eu_sanctions_data', 'eu_sanctions_last_refreshed_at'],
        'handelsregister' => ['handelsregister_data', 'handelsregister_last_refreshed_at'],
    ];

    public function up(): void
    {
        foreach ($this->enrichmentSources as $type => [$dataField, $refreshField]) {
            DB::statement("
                INSERT INTO public.transaction_enrichments (
                    backend_transaction_id,
                    enrichment_type,
                    data,
                    last_refreshed_at,
                    created_at,
                    updated_at
                )
                SELECT
                    bt.id as backend_transaction_id,
                    '{$type}' as enrichment_type,
                    t.{$dataField} as data,
                    t.{$refreshField} as last_refreshed_at,
                    NOW(),
                    NOW()
                FROM public.transactions t
                JOIN backend.transactions bt ON (
                    bt.tx_date = t.executed_at::text
                    AND bt.tx_amount = t.amount
                )
                WHERE t.{$dataField} IS NOT NULL
                ON CONFLICT (backend_transaction_id, enrichment_type) DO NOTHING
            ");
        }
    }

    public function down(): void
    {
        DB::statement("TRUNCATE public.transaction_enrichments");
    }
};

Phase 4: Model-Anpassung (2-3 Tage)

4.1 Neue Models erstellen

Backend\Company Model:

<?php
// app/Models/Backend/Company.php

namespace App\Models\Backend;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\HasMany;

class Company extends Model
{
    protected $connection = 'backend';
    protected $table = 'companies';

    protected $fillable = [
        'name', 'legal_name', 'ticker', 'sector',
        'country', 'headquarters', 'kyc_risk_level', 'summary'
    ];

    public function transactions(): HasMany
    {
        return $this->hasMany(Transaction::class);
    }
}

Backend\Transaction Model:

<?php
// app/Models/Backend/Transaction.php

namespace App\Models\Backend;

use App\Models\TransactionEnrichment;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\BelongsTo;
use Illuminate\Database\Eloquent\Relations\HasMany;

class Transaction extends Model
{
    protected $connection = 'backend';
    protected $table = 'transactions';

    protected $fillable = [
        'company_id', 'corporate_entity', 'corporate_counterparty',
        'tx_date', 'tx_amount', 'tx_currency', 'tx_purpose',
        'tx_country_outgoing', 'tx_country_incoming',
        'source_file', 'raw_payload', 'status'
    ];

    protected $casts = [
        'tx_amount' => 'decimal:2',
    ];

    public function company(): BelongsTo
    {
        return $this->belongsTo(Company::class);
    }

    public function enrichments(): HasMany
    {
        return $this->hasMany(TransactionEnrichment::class, 'backend_transaction_id');
    }

    public function getEnrichment(string $type): ?array
    {
        return $this->enrichments()
            ->where('enrichment_type', $type)
            ->first()
            ?->data;
    }
}

TransactionEnrichment Model:

<?php
// app/Models/TransactionEnrichment.php

namespace App\Models;

use App\Models\Backend\Transaction;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\BelongsTo;

class TransactionEnrichment extends Model
{
    protected $fillable = [
        'backend_transaction_id',
        'enrichment_type',
        'data',
        'last_refreshed_at'
    ];

    protected $casts = [
        'data' => 'array',
        'last_refreshed_at' => 'datetime'
    ];

    public function transaction(): BelongsTo
    {
        return $this->belongsTo(Transaction::class, 'backend_transaction_id');
    }
}

Phase 5: Code-Refactoring (5-7 Tage)

5.1 Controller anpassen

Vorher:

use App\Models\Transaction;

$transactions = Transaction::with('company')
    ->where('risk_score', '>', 50)
    ->get();

Nachher:

use App\Models\Backend\Transaction;

$transactions = Transaction::with(['company', 'enrichments'])
    ->where('risk_score', '>', 50)
    ->get();

// Enrichment-Daten abrufen
foreach ($transactions as $transaction) {
    $sanctionsData = $transaction->getEnrichment('sanctions');
    $pepData = $transaction->getEnrichment('pep');
}

5.2 Compatibility Layer (Optional)

<?php
// app/Models/Transaction.php (Legacy-Support)

namespace App\Models;

use App\Models\Backend\Transaction as BackendTransaction;

/**
 * @deprecated Use App\Models\Backend\Transaction instead
 */
class Transaction extends BackendTransaction
{
    // Proxy zu neuem Model für Backward-Compatibility

    public function __get($key)
    {
        // Prüfe ob Enrichment-Feld
        if (str_ends_with($key, '_data')) {
            $type = str_replace('_data', '', $key);
            return $this->getEnrichment($type);
        }

        return parent::__get($key);
    }
}

Phase 6: Testing (3-5 Tage)

6.1 Unit Tests

<?php
// tests/Unit/Models/Backend/TransactionTest.php

use App\Models\Backend\Transaction;
use App\Models\TransactionEnrichment;

test('transaction has enrichments relationship', function () {
    $transaction = Transaction::factory()->create();

    TransactionEnrichment::factory()->create([
        'backend_transaction_id' => $transaction->id,
        'enrichment_type' => 'sanctions',
        'data' => ['status' => 'clear']
    ]);

    expect($transaction->enrichments)->toHaveCount(1);
    expect($transaction->getEnrichment('sanctions'))->toBe(['status' => 'clear']);
});

6.2 Integration Tests

<?php
// tests/Feature/TransactionMigrationTest.php

test('data integrity after migration', function () {
    // Vergleiche Anzahl
    $oldCount = DB::table('public.transactions')->count();
    $newCount = DB::table('backend.transactions')->count();

    expect($newCount)->toBeGreaterThanOrEqual($oldCount);

    // Vergleiche Enrichment-Daten
    $oldEnrichments = DB::table('public.transactions')
        ->whereNotNull('sanctions_data')
        ->count();

    $newEnrichments = DB::table('public.transaction_enrichments')
        ->where('enrichment_type', 'sanctions')
        ->count();

    expect($newEnrichments)->toBe($oldEnrichments);
});

Phase 7: Deployment (1-2 Tage)

7.1 Deployment-Schritte

# 1. Backup
php artisan backup:database

# 2. Migrations ausführen (in Reihenfolge!)
php artisan migrate --path=database/migrations/2025_11_12_create_backend_companies_table.php
php artisan migrate --path=database/migrations/2025_11_12_create_transaction_enrichments_table.php
php artisan migrate --path=database/migrations/2025_11_12_migrate_companies_data.php
php artisan migrate --path=database/migrations/2025_11_12_migrate_transaction_core_data.php
php artisan migrate --path=database/migrations/2025_11_12_migrate_enrichment_data.php

# 3. Verification
php artisan tinker
>>> DB::table('backend.companies')->count()
>>> DB::table('backend.transactions')->count()
>>> DB::table('public.transaction_enrichments')->count()

# 4. Clear caches
php artisan cache:clear
php artisan config:clear
php artisan route:clear
php artisan view:clear

# 5. Run tests
php artisan test --filter=TransactionMigration

7.2 Rollback-Plan

# Falls etwas schief geht
php artisan migrate:rollback --step=5

# Restore from backup
psql -U username -d database_name < backup_20251112_120000.sql

Phase 8: Cleanup (nach 2-4 Wochen Monitoring)

-- Wenn alles stabil läuft, alte Tabellen entfernen
DROP TABLE public.transactions CASCADE;
DROP TABLE public.companies CASCADE;

-- Views für Backward-Compatibility (optional)
CREATE VIEW public.companies AS
SELECT * FROM backend.companies;

CREATE VIEW public.transactions AS
SELECT
    bt.id,
    bt.company_id,
    bt.corporate_counterparty as counterparty,
    bt.tx_amount as amount,
    bt.tx_currency as currency,
    bt.tx_date::timestamp as executed_at,
    bt.status,
    bt.created_at::timestamp,
    bt.last_modified_at::timestamp as updated_at
FROM backend.transactions bt;

---

Zeitplan

Phase	Dauer	Abhängigkeiten
1. Vorbereitung	1-2 Tage	-
2. Schema-Erweiterung	2-3 Tage	Phase 1
3. Datenmigration	3-5 Tage	Phase 2
4. Model-Anpassung	2-3 Tage	Phase 3
5. Code-Refactoring	5-7 Tage	Phase 4
6. Testing	3-5 Tage	Phase 5
7. Deployment	1-2 Tage	Phase 6
8. Cleanup	Nach 2-4 Wochen	Phase 7

Gesamtdauer: 17-27 Arbeitstage (3-5 Wochen)

---

Risiken & Mitigation

Risiko 1: Datenverlust

Mitigation:

• Vollständige Backups vor jedem Schritt
• Test-Migration in Staging-Umgebung
• Datenvalidierung nach jeder Phase

Risiko 2: Downtime

Mitigation:

• Migrations während Wartungsfenster
• Blue-Green Deployment
• Read-Replica für Zero-Downtime

Risiko 3: Performance-Probleme

Mitigation:

• Indizes auf Foreign Keys
• Batch-Processing für große Datasets
• Query-Optimierung mit EXPLAIN ANALYZE

Risiko 4: Code-Inkompatibilität

Mitigation:

• Compatibility Layer
• Schrittweises Refactoring
• Feature-Flags für graduelle Umstellung

---

Nächste Schritte

1. ✅ Klärung der Fragen oben
2. ⬜ Detaillierte Datenanalyse durchführen
3. ⬜ Test-Umgebung aufsetzen
4. ⬜ Erste Migration in Staging testen
5. ⬜ Review & Approval vom Team
6. ⬜ Production-Migration planen

---

Erstellt am: 2025-11-12 Status: ENTWURF - Wartet auf Klärung der kritischen Fragen