AFC-Demo/misc/MIGRATION_PLAN.md

# Migrationsplan: Schema-Restrukturierung

## Zielsetzung

### Aktueller Zustand
```
public.companies (11 Spalten)
    ↓ 1:N
public.transactions (49 Spalten) - Angereicherte Produktionsdaten

backend.transactions (14 Spalten) - Rohdaten
    ↓ N:M
backend.transaction_outputs (5 Spalten) - AI-generierte Outputs
```

### Ziel-Zustand
```
backend.transactions (14 Spalten) - Ersetzt public.companies
    ↓ 1:N
public.transactions (5 Spalten, ähnlich backend.transaction_outputs) - Vereinfacht
```

---

## ⚠️ Klärungsfragen (KRITISCH)

Bevor wir fortfahren, müssen folgende Fragen geklärt werden:

### 1. Companies → backend.transactions Mapping
**Problem**: Die Strukturen sind sehr unterschiedlich

| public.companies | backend.transactions | Kompatibilität |
|------------------|---------------------|----------------|
| id (bigint) | id (integer) | ⚠️ Typ-Unterschied |
| name | corporate_entity | ✅ Ähnlich |
| legal_name | - | ❌ Fehlt |
| ticker | - | ❌ Fehlt |
| sector | - | ❌ Fehlt |
| country | tx_country_outgoing? | ⚠️ Unklar |
| headquarters | - | ❌ Fehlt |
| kyc_risk_level | - | ❌ Fehlt |
| summary | - | ❌ Fehlt |
| - | corporate_counterparty | ❌ Neu |
| - | tx_date, tx_amount, tx_currency | ❌ Neu |
| - | tx_purpose | ❌ Neu |
| - | source_file, raw_payload | ❌ Neu |
| - | status | ❌ Neu |

**Frage**:
- Soll `backend.transactions` erweitert werden, um Company-Felder aufzunehmen?
- Oder sollen Companies als einzelne Zeilen ohne Transaktionsdaten gespeichert werden?
- Wie wird `corporate_entity` zu `companies.name` gemappt?

### 2. public.transactions → transaction_outputs Mapping
**Problem**: Drastischer Datenverlust bei Vereinfachung

| public.transactions (49 Felder) | backend.transaction_outputs (5 Felder) |
|---------------------------------|----------------------------------------|
| Alle Core-Felder (14) | ❌ Verloren |
| Alle Enrichment-Felder (34) | ❌ Verloren |
| - | transaction_id (Foreign Key) |
| - | prompt_id (Welcher?) |
| - | output_key (Welcher Typ?) |
| - | content (Wie strukturiert?) |
| - | run_id (Optional) |

**Frage**:
- Welche Daten aus den 49 Feldern sollen in `content` serialisiert werden?
- Welchen `output_key` verwenden wir? (z.B. "transaction_data", "risk_assessment"?)
- Welchen `prompt_id` verwenden wir? (Muss in `backend.prompt_templates` existieren)
- Was passiert mit den 11 Enrichment-Datenquellen?

### 3. Relationship & Foreign Keys
**Problem**: Beziehungen ändern sich fundamental

**Aktuell**:
```
companies.id → transactions.company_id (1:N)
```

**Ziel** (unklar):
```
backend.transactions.id → public.transactions.transaction_id (1:N)?
```

**Frage**:
- Bleibt die 1:N Beziehung erhalten?
- Wie wird `transaction_id` in der neuen `public.transactions` gemappt?

---

## Vorgeschlagene Alternative: Erweiterte Migration

Ich schlage eine modifizierte Zielstruktur vor, die Datenverlust minimiert:

### Option A: Erweitere backend.transactions (Empfohlen)

```sql
backend.companies (neue Tabelle)
    - id (integer)
    - name (text)
    - legal_name (text, nullable)
    - country (text)
    - kyc_risk_level (text)
    - ... weitere Company-Felder

backend.transactions (erweitert, bleibt)
    - id (integer)
    - company_id (integer FK → backend.companies)
    - corporate_counterparty (text)
    - tx_date (text → sollte timestamp werden)
    - tx_amount (double precision)
    - tx_currency (text)
    - ... bestehende Felder

public.transaction_enrichments (neue Tabelle)
    - id (bigint)
    - transaction_id (integer FK → backend.transactions)
    - enrichment_type (varchar) -- 'registry', 'sanctions', etc.
    - data (jsonb)
    - last_refreshed_at (timestamp)
    - created_at, updated_at
```

**Vorteile**:
- ✅ Kein Datenverlust
- ✅ Klare Trennung: Companies, Transactions, Enrichments
- ✅ Backend-Schema behält Rohdaten
- ✅ Public-Schema behält angereicherte Daten
- ✅ Laravel-Logik kann schrittweise migriert werden

### Option B: Vollständiger Umzug zu backend Schema

```sql
backend.companies (neu)
    - Alle Felder von public.companies

backend.transactions (bleibt)
    - Bestehende Struktur

backend.transaction_enrichments (neu)
    - Alle Enrichment-Daten aus public.transactions

public.* (deprecated, später löschen)
```

**Vorteile**:
- ✅ Alles im backend Schema
- ✅ Klare Schema-Trennung
- ❌ Laravel-App muss komplett umgeschrieben werden
- ❌ Größere Breaking Changes

---

## Migrationsplan (nach Klärung)

### Phase 1: Vorbereitung (1-2 Tage)

#### 1.1 Backup erstellen
```bash
# Vollständiges Backup
pg_dump -h localhost -U username -d database_name > backup_$(date +%Y%m%d_%H%M%S).sql

# Schema-spezifische Backups
pg_dump -h localhost -U username -d database_name -n public > backup_public_$(date +%Y%m%d).sql
pg_dump -h localhost -U username -d database_name -n backend > backup_backend_$(date +%Y%m%d).sql
```

#### 1.2 Datenanalyse
```sql
-- Anzahl Companies
SELECT COUNT(*) FROM public.companies;

-- Anzahl Transactions
SELECT COUNT(*) FROM public.transactions;

-- Datenintegrität prüfen
SELECT
    COUNT(*) as total_transactions,
    COUNT(DISTINCT company_id) as unique_companies,
    COUNT(*) FILTER (WHERE company_id NOT IN (SELECT id FROM public.companies)) as orphaned_transactions
FROM public.transactions;

-- Enrichment-Daten Analyse
SELECT
    COUNT(*) FILTER (WHERE registry_data IS NOT NULL) as has_registry,
    COUNT(*) FILTER (WHERE sanctions_data IS NOT NULL) as has_sanctions,
    COUNT(*) FILTER (WHERE pep_data IS NOT NULL) as has_pep
FROM public.transactions;
```

#### 1.3 Test-Umgebung aufsetzen
```bash
# Kopie der Datenbank für Tests
createdb -T production_db test_migration_db
```

### Phase 2: Schema-Erweiterung (2-3 Tage)

#### 2.1 backend.companies erstellen

**Laravel Migration**:
```php
<?php
// database/migrations/2025_11_12_create_backend_companies_table.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::connection('backend')->create('companies', function (Blueprint $table) {
            $table->id();
            $table->string('name');
            $table->string('legal_name')->nullable();
            $table->string('ticker')->nullable();
            $table->string('sector')->nullable();
            $table->string('country', 2)->default('DE');
            $table->string('headquarters')->nullable();
            $table->string('kyc_risk_level')->default('medium');
            $table->text('summary')->nullable();
            $table->timestamps();

            $table->index('name');
        });
    }

    public function down(): void
    {
        Schema::connection('backend')->dropIfExists('companies');
    }
};
```

#### 2.2 backend.transactions erweitern

**SQL Migration** (wenn Laravel Multi-Schema-Support limitiert ist):
```sql
-- Füge company_id zu backend.transactions hinzu
ALTER TABLE backend.transactions
ADD COLUMN company_id INTEGER REFERENCES backend.companies(id) ON DELETE CASCADE;

-- Index für Performance
CREATE INDEX idx_backend_transactions_company_id ON backend.transactions(company_id);
```

#### 2.3 public.transaction_enrichments erstellen

**Laravel Migration**:
```php
<?php
// database/migrations/2025_11_12_create_transaction_enrichments_table.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::create('transaction_enrichments', function (Blueprint $table) {
            $table->id();
            $table->integer('backend_transaction_id'); // FK zu backend.transactions
            $table->string('enrichment_type', 50); // 'registry', 'sanctions', etc.
            $table->jsonb('data');
            $table->timestamp('last_refreshed_at')->nullable();
            $table->timestamps();

            $table->index(['backend_transaction_id', 'enrichment_type']);
            $table->unique(['backend_transaction_id', 'enrichment_type']);
        });
    }

    public function down(): void
    {
        Schema::dropIfExists('transaction_enrichments');
    }
};
```

### Phase 3: Datenmigration (3-5 Tage)

#### 3.1 Companies migrieren

```php
<?php
// database/migrations/2025_11_12_migrate_companies_data.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;

return new class extends Migration
{
    public function up(): void
    {
        // Kopiere Companies von public → backend
        DB::statement("
            INSERT INTO backend.companies (
                id, name, legal_name, ticker, sector,
                country, headquarters, kyc_risk_level, summary,
                created_at, updated_at
            )
            SELECT
                id, name, legal_name, ticker, sector,
                country, headquarters, kyc_risk_level, summary,
                created_at, updated_at
            FROM public.companies
            ON CONFLICT (id) DO NOTHING
        ");

        // Aktualisiere Sequence
        DB::statement("
            SELECT setval('backend.companies_id_seq',
                (SELECT MAX(id) FROM backend.companies)
            )
        ");
    }

    public function down(): void
    {
        DB::statement("TRUNCATE backend.companies CASCADE");
    }
};
```

#### 3.2 Transaction Core-Daten migrieren

```php
<?php
// database/migrations/2025_11_12_migrate_transaction_core_data.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;

return new class extends Migration
{
    public function up(): void
    {
        // Füge company_id zu bestehenden backend.transactions hinzu
        // (falls diese Daten schon existieren und verknüpft werden müssen)

        // ODER: Migriere von public.transactions → backend.transactions
        DB::statement("
            INSERT INTO backend.transactions (
                corporate_entity,
                corporate_counterparty,
                tx_date,
                tx_amount,
                tx_currency,
                tx_purpose,
                tx_country_incoming,
                company_id,
                status,
                created_at,
                last_modified_at
            )
            SELECT
                c.name as corporate_entity,
                t.counterparty as corporate_counterparty,
                t.executed_at::text as tx_date,
                t.amount as tx_amount,
                t.currency as tx_currency,
                t.flagged_reason as tx_purpose,
                t.counterparty_country as tx_country_incoming,
                t.company_id,
                t.status,
                t.created_at::text,
                t.updated_at::text
            FROM public.transactions t
            JOIN public.companies c ON t.company_id = c.id
            WHERE NOT EXISTS (
                SELECT 1 FROM backend.transactions bt
                WHERE bt.corporate_entity = c.name
                AND bt.tx_date = t.executed_at::text
            )
        ");
    }

    public function down(): void
    {
        // Rollback nur für migrierte Daten
        DB::statement("
            DELETE FROM backend.transactions
            WHERE company_id IS NOT NULL
        ");
    }
};
```

#### 3.3 Enrichment-Daten migrieren

```php
<?php
// database/migrations/2025_11_12_migrate_enrichment_data.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;

return new class extends Migration
{
    private array $enrichmentSources = [
        'registry' => ['registry_data', 'registry_last_refreshed_at'],
        'genesis' => ['genesis_context', 'genesis_last_refreshed_at'],
        'govdata' => ['govdata_data', 'govdata_last_refreshed_at'],
        'bundesanzeiger' => ['bundesanzeiger_data', 'bundesanzeiger_last_refreshed_at'],
        'insolvency' => ['insolvency_data', 'insolvency_last_refreshed_at'],
        'rss' => ['rss_alerts', 'rss_last_refreshed_at'],
        'sanctions' => ['sanctions_data', 'sanctions_last_refreshed_at'],
        'pep' => ['pep_data', 'pep_last_refreshed_at'],
        'gleif' => ['gleif_data', 'gleif_last_refreshed_at'],
        'eu_sanctions' => ['eu_sanctions_data', 'eu_sanctions_last_refreshed_at'],
        'handelsregister' => ['handelsregister_data', 'handelsregister_last_refreshed_at'],
    ];

    public function up(): void
    {
        foreach ($this->enrichmentSources as $type => [$dataField, $refreshField]) {
            DB::statement("
                INSERT INTO public.transaction_enrichments (
                    backend_transaction_id,
                    enrichment_type,
                    data,
                    last_refreshed_at,
                    created_at,
                    updated_at
                )
                SELECT
                    bt.id as backend_transaction_id,
                    '{$type}' as enrichment_type,
                    t.{$dataField} as data,
                    t.{$refreshField} as last_refreshed_at,
                    NOW(),
                    NOW()
                FROM public.transactions t
                JOIN backend.transactions bt ON (
                    bt.tx_date = t.executed_at::text
                    AND bt.tx_amount = t.amount
                )
                WHERE t.{$dataField} IS NOT NULL
                ON CONFLICT (backend_transaction_id, enrichment_type) DO NOTHING
            ");
        }
    }

    public function down(): void
    {
        DB::statement("TRUNCATE public.transaction_enrichments");
    }
};
```

### Phase 4: Model-Anpassung (2-3 Tage)

#### 4.1 Neue Models erstellen

**Backend\Company Model**:
```php
<?php
// app/Models/Backend/Company.php

namespace App\Models\Backend;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\HasMany;

class Company extends Model
{
    protected $connection = 'backend';
    protected $table = 'companies';

    protected $fillable = [
        'name', 'legal_name', 'ticker', 'sector',
        'country', 'headquarters', 'kyc_risk_level', 'summary'
    ];

    public function transactions(): HasMany
    {
        return $this->hasMany(Transaction::class);
    }
}
```

**Backend\Transaction Model**:
```php
<?php
// app/Models/Backend/Transaction.php

namespace App\Models\Backend;

use App\Models\TransactionEnrichment;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\BelongsTo;
use Illuminate\Database\Eloquent\Relations\HasMany;

class Transaction extends Model
{
    protected $connection = 'backend';
    protected $table = 'transactions';

    protected $fillable = [
        'company_id', 'corporate_entity', 'corporate_counterparty',
        'tx_date', 'tx_amount', 'tx_currency', 'tx_purpose',
        'tx_country_outgoing', 'tx_country_incoming',
        'source_file', 'raw_payload', 'status'
    ];

    protected $casts = [
        'tx_amount' => 'decimal:2',
    ];

    public function company(): BelongsTo
    {
        return $this->belongsTo(Company::class);
    }

    public function enrichments(): HasMany
    {
        return $this->hasMany(TransactionEnrichment::class, 'backend_transaction_id');
    }

    public function getEnrichment(string $type): ?array
    {
        return $this->enrichments()
            ->where('enrichment_type', $type)
            ->first()
            ?->data;
    }
}
```

**TransactionEnrichment Model**:
```php
<?php
// app/Models/TransactionEnrichment.php

namespace App\Models;

use App\Models\Backend\Transaction;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\BelongsTo;

class TransactionEnrichment extends Model
{
    protected $fillable = [
        'backend_transaction_id',
        'enrichment_type',
        'data',
        'last_refreshed_at'
    ];

    protected $casts = [
        'data' => 'array',
        'last_refreshed_at' => 'datetime'
    ];

    public function transaction(): BelongsTo
    {
        return $this->belongsTo(Transaction::class, 'backend_transaction_id');
    }
}
```

### Phase 5: Code-Refactoring (5-7 Tage)

#### 5.1 Controller anpassen

**Vorher**:
```php
use App\Models\Transaction;

$transactions = Transaction::with('company')
    ->where('risk_score', '>', 50)
    ->get();
```

**Nachher**:
```php
use App\Models\Backend\Transaction;

$transactions = Transaction::with(['company', 'enrichments'])
    ->where('risk_score', '>', 50)
    ->get();

// Enrichment-Daten abrufen
foreach ($transactions as $transaction) {
    $sanctionsData = $transaction->getEnrichment('sanctions');
    $pepData = $transaction->getEnrichment('pep');
}
```

#### 5.2 Compatibility Layer (Optional)

```php
<?php
// app/Models/Transaction.php (Legacy-Support)

namespace App\Models;

use App\Models\Backend\Transaction as BackendTransaction;

/**
 * @deprecated Use App\Models\Backend\Transaction instead
 */
class Transaction extends BackendTransaction
{
    // Proxy zu neuem Model für Backward-Compatibility

    public function __get($key)
    {
        // Prüfe ob Enrichment-Feld
        if (str_ends_with($key, '_data')) {
            $type = str_replace('_data', '', $key);
            return $this->getEnrichment($type);
        }

        return parent::__get($key);
    }
}
```

### Phase 6: Testing (3-5 Tage)

#### 6.1 Unit Tests

```php
<?php
// tests/Unit/Models/Backend/TransactionTest.php

use App\Models\Backend\Transaction;
use App\Models\TransactionEnrichment;

test('transaction has enrichments relationship', function () {
    $transaction = Transaction::factory()->create();

    TransactionEnrichment::factory()->create([
        'backend_transaction_id' => $transaction->id,
        'enrichment_type' => 'sanctions',
        'data' => ['status' => 'clear']
    ]);

    expect($transaction->enrichments)->toHaveCount(1);
    expect($transaction->getEnrichment('sanctions'))->toBe(['status' => 'clear']);
});
```

#### 6.2 Integration Tests

```php
<?php
// tests/Feature/TransactionMigrationTest.php

test('data integrity after migration', function () {
    // Vergleiche Anzahl
    $oldCount = DB::table('public.transactions')->count();
    $newCount = DB::table('backend.transactions')->count();

    expect($newCount)->toBeGreaterThanOrEqual($oldCount);

    // Vergleiche Enrichment-Daten
    $oldEnrichments = DB::table('public.transactions')
        ->whereNotNull('sanctions_data')
        ->count();

    $newEnrichments = DB::table('public.transaction_enrichments')
        ->where('enrichment_type', 'sanctions')
        ->count();

    expect($newEnrichments)->toBe($oldEnrichments);
});
```

### Phase 7: Deployment (1-2 Tage)

#### 7.1 Deployment-Schritte

```bash
# 1. Backup
php artisan backup:database

# 2. Migrations ausführen (in Reihenfolge!)
php artisan migrate --path=database/migrations/2025_11_12_create_backend_companies_table.php
php artisan migrate --path=database/migrations/2025_11_12_create_transaction_enrichments_table.php
php artisan migrate --path=database/migrations/2025_11_12_migrate_companies_data.php
php artisan migrate --path=database/migrations/2025_11_12_migrate_transaction_core_data.php
php artisan migrate --path=database/migrations/2025_11_12_migrate_enrichment_data.php

# 3. Verification
php artisan tinker
>>> DB::table('backend.companies')->count()
>>> DB::table('backend.transactions')->count()
>>> DB::table('public.transaction_enrichments')->count()

# 4. Clear caches
php artisan cache:clear
php artisan config:clear
php artisan route:clear
php artisan view:clear

# 5. Run tests
php artisan test --filter=TransactionMigration
```

#### 7.2 Rollback-Plan

```bash
# Falls etwas schief geht
php artisan migrate:rollback --step=5

# Restore from backup
psql -U username -d database_name < backup_20251112_120000.sql
```

### Phase 8: Cleanup (nach 2-4 Wochen Monitoring)

```sql
-- Wenn alles stabil läuft, alte Tabellen entfernen
DROP TABLE public.transactions CASCADE;
DROP TABLE public.companies CASCADE;

-- Views für Backward-Compatibility (optional)
CREATE VIEW public.companies AS
SELECT * FROM backend.companies;

CREATE VIEW public.transactions AS
SELECT
    bt.id,
    bt.company_id,
    bt.corporate_counterparty as counterparty,
    bt.tx_amount as amount,
    bt.tx_currency as currency,
    bt.tx_date::timestamp as executed_at,
    bt.status,
    bt.created_at::timestamp,
    bt.last_modified_at::timestamp as updated_at
FROM backend.transactions bt;
```

---

## Zeitplan

| Phase | Dauer | Abhängigkeiten |
|-------|-------|----------------|
| 1. Vorbereitung | 1-2 Tage | - |
| 2. Schema-Erweiterung | 2-3 Tage | Phase 1 |
| 3. Datenmigration | 3-5 Tage | Phase 2 |
| 4. Model-Anpassung | 2-3 Tage | Phase 3 |
| 5. Code-Refactoring | 5-7 Tage | Phase 4 |
| 6. Testing | 3-5 Tage | Phase 5 |
| 7. Deployment | 1-2 Tage | Phase 6 |
| 8. Cleanup | Nach 2-4 Wochen | Phase 7 |

**Gesamtdauer**: 17-27 Arbeitstage (3-5 Wochen)

---

## Risiken & Mitigation

### Risiko 1: Datenverlust
**Mitigation**:
- Vollständige Backups vor jedem Schritt
- Test-Migration in Staging-Umgebung
- Datenvalidierung nach jeder Phase

### Risiko 2: Downtime
**Mitigation**:
- Migrations während Wartungsfenster
- Blue-Green Deployment
- Read-Replica für Zero-Downtime

### Risiko 3: Performance-Probleme
**Mitigation**:
- Indizes auf Foreign Keys
- Batch-Processing für große Datasets
- Query-Optimierung mit EXPLAIN ANALYZE

### Risiko 4: Code-Inkompatibilität
**Mitigation**:
- Compatibility Layer
- Schrittweises Refactoring
- Feature-Flags für graduelle Umstellung

---

## Nächste Schritte

1. ✅ **Klärung der Fragen oben**
2. ⬜ Detaillierte Datenanalyse durchführen
3. ⬜ Test-Umgebung aufsetzen
4. ⬜ Erste Migration in Staging testen
5. ⬜ Review & Approval vom Team
6. ⬜ Production-Migration planen

---

*Erstellt am: 2025-11-12*
*Status: ENTWURF - Wartet auf Klärung der kritischen Fragen*