chore: major project cleanup and restructuring
## Dokumentation - Entfernt AGENTS.md (redundant zu CLAUDE.md) - Verschoben: 3 Dokumentationen von root → docs/ - Gelöscht: 8 historische Migrations-Pläne aus docs/archive/ - Entfernt: misc/ Ordner komplett (43 Dateien) ## Struktur - Erstellt: database/queries/ für SQL Test-Queries (4 Dateien) - Erstellt: database/schemas/ für Schema-Dokumentation (17 Dateien) - output_keys_mapping/ (8 CSV-Mappings) - migration-diagrams/ (4 Diagramme) ## Code-Qualität - Formatiert: 7 Style-Issues in 74 Dateien mit Laravel Pint - Gefixed: Migration für PostgreSQL/SQLite Kompatibilität - Gefixed: 2 fehlgeschlagene Tests (CSRF + Text-Assertion) ## Tests - Alle 73 Tests bestehen jetzt (100% Success Rate) - AuthenticationTest: CSRF-Token Fix für Logout-Test - CompanySearchTest: Text-Assertion aktualisiert ## Ergebnis - Root ist sauber (nur CLAUDE.md) - Dokumentation strukturiert in docs/ - Database-Dateien organisiert in database/ - Code entspricht Style-Guide - Alle Tests bestehen
This commit is contained in:
@@ -0,0 +1,445 @@
|
||||
# Backend Data Pool Sync - Scheduling Setup
|
||||
|
||||
## Übersicht
|
||||
|
||||
Der Backend Data Pool Sync wurde mit automatischem Scheduling konfiguriert:
|
||||
|
||||
### 🔄 Incremental Sync
|
||||
- **Frequenz:** Alle 6 Stunden
|
||||
- **Zweck:** Synchronisiert nur neue oder geänderte Transaktionen
|
||||
- **Max. Laufzeit:** 30 Minuten
|
||||
- **Batch Size:** 1000 Records
|
||||
|
||||
### 🔃 Full Sync
|
||||
- **Frequenz:** Jeden Sonntag um 3:00 Uhr morgens
|
||||
- **Zweck:** Kompletter Rebuild des Data Pools (Truncate + Rebuild)
|
||||
- **Max. Laufzeit:** 60 Minuten
|
||||
- **Batch Size:** 1000 Records
|
||||
|
||||
---
|
||||
|
||||
## Setup & Aktivierung
|
||||
|
||||
### Option 1: Laravel Scheduler (Empfohlen für Produktion)
|
||||
|
||||
Der Laravel Scheduler benötigt einen Cron Job, der jede Minute läuft.
|
||||
|
||||
#### 1. Cron Job einrichten
|
||||
|
||||
Öffne die Crontab:
|
||||
```bash
|
||||
crontab -e
|
||||
```
|
||||
|
||||
Füge folgende Zeile hinzu:
|
||||
```cron
|
||||
* * * * * cd /Users/sebastianfrohlich/Herd/frontend && ~/Library/Application\ Support/Herd/bin/php artisan schedule:run >> /dev/null 2>&1
|
||||
```
|
||||
|
||||
**Oder** für besseres Logging:
|
||||
```cron
|
||||
* * * * * cd /Users/sebastianfrohlich/Herd/frontend && ~/Library/Application\ Support/Herd/bin/php artisan schedule:run >> /Users/sebastianfrohlich/Herd/frontend/storage/logs/scheduler.log 2>&1
|
||||
```
|
||||
|
||||
#### 2. Cron Job verifizieren
|
||||
|
||||
```bash
|
||||
# Prüfe ob Cron Job aktiv ist
|
||||
crontab -l
|
||||
|
||||
# Teste den Schedule manuell
|
||||
cd /Users/sebastianfrohlich/Herd/frontend
|
||||
~/Library/Application\ Support/Herd/bin/php artisan schedule:run
|
||||
```
|
||||
|
||||
#### 3. Schedule List prüfen
|
||||
|
||||
```bash
|
||||
php artisan schedule:list
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
┌─────────────────────────────────────────────┬─────────────┬─────────────────────────────────┬───────────────┐
|
||||
│ Command │ Interval │ Description │ Next Due │
|
||||
├─────────────────────────────────────────────┼─────────────┼─────────────────────────────────┼───────────────┤
|
||||
│ App\Jobs\SyncBackendDataPool │ 0 */6 * * * │ Sync new/updated backend trans… │ in 5 hours │
|
||||
│ App\Jobs\SyncBackendDataPool │ 0 3 * * 0 │ Full sync and rebuild of backe… │ in 6 days │
|
||||
└─────────────────────────────────────────────┴─────────────┴─────────────────────────────────┴───────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Option 2: Schedule Worker (Empfohlen für Development)
|
||||
|
||||
Für lokales Development kannst du den Schedule Worker nutzen:
|
||||
|
||||
```bash
|
||||
php artisan schedule:work
|
||||
```
|
||||
|
||||
Dieser Befehl läuft dauerhaft und führt die Schedules automatisch aus.
|
||||
|
||||
**Vorteile:**
|
||||
- ✅ Kein Cron Job notwendig
|
||||
- ✅ Echtzeit-Ausgabe in der Console
|
||||
- ✅ Einfaches Debugging
|
||||
|
||||
**Nachteile:**
|
||||
- ❌ Muss manuell gestartet werden
|
||||
- ❌ Stoppt wenn Terminal geschlossen wird
|
||||
|
||||
**Lösung:** Nutze einen Process Manager wie Supervisor oder Screen:
|
||||
|
||||
```bash
|
||||
# Mit screen
|
||||
screen -S scheduler
|
||||
php artisan schedule:work
|
||||
# Ctrl+A, dann D zum Detachen
|
||||
|
||||
# Später wieder attachen
|
||||
screen -r scheduler
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Logs
|
||||
|
||||
### 1. Laravel Logs prüfen
|
||||
|
||||
```bash
|
||||
# Live-Logs anzeigen
|
||||
tail -f storage/logs/laravel.log
|
||||
|
||||
# Nur Sync-Logs filtern
|
||||
tail -f storage/logs/laravel.log | grep "SyncBackendDataPool"
|
||||
|
||||
# Scheduler-Logs (falls Cron Log aktiviert)
|
||||
tail -f storage/logs/scheduler.log
|
||||
```
|
||||
|
||||
### 2. Schedule-Status prüfen
|
||||
|
||||
```bash
|
||||
# Nächste geplante Ausführungen
|
||||
php artisan schedule:list
|
||||
|
||||
# Alle Schedules testen (ohne Ausführung)
|
||||
php artisan schedule:test
|
||||
```
|
||||
|
||||
### 3. Manuelle Sync-Ausführung
|
||||
|
||||
```bash
|
||||
# Incremental Sync manuell ausführen
|
||||
php artisan backend:sync-data-pool --incremental
|
||||
|
||||
# Full Sync manuell ausführen
|
||||
php artisan backend:sync-data-pool --full
|
||||
|
||||
# Mit Queue (asynchron)
|
||||
php artisan backend:sync-data-pool --incremental --queue
|
||||
```
|
||||
|
||||
### 4. Queue Worker (falls Jobs in Queue laufen)
|
||||
|
||||
Wenn du `--queue` nutzt, muss ein Queue Worker laufen:
|
||||
|
||||
```bash
|
||||
# Queue Worker starten
|
||||
php artisan queue:work --queue=default --tries=3
|
||||
|
||||
# Oder mit Supervisor für Produktion
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Zeitplan Übersicht
|
||||
|
||||
### Incremental Sync (alle 6 Stunden)
|
||||
|
||||
| Zeit | Aktion |
|
||||
|-----------|------------------|
|
||||
| 00:00 Uhr | Incremental Sync |
|
||||
| 06:00 Uhr | Incremental Sync |
|
||||
| 12:00 Uhr | Incremental Sync |
|
||||
| 18:00 Uhr | Incremental Sync |
|
||||
|
||||
### Full Sync (Sonntags)
|
||||
|
||||
| Tag | Zeit | Aktion |
|
||||
|---------|-----------|-----------|
|
||||
| Sonntag | 03:00 Uhr | Full Sync |
|
||||
|
||||
**Wichtig:** Am Sonntagmorgen um 3 Uhr läuft nur der Full Sync (nicht zusätzlich Incremental).
|
||||
|
||||
---
|
||||
|
||||
## Anpassungen
|
||||
|
||||
### Schedule-Zeiten ändern
|
||||
|
||||
Editiere [routes/console.php](routes/console.php):
|
||||
|
||||
#### Beispiele für andere Frequenzen:
|
||||
|
||||
```php
|
||||
// Incremental Sync: Stündlich
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->hourly();
|
||||
|
||||
// Incremental Sync: Alle 2 Stunden
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->everyTwoHours();
|
||||
|
||||
// Incremental Sync: Täglich um 2 Uhr
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->dailyAt('02:00');
|
||||
|
||||
// Full Sync: Täglich um 3 Uhr
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->dailyAt('03:00');
|
||||
|
||||
// Full Sync: Monatlich am 1. um 4 Uhr
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->monthlyOn(1, '04:00');
|
||||
```
|
||||
|
||||
### Batch Size ändern
|
||||
|
||||
```php
|
||||
// Kleinere Batches für weniger Speicherverbrauch
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 500))
|
||||
->everySixHours();
|
||||
|
||||
// Größere Batches für schnellere Verarbeitung
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 5000))
|
||||
->everySixHours();
|
||||
```
|
||||
|
||||
### Overlap Protection anpassen
|
||||
|
||||
```php
|
||||
// Längere Lock-Zeit (z.B. für große Datenmengen)
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->weeklyOn(0, '03:00')
|
||||
->withoutOverlapping(maxLockTime: 7200); // 2 Stunden
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Problem: "No scheduled commands are ready to run"
|
||||
|
||||
**Ursache:** Es ist noch nicht Zeit für die nächste Ausführung.
|
||||
|
||||
**Lösung:** Prüfe `php artisan schedule:list` für nächste Ausführungszeit.
|
||||
|
||||
---
|
||||
|
||||
### Problem: Schedule läuft nicht
|
||||
|
||||
**Ursache:** Cron Job nicht aktiv oder falsch konfiguriert.
|
||||
|
||||
**Lösung:**
|
||||
```bash
|
||||
# Prüfe Cron Job
|
||||
crontab -l
|
||||
|
||||
# Teste Schedule manuell
|
||||
php artisan schedule:run
|
||||
|
||||
# Prüfe Logs
|
||||
tail -f storage/logs/laravel.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem: "Class SyncBackendDataPool not found"
|
||||
|
||||
**Ursache:** Autoload-Cache ist veraltet.
|
||||
|
||||
**Lösung:**
|
||||
```bash
|
||||
composer dump-autoload
|
||||
php artisan optimize:clear
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem: Jobs laufen mehrfach parallel
|
||||
|
||||
**Ursache:** `withoutOverlapping()` funktioniert nicht.
|
||||
|
||||
**Lösung:** Stelle sicher, dass ein Cache-Driver konfiguriert ist:
|
||||
```bash
|
||||
# In .env
|
||||
CACHE_STORE=database
|
||||
```
|
||||
|
||||
Dann Cache-Tabellen migrieren:
|
||||
```bash
|
||||
php artisan cache:table
|
||||
php artisan migrate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Problem: Sync dauert zu lange
|
||||
|
||||
**Lösungen:**
|
||||
|
||||
1. **Batch Size erhöhen:**
|
||||
```php
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false, batchSize: 5000))
|
||||
```
|
||||
|
||||
2. **Queue nutzen:**
|
||||
```php
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: false))
|
||||
->everySixHours()
|
||||
->runInBackground(); // Bereits aktiviert
|
||||
```
|
||||
|
||||
3. **Max Lock Time erhöhen:**
|
||||
```php
|
||||
->withoutOverlapping(maxLockTime: 3600) // 1 Stunde
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notifications (Optional)
|
||||
|
||||
Du kannst Notifications hinzufügen um bei Erfolg/Fehler benachrichtigt zu werden:
|
||||
|
||||
### Slack Notification
|
||||
|
||||
```php
|
||||
use Illuminate\Support\Facades\Notification;
|
||||
use App\Notifications\SyncCompletedNotification;
|
||||
use App\Notifications\SyncFailedNotification;
|
||||
|
||||
Schedule::job(new SyncBackendDataPool(fullSync: true))
|
||||
->weeklyOn(0, '03:00')
|
||||
->onSuccess(function () {
|
||||
// Notification::route('slack', env('SLACK_WEBHOOK'))
|
||||
// ->notify(new SyncCompletedNotification());
|
||||
})
|
||||
->onFailure(function () {
|
||||
// Notification::route('slack', env('SLACK_WEBHOOK'))
|
||||
// ->notify(new SyncFailedNotification());
|
||||
});
|
||||
```
|
||||
|
||||
### Email Notification
|
||||
|
||||
```php
|
||||
->onSuccess(function () {
|
||||
Mail::to('admin@example.com')
|
||||
->send(new SyncCompletedMail());
|
||||
})
|
||||
->onFailure(function () {
|
||||
Mail::to('admin@example.com')
|
||||
->send(new SyncFailedMail());
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Monitoring
|
||||
|
||||
### Database Queries überwachen
|
||||
|
||||
```sql
|
||||
-- Anzahl Syncs heute
|
||||
SELECT DATE(synced_at) as date, COUNT(*) as syncs
|
||||
FROM public.backend_data_pool
|
||||
WHERE synced_at >= CURRENT_DATE
|
||||
GROUP BY DATE(synced_at);
|
||||
|
||||
-- Letzte Sync-Zeiten
|
||||
SELECT MAX(synced_at) as last_sync,
|
||||
MIN(synced_at) as first_sync,
|
||||
COUNT(*) as total_records
|
||||
FROM public.backend_data_pool;
|
||||
|
||||
-- Sync-Performance (Records pro Minute)
|
||||
SELECT
|
||||
DATE_TRUNC('minute', synced_at) as minute,
|
||||
COUNT(*) as records_synced
|
||||
FROM public.backend_data_pool
|
||||
WHERE synced_at >= NOW() - INTERVAL '1 hour'
|
||||
GROUP BY DATE_TRUNC('minute', synced_at)
|
||||
ORDER BY minute DESC;
|
||||
```
|
||||
|
||||
### Laravel Telescope (Optional)
|
||||
|
||||
Installiere Telescope für besseres Monitoring:
|
||||
|
||||
```bash
|
||||
composer require laravel/telescope --dev
|
||||
php artisan telescope:install
|
||||
php artisan migrate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Produktion Deployment
|
||||
|
||||
### Supervisor Configuration (Empfohlen)
|
||||
|
||||
Erstelle `/etc/supervisor/conf.d/laravel-scheduler.conf`:
|
||||
|
||||
```ini
|
||||
[program:laravel-scheduler]
|
||||
process_name=%(program_name)s
|
||||
command=php /Users/sebastianfrohlich/Herd/frontend/artisan schedule:work
|
||||
autostart=true
|
||||
autorestart=true
|
||||
user=sebastianfrohlich
|
||||
redirect_stderr=true
|
||||
stdout_logfile=/Users/sebastianfrohlich/Herd/frontend/storage/logs/scheduler.log
|
||||
stopwaitsecs=3600
|
||||
```
|
||||
|
||||
Dann:
|
||||
```bash
|
||||
sudo supervisorctl reread
|
||||
sudo supervisorctl update
|
||||
sudo supervisorctl start laravel-scheduler
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Schedule testen (ohne Ausführung)
|
||||
|
||||
```bash
|
||||
php artisan schedule:test
|
||||
```
|
||||
|
||||
### Nächste Ausführung simulieren
|
||||
|
||||
```bash
|
||||
# Teste Incremental Sync
|
||||
php artisan backend:sync-data-pool --incremental --no-interaction
|
||||
|
||||
# Teste Full Sync
|
||||
php artisan backend:sync-data-pool --full --no-interaction
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Zusammenfassung
|
||||
|
||||
✅ **Scheduling konfiguriert** in [routes/console.php](routes/console.php)
|
||||
✅ **Incremental Sync:** Alle 6 Stunden
|
||||
✅ **Full Sync:** Sonntags 3:00 Uhr
|
||||
✅ **Overlap Protection:** Aktiviert
|
||||
✅ **Background Execution:** Aktiviert
|
||||
✅ **Success/Failure Callbacks:** Implementiert
|
||||
|
||||
**Nächster Schritt:** Cron Job einrichten oder `php artisan schedule:work` starten!
|
||||
Reference in New Issue
Block a user