Deployment Guide
LiteJoin is a single binary with no external dependencies. This makes deployment simple, but there are operational considerations for production use.
Resource Planning
CPU
LiteJoin is lightweight. A 2-vCPU machine handles most workloads. CPU usage scales with:
- Number of sources (each poll cycle parses JSON, diffs, and writes)
- Join complexity (each write triggers join re-evaluation)
- Window evaluation frequency
Memory
Default memory usage is ~50–100 MB. SQLite’s page cache is the main consumer. If tiered storage is enabled with DuckDB, allocate additional memory (default DuckDB limit: 256 MB).
Disk
Disk usage depends on:
- Shard count × retention duration × ingest rate
- Each shard is a separate SQLite database file
- WAL files can temporarily double the on-disk size during heavy writes
Rule of thumb: Allocate 2–3× your expected data size to account for WAL files and temporary compaction during archival.
Network
API source polling generates outbound HTTP requests. Calculate your request budget:
requests_per_minute = (sources × pages_per_poll) / interval_seconds × 60
Monitoring
Health Check
curl http://localhost:9100/health
# {"status": "ok"}
Log Levels
LiteJoin logs to stdout. Key patterns to monitor:
| Pattern | Level | Meaning |
|---|
api-source X: poll failed (N/max) | WARN | Transient source failure |
api-source X: ERROR N consecutive failures | ERROR | Source degraded |
dlq: enqueued result for sink | INFO | Delivery failure captured |
dlq: entry exceeded max retries | ERROR | Permanent delivery failure |
joiner: FAILED sink send | WARN/ERROR | Sink unavailable |
Pipeline Monitoring
LiteJoin logs pipeline queue depths periodically to help diagnose bottlenecks:
pipeline-monitor: msgCh=12/1000 (1.2%) | writer-batch=45 | joiner-pending=3
Graceful Shutdown
LiteJoin handles SIGTERM and SIGINT gracefully:
- Sources stop polling / consuming
- In-flight messages are flushed to storage
- Pending DLQ retries complete
- Sink connections are closed
- SQLite databases are checkpointed and closed
Allow 10–30 seconds for graceful shutdown in your orchestrator’s termination grace period.
Multiple Pipelines
Run multiple LiteJoin instances with separate configs and data directories:
# Pipeline A
litejoin run -c pipeline-a.yaml
# Pipeline B (different data dir to avoid conflicts)
litejoin run -c pipeline-b.yaml
Never point two LiteJoin instances at the same data_dir. SQLite requires exclusive write access.
