Ingestion Tuning

Architecture

HTTP Request --> Token Auth --> Parse --> Enqueue --> [Worker Pool] --> ClickHouse
                    |                        |
               Invalid? 401            Queue full? 429

Incoming logs are queued in memory and written to ClickHouse in batches by a pool of workers. If the queue is full, Strix returns 429 Too Many Requests until capacity frees up. With default settings, the queue can buffer approximately 2.5 million logs.

Configuration

Variable	Default	When to Tune
STRIX_INGEST_WORKERS	4	ClickHouse CPU is underutilized during bulk imports
STRIX_INGEST_QUEUE_SIZE	500	Frequent 429 responses during ingestion spikes
STRIX_INGEST_RATE_LIMIT	10000	Legitimate sources are being rate-limited
STRIX_INGEST_RATE_BURST	20000	Bursty sources need higher peak throughput
STRIX_MAX_BODY_SIZE	209715200	Need to accept requests larger than 200MB