Python · asyncio · SQLite/FTS5 · TCP · DockerCurrent build
Problem
Grepping a fast-moving log file tells you almost nothing. LogScope follows files live through rotation, groups similar lines into ranked patterns, flags error spikes with statistics it can show you, and searches everything. All of it from the terminal, all on one asyncio event loop.
Design decisions
- Pattern grouping runs on a hand-built Drain implementation. One bug printing 10,000 near-identical lines becomes one row with a count.
- Spike detection uses a rolling z-score, and the tool renders the statistic as a small graph. An alert should be an explanation, not a mystery.
- Every source feeds one bounded queue, drained through parse, index, cluster, anomaly, and sink stages. Bursty sources slow themselves down instead of eating memory.
- Multi-machine collection runs over a custom length-prefixed TCP protocol with at-least-once delivery. Per-line IDs make ingestion idempotent; agents buffer through disconnects and replay with backoff. Restart the server: no line dropped, none duplicated.
- An optional AI summary explains the root cause of a problem cluster. It is cached, time-limited, and the tool works fine without it.
- LogScope watches itself the way it watches your app: events per second, queue depth, lag, query speed.
Outcome
Built to learn how real log tools work on the inside, not to replace Datadog or Loki. Tested like it matters anyway: 100+ tests under GitHub Actions CI, including crash-recovery paths.