LoginSign Up
← Back to all sparks
H

HOMER

MEETINGS
Velocity5.0

Open source VoIP and RTC packet capture and monitoring

sipcapture.org ↗

Homer's native compaction engine goes from do-not-touch to safe-to-try in one day.

voip monitoringduckdbducklakecompactiondistributed queriessecurity fixes
◆Current state
The release stream is a rapid sequence of production failures found and closed, nearly all of them in the DuckDB/DuckLake storage layer. The last two entries tell one story: 11.0.320 bounded a DuckDB compaction merge that was silently SIGKILLing the process with the catalog lock held, and explicitly told operators not to switch to the native engine; 11.0.321 then made native compaction safe for a live catalog. Around that sit alert-query replay, per-user dashboard IDs, raised DuckDB memory caps for SIPREC ingest, an Arrow timestamp-unit fix, opt-in time-range routing, and two inline GHSA fixes hardening node /query auth.
◆Where it's heading
Homer is stabilizing a storage rewrite by moving work out of DuckDB rather than tuning it further. Native compaction merges in Go, outside DuckDB's memory_limit, and lets DuckLake alone allocate snapshot and file IDs — the exact behavior that corrupted catalogs before. The same instinct shows in the coordinator's smart routing, which avoids querying storage nodes that cannot hold the window, and in raising container memory ceilings rather than shrinking workloads. Defaults stay conservative while each escape hatch is proven.
◆Prediction
If native compaction holds up in the field, expect it to become the default engine and the DuckDB compaction path to be deprecated; the near-term releases will likely keep closing memory and catalog-locking edges around it.

◆Recent moves

  1. 1d ago

    Native compaction made safe for live catalogs; retention in hours

    Native compaction becomes safe to run against a live catalog: every catalog write goes through ducklake_add_data_files so DuckLake alone allocates snapshot and file IDs, merging happens in Go outside DuckDB's memory_limit, and unsafe or unprofitable partitions are skipped rather than merged. Retention also gains hours as a unit. The default engine stays duckdb, so this is an opt-in path proven rather than a switch thrown.

    View source ↗
  2. 2d ago

    Compaction merge bounded after silent OOM kills on live catalogs

    Bounds the DuckDB compaction merge that could SIGKILL the process with no out-of-memory log, holding the catalog lock while the UI returned 5xx. Merge now defaults to 32 operations per cycle, 64MB max file size, and a single thread on a dedicated connection. The release explicitly warns operators off the native engine — advice the very next release reverses.

    View source ↗
  3. 4d ago

    Alerts store the firing query and replay it in search

    Alerts posted from Grafana or any source can carry the firing query, which Settings and the alert widget then replay as dashboard filters. Closes the gap between an alert firing and finding the traffic that caused it.

    View source ↗
  4. 5d ago

    DuckDB memory caps raised so high ingest stops killing the engine

    Container images raise the DuckDB memory limit to 4GB with two threads and a spill directory, so SIPREC and high-ingest flushes stop hitting the 2GB cap and fatally invalidating the engine. Overridable at runtime without a rebuild.

    View source ↗
  5. 5d ago

    Per-user dashboard IDs unblock multi-user installs

    Default dashboard IDs become unique per user, so the first account to seed no longer locks every later user out with an ownership error. Fresh installs also follow the system color scheme.

    View source ↗
  6. 5d ago

    FlightSQL timestamp units fixed after captures displayed as 1882

    FlightSQL writes DuckDB timestamps as Arrow microseconds instead of nanoseconds, fixing Grafana's conversion overflowing int64 and displaying capture times as 1882. Stored DuckLake data is untouched.

    View source ↗