LoginSign Up
← Back to all sparks
S

Sonic

DEVOPS
Velocity6.3

Lightweight and schema-less search backend

github.com/valeriansaliou/sonic ↗

Sonic added real relevance ranking — and a benchmark to measure it.

search-indexrustbm25-rankingrelevance-qualityunicodemonorepo-tagging
◆Current state
Sonic is a lightweight Rust search index that has traded on speed and small footprint rather than ranking quality. The 1.8.0 cycle changes that: BM25 lite (idf-only) scoring and a minimum-term-idf floor replace flat term matching, and a BEIR benchmark now sits in the repo. The same cycle added Unicode normalization and custom stopwords, closing gaps that made non-English corpora awkward.
◆Where it's heading
The project now ships as three lockstep tags — sonic-server, sonic-core, sonic-client — with the ranking work landing in core first and the server bundling it hours later. Adding a retrieval benchmark alongside the scoring change is the tell: quality is becoming a tracked metric, not an assumption. The immediate cost showed up the same day, when renamed RocksDB config keys broke upgrades and forced a 1.8.1 correction.
◆Prediction
The idf-only qualifier on BM25 points to the term-frequency and length-normalization components landing next, with BEIR runs used to justify the tuning. Expect the experimental TRIGGER flush to either graduate or disappear once the benchmark harness gets exercised.

◆Recent moves

  1. 2d ago

    Restores config compatibility broken hours earlier by 1.8.0

    A same-day correction to 1.8.0, whose renamed RocksDB configuration keys turned out to be a breaking change. Compatibility tests were added alongside the fix to catch config regressions before they ship, which suggests the ranking work in 1.8.0 outpaced the project's release checks.

    View source ↗
  2. 2d ago

    Core re-tag carrying the config fix through to the server

    The core-library half of the 1.8.1 fix, released only so sonic-server could depend on it. The notes state plainly that sonic-core users are unaffected, making this a plumbing tag in the three-package release train rather than a change anyone consumes directly.

    View source ↗
  3. 2d ago

    BM25 relevance scoring lands; config rename breaks upgrades

    ⚡ SPARK

    The release where Sonic stops being a term-matching index and starts being a ranked one. BM25 lite and a minimum-idf floor push low-quality results off the tail, while Unicode normalization and custom stopwords make the tokenizer usable outside English — the natural continuation of the tokenizer control shipped in 1.7.x.

    View source ↗
  4. 2d ago

    Client gains a raw API escape hatch

    The Rust client exposes its raw API, letting callers issue protocol commands the typed wrapper does not cover. Useful timing given the server is adding commands like TRIGGER flush faster than the client can wrap them, and a sign the client is being treated as a real surface rather than a convenience shim.

    View source ↗
  5. 2d ago

    Core ships the BM25 and Unicode work ahead of the server

    The core library landing the same BM25 lite, Unicode normalization, and custom stopword changes that reach users four hours later in server 1.8.0. Cataloguing it separately reflects how this repo releases — core first, server bundling it — rather than a distinct change; the server tag is where this work becomes visible.

    View source ↗
  6. 1mo ago

    Opt-in tokenizer pattern matching plus a full dependency pin

    The July release that made tokenizer pattern matching opt-in and non-breaking while pinning every dependency and stripping default features. In hindsight it set up 1.8.0: tokenizer control had to be settled before Unicode normalization and stopwords could be layered on top.

    View source ↗