LoginSign Up
← Back to all sparks
C

CBTF

INFRA · APIS
Velocity0.0

Caught by the Fuzz! - A Minimalistic Fuzz-Test Runner

github.com/mcol/caught-by-the-fuzz ↗

A fuzzer for R packages that grew from one argument at a time to parallel runs across whole namespaces.

testingfuzzingr-packagedeveloper-toolsparallel-computation
◆Current state
CBTF throws deliberately problematic inputs at exported functions and reports which ones error, warn, or survive. It now covers 85 input classes, fuzzes multiple named arguments per call, runs in parallel with a per-function timeout, and reports results as a classed object with print, summary, length, and whitelist methods for pruning false positives. Output control has been refined repeatedly — the latest release adds grouping by input or by function and a summary-only mode.
◆Where it's heading
The package has moved from a script-shaped tool to a test-suite component. The single-argument, sequential fuzz() of the early releases could not finish a large namespace or survive a function that simply hangs; the mirai rewrite fixed both, and multi-argument support widened what a run can actually reach. Nearly everything since has gone into making results triageable rather than merely produced: whitelisting, result classes, grouping, and terminal output that fits real function signatures. Note that the 0.1.0 through 0.3.0 entries carry backfilled timestamps recorded in reverse order within under a minute, so their published dates do not reflect release order.
◆Prediction
The steady expansion of the input catalogue from 70 to 85 with a new time class suggests more input classes are the cheapest next win; the reporting surface is now detailed enough that machine-readable output for CI would be the natural follow-on.

◆Recent moves

  1. 1mo ago

    Time inputs added; results groupable by input or function

    The input catalogue grows from 70 to 85 with a new time class, and print() gains grouping by input or by function plus a summary-only mode. Continued work on the triage side of the tool rather than the fuzzing side, alongside fixes for multi-message conditions and dropped warnings.

    View source ↗
  2. 6mo ago

    Parallel fuzzing with timeouts, and multi-argument support

    ⚡ SPARK

    The release that made the tool usable on a whole package rather than a handful of functions. Rewriting fuzz() on mirai brings parallel execution and a timeout that can interrupt a function which neither fails nor returns, while the args argument lifts the long-standing restriction to fuzzing only the first argument.

    View source ↗
  3. 0y ago

    Whitelisting for false positives, plus clearer result semantics

    whitelist() lets known-benign failures be dropped from a results object, and length() switches to counting tests rather than inputs. Both are triage features, and they arrive alongside errors now taking precedence over warnings when a function raises both — a reporting correctness fix worth more than it sounds.

    View source ↗
  4. 1y ago

    Function discovery skips unfuzzable functions; failures no longer crash

    get_exported_functions() stops returning functions with no arguments and survives a failing namespace lookup, so a run no longer dies on discovery. Small, but it is the layer everything else depends on before a single input is thrown.

    View source ↗
  5. 1y ago

    First release; notes carry only a website link

    The package's first version, though the entry says nothing about what it does beyond pointing at the new website — the capability description has to be read from the releases that follow. Its timestamp is backfilled and lands after 0.2.0 and 0.3.0 in this feed, so its position here does not reflect release order.

    View source ↗
  6. 1y ago

    Raw results returned as an object with summary and print methods

    fuzz() starts returning a structured result with S3 methods instead of printing and discarding, which is the foundation every later reporting feature builds on. Functions calling readline() are skipped so interactive runs stop stalling — an early version of the hang problem that the timeout in 0.6.0 solved properly.

    View source ↗