LoginSign Up
← Back to all sparks
F

Firecrawl

AI-ASSISTANTS
Velocity7.5

Web scraping and crawling API that turns websites into clean, LLM-ready markdown and structured data.

firecrawl.dev ↗

Firecrawl stopped selling pages and started selling answers — now it is giving the corpus away.

agent-infrastructuretoken-efficiencyvertical-indexesbenchmarksweb-monitoringfree-tier
◆Current state
Firecrawl has spent four months converting a scraping API into an answer-retrieval layer for agents. Question, Highlights and the excerpt-scoring rebuild of /search all trade full-page delivery for the specific lines that answer a query, each pitched on token cost rather than coverage. Alongside that it has started owning corpora outright — the Research Index now spans 3M+ arXiv papers and 41M+ life-sciences papers — and /monitor turns crawling into a subscribable event stream.
◆Where it's heading
The centre of gravity is moving from generic crawl infrastructure to curated indexes with published benchmark numbers attached, and now to giving those indexes away. Every recent release argues the same point in a different register: the crawler should return the smallest correct thing, and Firecrawl should already have it indexed. Free access to Research Index converts a metered data product into a distribution channel for the paid scraping and monitoring endpoints around it.
◆Prediction
Expect a third vertical index after AI/ML and life sciences — the pattern of a benchmark claim, daily refresh and API-plus-MCP-plus-CLI availability is now a repeatable template. Whether the free tier stays free once query volume lands is the open question the entries do not answer.

◆Recent moves

  1. 6d ago

    Life Sciences in Firecrawl Research Index

    ⚡ SPARK

    The Research Index gains a second vertical — 41M+ life-sciences papers covering drug discovery, clinical trials and biology — and the entire index drops to free. This is the arXiv corpus playbook run again in a different domain, with the pricing removed.

  2. 28d ago

    Introducing our most accurate /search yet

    ⚡ SPARK

    The excerpt-scoring rebuild of /search is the clearest statement of the token-efficiency thesis running through this whole stretch: same API shape, dramatically less returned, a benchmark number attached to prove it did not cost accuracy.

  3. 1mo ago

    Web-scale /monitor

    Web-scale /monitor generalises the earlier URL-watching endpoint from named pages to standing search queries, so an agent subscribes to a topic rather than a location. It broadens the monitoring surface without changing what monitoring is.

  4. 1mo ago

    v2.11.0: Research Index, keyless access, PII redaction

    The v2.11.0 roll-up bundles the Research Index launch with keyless endpoint access, PII redaction and a cached deterministic JSON extractor. Individually incremental, together they lower the friction of putting Firecrawl inside an agent loop that handles regulated data.

  5. 2mo ago

    Firecrawl Research Index

    ⚡ SPARK

    The original Research Index launch is the pivot the last two months turn on — the moment Firecrawl stopped only fetching pages on demand and started maintaining a corpus of its own with a benchmark attached.

  6. 2mo ago

    Introducing /monitor

    ⚡ SPARK

    The first /monitor release turned scraping from a pull into a push, and the web-scale version three weeks ago is the direct continuation. It also introduced the plain-English goal as the configuration surface, a pattern the later releases reuse.