Layer 2 · crw

One request in. A metasearch, eight renders and an LLM call out.

fastCRW is where the fan-out happens. It owns the MCP endpoint, asks SearXNG which pages exist, drives the renderer ladder over each of them, converts the results to markdown, and optionally hands the whole pile to a summariser that is not allowed to do anything else.

The thirty-nine lines of config/crw.toml in this repository are mostly comments. What they configure is small on purpose — a SearXNG address, query expansion, renderer mode, and two blocks left commented out. Everything else about this layer is the container definition, and that is written the way you write a definition for a process that parses bytes written by strangers.

TraceA search request, end to end

The diagram below follows a single search_answer call down through every process it touches and back. Time runs downward. The dashed red arrow is the alternative: what crw_search does instead, which is hand the same page text directly to the model that asked.

Figure 4 — one request, six processes Steps 4 and 5 are the part that breaks: SearXNG queries every enabled engine in parallel, and when the defaults are suspended together, step 6 returns nothing. Step 7 is the expensive part — up to eight renders, which is why websearch-mcp.py allows a 300-second timeout. Step 10 is the only place page text meets a model, and that model has no tools.

Configuration`config/crw.toml`, in full

The file is mounted read-only into the container at a path that looks odd until you read the environment: CRW_CONFIG=config.docker selects a configuration name, and the compose file mounts ./config/crw.toml at /app/config.docker.toml to satisfy it.

config/crw.tomllines 3–19, comments abridged

3   [search]
4   searxng_url = "http://searxng:8080"
5   # Multi-query expansion. Upstream measured this lifting SimpleQA 64%->76% and
6   # FRAMES 56%->80% by unioning an entity-rewrite's results with the original.
7   query_expand = true
8
9   [renderer]
10  mode = "auto"
…
16  # [renderer.camoufox]
17  # base_url = "http://host.docker.internal:9377"
18  # include_in_auto = true
19  # camoufox_timeout_ms = 60000

line 4 — http://searxng:8080: A compose service name, not a loopback address. crw talks to SearXNG over the private compose network; the 127.0.0.1:8888 publish exists only so you can query the JSON API by hand while debugging.
line 7 — query_expand = true: One user query becomes more than one engine query: the original, plus an entity rewrite, unioned. The figures in the comment are upstream fastCRW's measurements on SimpleQA and FRAMES, quoted as such — this repository did not run those benchmarks. The operational consequence is that a search costs more engine requests than you typed, which matters when engines are rate-limiting you.
line 10 — mode = "auto": Renderer escalation, rather than a fixed strategy. Cheap path first; browser only when the cheap path fails. The ladder is its own page.
lines 16–19 — the commented stealth tier: Commented out because it needs two things this compose file does not give you by default: an image built with --features camoufox, and the sidecar running under the stealth profile. include_in_auto is the important line — it appends the tier to the ladder rather than moving it to the front.

ConfigurationThe block that turns on synthesis

The second commented block is what makes search_answer work at all. Per the file's own comment it powers three things: formats: ["summary"], /v1/search answer synthesis, and /v1/extract.

config/crw.tomllines 31–38 — commented by default

31  # [extraction.llm]
32  # provider = "openai-compatible"
33  # api_key = "..."
34  # model = "..."
35  # base_url = "http://host.docker.internal:8002/v1"
36  # max_tokens = 4096
37  # max_concurrency = 3
38  # max_html_bytes = 100000

base_url — host.docker.internal: The summariser runs on the host, not in this compose file. It is reachable because the crw service declares extra_hosts: host.docker.internal:host-gateway.
max_concurrency = 3, max_html_bytes = 100000: Bounds on what the summariser is asked to do: at most three concurrent calls, at most 100 kB of HTML per page. Both are backpressure against a single search saturating a local GPU.
Pick a small model: The comment is blunt about it: "Do NOT point it at your main coding model: it will contend with your own sessions for the same GPU." Summarising is easy work sitting on a hot path.

Gotcha — reasoning models silently break synthesis

From the same comment block: if the summariser emits reasoning tokens, serve it with a reasoning parser and thinking disabled. Otherwise vLLM returns content=null once reasoning eats the token budget, and crw fails with openai response missing content — dropping answer synthesis without any obvious signal that it has done so. search_answer is written to surface that case rather than fall back to raw text; see lines 96–104.

ContainmentThe service definition

crw is the process that parses HTML, JavaScript and PDFs fetched from addresses chosen by whatever asked it a question. Its compose entry is written accordingly — the comment above the limits reads simply Defence in depth for untrusted page content.

From `docker-compose.yml` lines 13–39. The `depends_on` condition is `service_healthy`, not `service_started` — crw does not come up until SearXNG answers `/healthz`.
directive	value	what it prevents
read_only	true	Anything written to the image filesystem, including a dropped payload
tmpfs	/tmp	The one writable path, in RAM, gone on restart
cap_drop	[ALL]	Every Linux capability, including the ones a container rarely needs and never audits
security_opt	no-new-privileges:true	setuid escalation inside the container
mem_limit / memswap_limit	2g / 2g	A parser bomb taking the host down; equal values mean no swap
pids_limit	512	Fork bombs
ports	127.0.0.1:3002 → 3000	Anyone but you reaching an unauthenticated URL fetcher

Two of those are overridable by .env — CRW_BIND_ADDRESS and CRW_HOST_PORT — and the first is the one to leave alone. The reasoning is on the trust boundaries page.

VerificationAsking the engine directly

Every layer in this stack can be exercised without the layer above it, which is the point of health.sh checking four things separately. The scrape check is the most useful one to run by hand, because it proves fetch, render and markdown conversion in a single call:

engine checkthe pipelines from scripts/health.sh, run by hand

$ curl -s --max-time 60 "http://127.0.0.1:3002/v1/scrape" \
    -H 'Content-Type: application/json' \
    -d '{"url":"https://example.com"}' \
  | python3 -c 'import sys,json; print(len((json.load(sys.stdin).get("data") or {}).get("markdown") or ""))'
238   # health.sh passes this check at anything over 50

# the crw check asserts one thing: that /health reports a version
$ curl -s --max-time 10 "http://127.0.0.1:3002/health" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin).get("version","?"))'
(the version string of whichever crw image you pinned)

The health script treats anything over 50 characters of markdown as a pass. That threshold is doing real work: a blocked fetch, a CAPTCHA interstitial or a JavaScript shell all return something, and a byte count is a cheap way to tell content from a challenge page. It is the same distinction the Camoufox results table turns on.