Layer 4 · renderer

Reading a page is a ladder, and the top rung handles most of the web.

Two lines of configuration — [renderer], mode = "auto" — select escalation rather than a fixed strategy. Cheap fetch first; a browser only when the cheap fetch fails. The expensive rung is opt-in, and the repository's own measurements say it rarely helps.

The interesting thing about this layer is not the technology on each rung, which is upstream fastCRW's. It is the ordering, and the fact that the ordering was tested rather than assumed — including the test that produced an answer the author did not want.

MechanismThe escalation ladder

The renderer escalation ladder under mode = auto Three tiers step down and to the right as cost increases: plain HTTP, a JavaScript-rendering browser, and the opt-in Camoufox stealth tier. Each tier exits right to clean markdown on success, and drops to the next tier on failure. Below the last tier, two contrasting terminal states: an honest reported failure, or a challenge page scored as content. mode = "auto" · CHEAPEST RUNG FIRST · ESCALATE ONLY ON FAILURE cost per page · ms → seconds one URL 1 · plain HTTP A normal request, with correct TLS fingerprinting Handles most of the web, in milliseconds always compiled in · always first 2 · JavaScript rendering A real DOM, for pages that build themselves client-side Hundreds of milliseconds, plus a browser process built with the `cdp` feature · LightPanda is the fast one 3 · Camoufox — opt-in Firefox fork with C++-level anti-detection, driven over REST Sidecar on 127.0.0.1:9377 · camoufox_timeout_ms = 60000 include_in_auto = true appends it — it never runs first absent from the published image; needs --features camoufox fail ↓ fail ↓ success clean markdown back up to crw, then to the summariser or straight to your agent, depending on the tool TERMINAL STATES ✓ honest failure Camoufox reports: bot challenge detected crw knows it has nothing. So do you. ✕ dishonest success Plain HTTP returns "Please enable JS" or "Security Check" — a challenge page, scored as a fetch, fed to your model.
Figure 7 — the ladder The two boxes at the bottom are the reason the stealth tier exists in this repository at all. Neither of them produces content; only one of them tells you so.

Rung one is the surprising one. A plain HTTP fetch is not a naive fetch: crw's ordinary path already does proper TLS fingerprinting, which is the check most anti-bot layers apply before they ever look at a DOM. That is why escalation is rarely needed, and why adding a heavier browser further down buys less than intuition suggests.

MeasurementThe Camoufox result

Camoufox is a Firefox fork with anti-detection patches at the C++ level, run here through the camofox-browser REST sidecar. crw supports it as an opt-in renderer tier. The repository tested it against the sites people actually complain about, and published what happened:

Reproduced verbatim from README.md, "Camoufox stealth tier (optional, modest benefit)". Character counts are the size of the extracted markdown — the same measure health.sh uses to tell content from a challenge page.
siteplain HTTPcamoufoxverdict
g2.comjunk ("Please enable JS")bot challenge detectedtie — neither returned content
indeed.comjunk ("Security Check")bot challenge detectedtie — neither returned content
zillow9,149 charsfailedloss
yahoo finance48,971 chars7,574 charsloss — 85% less content
redditblockedblockedtie

It lost or tied everywhere.

README.md — Camoufox stealth tier

Two reasons are given, and both are structural rather than fixable by configuration. First, crw's plain HTTP path already does the TLS fingerprinting that a stealth browser would otherwise contribute. Second — and this is the general lesson — these sites block on IP and rate, not on browser fingerprint. A better disguise does not help when the thing being recognised is the address, not the face.

So why is it in the repository at all?

Because failing honestly has value that beating the bot check does not. Plain HTTP hands your model a CAPTCHA page scored as a successful fetch; Camoufox reports bot challenge detected. Feeding a model a challenge page and calling it a source is worse than a clean failure — the model will summarise it, cite it, and reason from it. And because include_in_auto = true places the tier at the end of the ladder, it only ever runs on pages the cheap rungs already failed. It cannot make working pages worse.

OperationTurning the tier on

Three things have to be true, and the first one is not a configuration change: the published ghcr.io/us/crw image does not contain the feature, so you have to compile it.

stealth tierscripts/build-crw-camoufox.sh
$ git clone https://github.com/us/crw.git ./vendor/crw
$ ./scripts/build-crw-camoufox.sh

# the build it runs — note the capped job count
docker build \
  --build-arg CARGO_PKGS="-p crw-server --features cdp,camoufox -p crw-mcp -p crw-cli" \
  --build-arg CARGO_BUILD_JOBS="6" \
  -t crw-local:camoufox ./vendor/crw

Built crw-local:camoufox. Now:
  1. set CRW_IMAGE=crw-local:camoufox in .env
  2. uncomment [renderer.camoufox] in config/crw.toml
  3. build + run the sidecar:
       git clone https://github.com/jo-inc/camofox-browser.git && cd camofox-browser && make build
  4. docker compose --profile stealth up -d
Why CARGO_BUILD_JOBS is capped at 6

Straight from the script's comment: a 524-crate build at full parallelism will OOM a default 8 GB Docker Desktop VM. This is the kind of number that only exists in a repository because somebody's build died at 3am.

The camofox service, docker-compose.yml lines 69–76, and the matching renderer block in config/crw.toml lines 16–19.
settingvaluewhy
profiles["stealth"]The service does not start unless you ask for it by name
shm_size2gbA real Firefox needs real shared memory; this is why the stealth profile roughly doubles the stack's RAM footprint
ports127.0.0.1:9377:9377camofox-browser's own make up publishes on 0.0.0.0. This compose file does not.
base_urlhttp://host.docker.internal:9377crw reaches it through the host gateway, not the compose network
camoufox_timeout_ms60000A full minute — acceptable only because this rung runs last, on pages that already failed

VerificationAsking crw what it can render

Whether the feature is actually compiled in is not something you should have to infer from a failed scrape. crw reports it, and health.sh goes looking — with a deliberately loose search, because the exact shape of the capabilities document depends on your build:

renderer checkscripts/health.sh lines 25–36
$ curl -s --max-time 10 "http://127.0.0.1:3002/v1/capabilities" \
  | python3 -c '
import sys,json
def walk(o):
    if isinstance(o,dict):
        for k,v in o.items():
            if "render" in k.lower(): print("  ",k+":",json.dumps(v)); return
            walk(v)
    elif isinstance(o,list):
        for i in o: walk(i)
walk(json.load(sys.stdin))'

   (whichever renderer key your build reports, dumped as JSON)

That walk() is a recursive search for the first key whose name contains render, anywhere in the document. It is a small piece of defensive scripting: the check keeps working when upstream reorganises its capabilities schema, and prints (unavailable) rather than failing the whole health run if the endpoint is missing — note that unlike the other three checks, this one never sets fail=1. Renderer introspection is diagnostic, not a pass/fail condition.