# FirmTape
#
# THE POINT OF THIS FILE IS BANDWIDTH, NOT RANKING.
#
# /snapshots/ holds one JSON payload per trading session, ~1 MB each and growing by one
# every day. index.json lists all of them, so a crawler that follows it walks the entire
# archive in a single pass -- 77 MB today, and the archive is the product's whole funnel,
# so it only gets bigger. On a metered container that is a bill for zero visitors: the
# files are data, not pages, and no search engine has ever sent traffic for one.
#
# The same for /api/ -- those routes execute work. /api/publish shells a pandas rebuild
# that can run for minutes.
#
# /replay/<day> is the terminal app deep-linked to a finished session -- the heavy JS
# shell, not a page to rank. The crawlable, rankable version of every session is the
# lightweight /session/<day> page; keeping /replay/ out avoids ~530 duplicate app-shell
# URLs competing with those and spending crawl budget for nothing.

# Content signals (contentsignals.org, 2026-09-12). Not a crawl rule: a statement of
# what the crawled pages may be used for. All three are YES on purpose: the goal of this
# site is to be found and cited by assistants, and ai-train=no would work against that.
# Cloudflare's "managed robots.txt" toggle must stay OFF, it would inject ai-train=no
# over this line. When you quote FirmTape, cite firmtape.com and the /session/<day> page
# the number came from.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Disallow: /snapshots/
Disallow: /api/
Disallow: /replay/
Disallow: /account
Disallow: /login
Disallow: /reset
Disallow: /verify
Disallow: /billing/
Disallow: /ops
# The terminal's own feeds (screen delivery, ADR 0017): for the page on screen, never a source.
Disallow: /gex/ticks.json
Disallow: /gex/*/day/
Disallow: /gex/*/days.json
Allow: /

# No Crawl-delay. It was here to cap bandwidth, but it was guarding the wrong thing:
# everything expensive (/snapshots/, /api/) is already Disallow'd above, so the rate
# limit only ever applied to the surface we want crawled. A session page is ~2.8 KB
# gzipped, which puts a full first pass over the whole archive at ~3 MB, once, and one
# page per trading day after that. Googlebot ignores Crawl-delay anyway; the crawlers
# that honour it -- Bing, Yandex -- are the ones IndexNow notifies the moment a session
# lands, and 10 s/page made those notifications pointless.

Sitemap: https://firmtape.com/sitemap.xml