robots.txt can express permission to crawl a path. It cannot make that path reachable, stop a CDN from challenging the request or guarantee a usable JSON response. On a live WooCommerce store we measured 200 for robots.txt and the Bridge’s static discovery file, but 403 with cf-mitigated: challenge for the live discovery API. Agent visibility needs an outside probe, not an assumption.Permission and delivery are different layers
The Robots Exclusion Protocol gives service owners rules that compliant crawlers are requested to honor. RFC 9309 is explicit that those rules are not access authorization. The inverse matters too: an Allow rule is not a network guarantee.
An automated client can be allowed by robots.txt and still fail before WordPress runs. DNS, TLS, a CDN, a web application firewall, rate limiting, hosting rewrites and the application itself all stand between the request and the catalog response.
A live store: static files pass, the API is challenged
On 26 September we repeated the same external request against three public paths on a participating WooCommerce store behind Cloudflare:
| Path | Observed response | What it proves |
|---|---|---|
/robots.txt | 200 text/plain | The policy file is reachable. |
/.well-known/kalicart-bridge.json | 200 application/json | A static Bridge signal is reachable. |
/wp-json/kalicart/v1/discovery | 403 HTMLcf-mitigated: challenge | The live WordPress route is intercepted before it can answer. |
Cloudflare documents cf-mitigated: challenge as the response marker for a Challenge Page. For a browser that page may be an interstitial. For a catalog client expecting JSON, it is not the requested resource.
Why checking in your own browser is insufficient
A merchant may open the same URL successfully because their browser has JavaScript, cookies, a recent challenge clearance or a different network reputation. The agent approaches from another network and may not execute an interactive challenge at all. “It works for me” and “it is reachable by an external catalog client” are different tests.
The useful observation must therefore come from outside the store and preserve the cause: reachable, disallowed by robots, challenged by an anti-bot layer, or no Bridge response. Folding all four into “not reachable” hides the decision the merchant actually needs.
What KaliCart Bridge 1.0.137 reports
The Federated Catalog panel shows the latest observation made by KaliCart Global. Version 1.0.137 distinguishes an anti-bot challenge and a robots exclusion from a missing Bridge. A challenged check is shown as Limited external access, with the observed provider when known and with the store’s current catalog status.
It is a read-only snapshot, not a live scan and not a firewall bypass. KaliCart Global respects robots.txt. The panel warns when the observation is old; it does not tell the merchant to weaken security or create a blanket allow rule.
Use three separate statements
- Allowed: robots.txt does not ask this client to stay away.
- Reachable: an external probe received the expected Bridge response.
- Discoverable: the store is currently included in a catalog or platform that an agent searches.
None of these implies ranking, a sale or authorization to transact. Keeping them separate is how an agent-readiness panel stays useful instead of optimistic.