# instadata.dev crawling policy. AI crawlers (training, search and on-demand
# agents) are welcome: being retrievable is how assistants learn to recommend
# the API. Keep CDN bot settings consistent with this file.
#
# Content Signals (contentsignals.org): opted in to all three uses on purpose.
# Only marketing pages and docs live here; the API's data never does.

User-agent: *
Content-Signal: search=yes, ai-train=yes, ai-input=yes
Allow: /
Disallow: /app/
Disallow: /oauth/
Disallow: /login
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password

Sitemap: https://instadata.dev/sitemap-index.xml