# [SCRUM-687] Candidate/collaborator personal data is reachable via # unguessable tokens, not real auth (interview links, decline links, # collaboration-acceptance links, booking links, objection links, # legacy client-share # links) — these must never be indexable, and today's SPA-shell-only # rendering only accidentally limits what crawlers can see. Once # SSR/prerendering ships (planned), these routes would serve real, # server-rendered candidate data to any crawler that reads this file. # # Every User-agent group below repeats the SAME Disallow list on purpose: # robots.txt has no shared/include mechanism, and per spec a crawler that # matches a *named* group (e.g. "Googlebot") uses ONLY that group's rules # and ignores "User-agent: *" entirely. A Disallow added only under "*" # would silently not apply to any bot with its own named group below — # which is exactly the mistake that would let e.g. GPTBot or Googlebot # crawl a token URL despite the wildcard block. # # This list is intended to be provably complete against src/App.tsx's # route table (tests/unit/robotsTokenSurfaces.test.ts pins it): every # authenticated app route and every candidate/token route is listed. # Marketing, blog, legal AND the public careers pages (/jobs, /jobs/:id, # /jobs/confirm — careers-meta-proxy renders JobPosting markup for them) # are deliberately left off so they stay fully crawlable; the recruiter # routes under a job id (/jobs/:jobId/pipeline, /shortlist, …) are blocked # with the wildcard form "/jobs/*/", and the advert wizard explicitly (see the per-route noindex meta tag — # src/components/NoIndexMeta.tsx — for the second, page-rendered half of # this fix on the token routes themselves). User-agent: Googlebot Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / User-agent: Bingbot Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / User-agent: Twitterbot Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / User-agent: facebookexternalhit Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / # AI crawlers — explicitly allowed onto marketing/blog content (not # blanket-blocked), but bound by the same candidate/app Disallow list as # every other crawler above. User-agent: GPTBot Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / User-agent: OAI-SearchBot Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / User-agent: ChatGPT-User Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / User-agent: ClaudeBot Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / User-agent: Claude-SearchBot Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / User-agent: PerplexityBot Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / User-agent: * Disallow: /admin Disallow: /auth Disallow: /logout Disallow: /qa-join Disallow: /qa-login Disallow: /reset-password Disallow: /verify-email Disallow: /onboarding Disallow: /waitlist Disallow: /organisation-suspended Disallow: /join-pending Disallow: /dashboard Disallow: /ai-job-description Disallow: /jobs/wizard Disallow: /jobs/*/ Disallow: /job-descriptions Disallow: /cv-scoring Disallow: /candidates Disallow: /interview Disallow: /i/ Disallow: /decline/ Disallow: /decline-feedback/ Disallow: /object/ Disallow: /profile Disallow: /team Disallow: /billing Disallow: /blog-admin Disallow: /config Disallow: /organisation-config Disallow: /organisation-profile Disallow: /cv-recovery Disallow: /brand-profiles Disallow: /company-config Disallow: /crm Disallow: /tutorials Disallow: /manage-schedules Disallow: /calendar Disallow: /activate-calendar/ Disallow: /collaborators Disallow: /collaborate/ Disallow: /accept-invitation/ Disallow: /schedule/ Disallow: /shared-candidate/ Disallow: /c/ Allow: / Sitemap: https://skreen.ai/sitemap.xml