I scanned 500 YC startups to see if AI search can read them

As far as robots.txt and the homepage HTML show, almost all of them can. On 12 October 2026 I checked both for 500 small YC B2B startups from the 2021–2024 batches, picked at random. Only 3 of 471 sites used robots.txt to block a crawler that fetches pages for AI answers, and one of those three blocked Google too. The more common gap was JavaScript: 17 of 453 homepages had no readable text until a script ran.

Our own guides tell you to check robots.txt for AI crawler rules, and so does most of the AI-search advice I've read. I expected to find small teams shutting ChatGPT out by accident, with a rule copied from a template or left over from staging. In this sample, they mostly aren't. The last section turns that into three checks for your own site.

The published counts I found come from big sites, and they look different. In June 2025, Cloudflare found GPTBot, OpenAI's training crawler, fully disallowed by 250 of 3,816 of its top 10,000 domains that had a robots.txt file (6.6%). Here it was 7 of the 403 sites that served an actual robots.txt file (1.7%, 0.8–3.5%). The two counts weren't made the same way, so treat the comparison as rough.

Few of these startups block AI answer crawlers

Six of the crawlers I checked fetch pages for AI answers, either to build a search index or because a user's question needs the page right now. 3 of 471 sites (0.6%, 0.2–1.9%) use robots.txt to block at least one of them. One of the three blocks every crawler, Googlebot included, with User-agent: * and Disallow: /, so that rule isn't aimed at AI in particular. Its homepage loads normally for people. The other two list each AI crawler they block by name.

Each count includes the site that blocks everything. Blocks on two of the user-triggered crawlers may not hold anyway, because a person starts those fetches. OpenAI's page on its bots warns that "robots.txt rules may not apply" to ChatGPT-User, and Perplexity's says Perplexity-User "generally ignores robots.txt rules". Anthropic says its bots honour robots.txt.

The startups that name AI crawlers mostly let them in

75 of the 471 sites (15.9%, 12.9–19.5%) name at least one of the nine in a User-agent line. 69 of those 75 block none of them from the homepage. Five I opened by hand, picked at random from those that block no answer crawler, list each crawler they name with Allow: /.

One split does show up, and OpenAI's own documentation describes it. Five sites block GPTBot, which collects training data, and leave OAI-SearchBot, which feeds ChatGPT's search, open. None does it the other way round. OpenAI says "each setting is independent of the others", so that combination tells OpenAI not to train on your pages without taking them out of ChatGPT's search results.

Seven block Google-Extended, Google's token for Gemini training and grounding. Google says blocking it doesn't keep a site out of Google Search (see the questions at the end).

61 of 471 have no robots.txt, and that blocks nothing

61 of 471 sites (13.0%, 10.2–16.3%) have no robots.txt: the address returns an error such as 404 instead of a file. The robots.txt standard, RFC 9309, lets a crawler fetch anything when the file is missing. Of the 410 sites that did return a file, 7 sent their app's web page in place of a robots file. A crawler finds no rules in it, so that blocks nothing either.

The case that does block everything is a server error. RFC 9309 tells a crawler to treat a robots.txt that returns a 5xx error as a ban on the whole site, for a while: after a long outage (the standard's example is 30 days) it may fall back to a cached copy or treat the file as missing. None of the 471 returned one (0%, 0–0.8%).

Empty JavaScript homepages were the more common gap

17 of 453 homepages (3.8%, 2.4–5.9%) sent HTML with no readable text in it: zero characters, an empty element for the app to mount into (such as #root), and everything built by JavaScript in the browser. The 413 homepages with 500 or more characters of text had a median of about 6,500.

Google can still read these 17 homepages, because Googlebot runs JavaScript before it indexes a page. The crawlers behind ChatGPT, Claude and Perplexity likely see none of their content, whatever robots.txt allows. That rests on Vercel and MERJ's crawler data: of the crawlers it named, OpenAI's, Anthropic's and Perplexity's included, only Apple's and Google's rendered pages. Vercel published it in December 2024 from a few months of traffic, mostly on its own network, so it may have changed since.

How I checked

The sample comes from YC's public company directory, through the open-source yc-oss mirror, as it stood on 12 October 2026. I kept companies listed as B2B and Active, from the Winter 2021 to Fall 2024 batches, with 1 to 20 people listed, and a website I could use. That left 788, and I drew 500 at random.

Each site got two checks, scored by the same code as the free scan on Shruwd, the tool we are building.

• robots.txt: for each of the nine names in the table, plus Googlebot for comparison, whether the rules that apply to it allow the homepage. A group that names it takes priority over User-agent: *, as the standard says.
• Homepage HTML: how much readable text arrives before any script runs. A page counts as JavaScript-only when it has under 500 characters of text and an empty mount element.

The scan fetched each site as a normal browser and named itself in a request header. It never posed as an AI crawler. I re-checked 17 sites by hand with a separate parser: all three answer-crawler blocks (the Googlebot one among them), 6 of the 17 JavaScript-only homepages, 3 homepages with text, and 5 of the 7 sites that sent a web page as robots.txt. All 17 agreed. Because 500 is a sample of the 788, each percentage carries a 95% range.

What this scan can't see

• Firewall and CDN rules. Cloudflare and other CDNs can turn an AI crawler away before robots.txt matters, and from outside, that usually shows only if you pretend to be the crawler. I didn't. A fake crawler from my address isn't the real one, and sites that check a crawler's IP address refuse fakes on purpose, so the test would report a block the real crawler never meets.
• Pages other than the homepage. The robots.txt verdicts are for the homepage's path. A rule that blocks /docs or /blog wouldn't show here.
• 31 homepages I couldn't read. 15 now redirect to a different domain, usually after a rename, so YC's link is out of date. 8 domains failed to resolve, and 8 sites didn't answer in time or refused the connection. Another 16 answered with an error or a login page. Nine of those were a 403 or 429, which can be a bot filter turning away an unknown visitor. I can't tell which from outside. That's why the homepage figures are out of 453. The robots.txt figures are out of 471, because 29 sites gave no usable answer at /robots.txt.
• Companies outside YC. These are young B2B companies with at most 20 people, by their own YC listing. Bigger companies, publishers and online shops may look different.

What to check on your own site

If your robots.txt looks like most of these, it probably isn't what keeps you out of AI answers. Check these three things instead, in this order.

Your homepage HTML

A crawler that skips JavaScript reads only what your server sends. This fetches that HTML without running any script and looks for your headline in it:

$html = (Invoke-WebRequest -Uri "https://yoursite.com" -UseBasicParsing).Content
$html.Contains("a phrase from your headline")

True means your headline is in the HTML. False means it arrives by JavaScript, or the phrase didn't match exactly: the check is case-sensitive, and markup such as a styled word or an & can break a phrase up. Try a second, plainer phrase before you conclude anything. If the text is missing, server-render or prerender the pages you want cited.

Your CDN's bot settings

A bot rule at Cloudflare, Vercel or another CDN can refuse a crawler that your robots.txt lets in. Look in the CDN's own bot or security settings, and in any record it keeps of what it refused, because those requests stop before they reach your server. For Cloudflare, what its AI settings do shows where to look.

The pages AI answers cite

If both are clean, access probably isn't the problem. Look at which pages the answers to your buyers' questions cite, and whether those pages name you. Why ChatGPT recommends your competitor goes through the citation side.

The free scan does the first check and the robots.txt read for any domain, with no account. With your crawler logs connected, Shruwd matches each request against the IP ranges the AI companies publish, so a page that keeps refusing the real OAI-SearchBot is flagged, and a fake one isn't counted. Your CDN's own refusals reach Shruwd only through some log paths, and the docs compare them.

Frequently asked questions

How many websites block AI crawlers?

It depends on which websites. News publishers block far more often: the Reuters Institute found that 48% of the 15 most-used news sites in each of ten countries were blocking OpenAI's crawlers by the end of 2023. The startups I checked block much less. Of 500 YC B2B startups, checked on 12 October 2026, 471 gave an answer at /robots.txt: 3 of those blocked a crawler that fetches pages for AI answers, and 7 blocked GPTBot, OpenAI's training crawler.

Should I block GPTBot?

Only if you want to tell OpenAI not to train on your pages. GPTBot is the training crawler. ChatGPT's search uses OAI-SearchBot, and OpenAI treats the robots.txt rule for each one independently. You can block the first and allow the second, as five startups in this sample did.

Does blocking Google-Extended remove you from AI Overviews?

Not by Google's account. Its crawler page says Google-Extended "does not impact a site's inclusion in Google Search". Search Console's help names it as the way "to limit training of the models used to generate responses" in Search's AI features.

Appearing has its own requirement. Google's guide to its AI features says a site "must be included in Search generative AI features in Search Console to be eligible" to show in them. The setting in Search Console is under Settings > Search generative AI and covers AI Overviews, AI Mode and Discover's generative AI features. It reached every site by 31 August 2026 and is set to include by default. An older Google page, on AI features in Search, lists two more controls: Googlebot's robots.txt rules, and page controls such as nosnippet and noindex. Both apply to Search as a whole, not to AI Overviews alone.

Is it bad to have no robots.txt?

Not for crawler access. A missing file, one that returns "not found", lets crawlers fetch everything under the robots.txt standard, and 61 of the 471 startups I could check have none. A robots.txt that returns a server error is the one to avoid: under the standard, crawlers treat the whole site as off limits for a while, and after a long outage (its example is 30 days) they may fall back to a cached copy or treat the file as missing.