Social & communities · Original data
7 of 11 social platforms disallow public posts for every AI search crawler
We evaluated the robots.txt of 11 social platforms against their public post URL patterns. 7 block all three AI search crawlers, 2 (YouTube, Medium) allow all three, and 10 allow posts for Googlebot. Reddit blocks every crawler, Googlebot included.
Whether a brand's Instagram, YouTube or LinkedIn posts can become sources in AI answers starts with whether the platform allows crawlers to access them, and that is set in robots.txt.
We fetched the robots.txt of 11 social platforms on 2026-09-14 and tested 11 crawlers against each platform's public post URL pattern.
Finding 01
Only 2 platforms allow all three AI search crawlers
- Allowed
- Blocked
- No robots.txt (404)
Fetched on 2026-09-14. Hover a cell for the platform and user-agent name.
All three allowed: YouTube and Medium. Partially allowed: LinkedIn and Facebook. The rest disallow public posts for all three.
Finding 02
10 platforms allow Googlebot; Reddit blocks every crawler
One bar per crawler. Length = platforms, out of 11, whose post URLs are allowed for that crawler.
# Reddit believes in an open internet, but not the misuse of public content.
Supporting links in Google's AI Overviews come from pages indexed in Google Search, so posts allowed for Googlebot keep that route. Posts disallowed for AI search crawlers are out of reach for ChatGPT search and Perplexity.
Reddit content reaches Google and OpenAI through Data API agreements, not robots.txt; we compare those documents in the companion article on platform settings.
Finding 03
Facebook, Instagram and Threads belong to one company but ship different robots.txt rules
| Platform | AI search crawlers | AI training crawlers | Googlebot |
|---|---|---|---|
| 0 / 3 | 0 / 4 | Allowed | |
| 1 / 3 | 3 / 4 | Allowed | |
| Threads | 0 / 3 | 0 / 4 | Allowed |
So how do you make social content count in AI answers?
- For ChatGPT search and Perplexity, keep the original where those crawlers are allowed: YouTube, Medium, or your own site.
- On platforms that allow Googlebot, such as Instagram and TikTok, write captions and profiles in the words people search.
- Link social posts back to an original on your own domain, where you control robots.txt.
Methodology
- Evaluated with RFC 9309: the group naming the user agent applies if present, otherwise the * group; the longest matching rule wins.
- Paths follow each platform's public post URL pattern with placeholder IDs. We did not fetch any post.
- LinkedIn was tested on /posts/ URLs; /feed/update/ URLs in the same file are blocked even for Googlebot. Substack was excluded because its post URL pattern could not be confirmed.
- Fetched on 2026-09-14 with the user agent MarketResearchBot/0.1, robots.txt files only.
- robots.txt is a set of crawl directives, not an access control. An allowed cell is not a record that the crawler actually fetched a post.