Crawling & platforms · Original data
Sitemap lines in social platforms' robots.txt range from 182 on Facebook to none on LinkedIn, TikTok, Quora, and Reddit
We fetched the robots.txt of 11 platforms on 2026-09-14. The most Sitemap lines were Facebook (182), Pinterest (50), and Threads (15); LinkedIn, TikTok, Quora, and Reddit had none. 166 of Facebook's 182 sitemap files are named as group post lists. Search result URLs are disallowed for Googlebot on 5 of 8 platforms.
A sitemap tells crawlers where a site keeps its lists of URLs, and a Sitemap line in robots.txt is one place to announce it. Reading those lines shows which bundles of URLs a platform puts forward on its own.
We fetched the robots.txt of 11 social platforms on 2026-09-14, counted the Sitemap lines, and ran a Googlebot check on each platform's URL types.
Finding 01
Sitemap lines range from 182 on Facebook to zero
Fetched on 2026-09-14. We counted lines in robots.txt only and did not fetch the sitemap files.
LinkedIn, TikTok, Quora, and Reddit have no Sitemap line in robots.txt. Whether they submit sitemaps some other way can't be told from the file.
Finding 02
By file name, Facebook lists mostly group posts, Instagram lists mostly profiles & creators and places & local businesses, and Threads lists mostly profiles & creators
| Platform | Sitemap lines | Groups by file name | Example file name (largest group) |
|---|---|---|---|
| 182 | Group posts 166 · Profiles & creators 8 · Places & local businesses 1 · Groups & boards 1 · Other (help pages, events, general lists) 6 | groups_popular_posts_sitemap_qe | |
| 50 | Posts & pins 18 · Groups & boards 11 · Topics, tags & keywords 7 · Other (help pages, events, general lists) 14 | image_pin_sitemap_recent_engagement_tier_1_www.pinterest.com | |
| Threads | 15 | Posts & pins 1 · Profiles & creators 10 · Topics, tags & keywords 3 · Other (help pages, events, general lists) 1 | threads_profiles_sitemap |
| 5 | Profiles & creators 2 · Places & local businesses 2 · Topics, tags & keywords 1 | ig_seo_profile_sitemap | |
| YouTube | 2 | Other (help pages, events, general lists) 2 | sitemap |
| X | 1 | Other (help pages, events, general lists) 1 | sitemap |
| Medium | 1 | Other (help pages, events, general lists) 1 | sitemap |
Facebook keeps separate profile lists with follower counts in the file name (for example profiles_2500_followers). All 5 of Instagram's files fall under profiles & creators / places & local businesses / topics, tags & keywords; none has posts in its name.
File names are internal labels, so they don't tell you how many URLs a file holds or how entries are chosen.
Finding 03
5 of 8 platforms disallow search result URLs for Googlebot
| Platform | Allowed for Googlebot | Disallowed, and the matching rule |
|---|---|---|
| Member profile, Company page, Post (/posts/), Article (Pulse), Newsletter, Job posting, Event, School page, Showcase page, Hashtag feed | Feed post (/feed/update/) (Disallow: /feed/update/), Group (Disallow: /groups/), Search results (Disallow: /search*) | |
| Post, Reel, Profile, Hashtag, Place, Story | Post comments (Disallow: /*/comments/) | |
| YouTube | Video, Short, Channel (@handle), Playlist, Community post, Hashtag | Search results (Disallow: /results) |
| TikTok | Video, Profile, Tag, Discover topic | Search results (Disallow: /search?) |
| Pin, Profile, Ideas topic | Search results (Disallow: /*/pins/) | |
| X | Post, Profile, Hashtag, Search results | None |
| Page post, Page, Group post, Event, Search results | None | |
| Threads | Post, Profile, Search results | None |
| Quora | Question, Profile, Topic | Answer (Disallow: */answer/) |
| Medium | Story, Profile, Tag | Search results (Disallow: /search?q=) |
X, Facebook, and Threads have no rule disallowing search URLs in their Googlebot group. Other types disallowed for Googlebot: LinkedIn feed post (/feed/update/), LinkedIn group, Instagram post comments, and Quora answer.
So where should social content live if you want it reachable from Google?
- Match the bundles the platform submits itself. The largest groups were Facebook: group posts; Pinterest: posts & pins; Threads: profiles & creators; Instagram: profiles & creators and places & local businesses.
- Content left only in Instagram post comments and Quora answer sits on URLs disallowed for Googlebot. Put it in the main post.
- What goes into a platform's sitemaps is the platform's call. Your own site's sitemap is yours to write and submit in Search Console and Naver Search Advisor.
Methodology
- Fetched on 2026-09-14 with the user agent MarketResearchBot/0.1, robots.txt only. We did not fetch sitemap files or posts.
- Sitemap lines were counted case-insensitively. Grouping by file name may not match what the files contain.
- URL checks follow RFC 9309: the group naming the user agent applies if present, otherwise the * group; the longest matching rule wins.
- Reddit disallows every crawler, so it is excluded from the URL type table. AI crawler results are in the companion article on social platforms and AI crawlers.
- Allowed in robots.txt means a crawler may fetch the URL, not that the URL is indexed.