3 of 18 Korean shopping, clinic and booking platforms disallow listing pages even for Googlebot

DATAPREPREP · data as of 2026-09-14 · 2 min read한국어로 읽기

We checked the robots.txt of 18 Korean platforms against their listing-page URL patterns. 12 allow all three AI search crawlers, while Naver Smart Store, Naver Place, and Kakao Map places block Googlebot as well.

Many Korean businesses sell through a marketplace or list their store on a map platform instead of running their own site. What Google and AI search can read about them is set by that platform's robots.txt.

We fetched the robots.txt of 18 platforms on 2026-09-14 and tested 11 crawlers against each platform's listing-page URL pattern.

Finding 01

The pages disallowed for every crawler all belong to Naver and Kakao

Platform × crawler: is a listing page URL allowed?

Allowed
Blocked
No robots.txt (404)

Fetched on 2026-09-14. Rows run shopping → beauty & medical → local, booking & services.

The robots.txt of Naver Smart Store, Naver Place, and Kakao Map places blocks every crawler it does not name, Googlebot included. The file applies to outside crawlers; how those listings appear in Naver's or Kakao's own search is a separate matter.

Finding 02

12 platforms allow listings for AI search crawlers, and 4 of them still block at least one AI training crawler

Beauty & medical platforms allowed all three AI search crawlers most often

Share of platforms in each group that allow OAI-SearchBot, Claude-SearchBot and PerplexityBot.

# Content-Signal: search=yes, ai-input=yes, ai-train=no

First line of Gangnam Unni's robots.txt · fetched 2026-09-14

Finding 03

Only 2 platforms publish an llms.txt

8 of 18 platforms fell in the largest group

We only requested /llms.txt where robots.txt allowed that path.

Yeogi and Kmong publish one. Google's own guide says Google Search does not use such AI-specific files, so crawler access to the listing page matters first.

So what should a brand entering Korea check first?

Pick one listing URL on each marketplace you use and check it against that platform's robots.txt.
If citations in Google or outside AI answers matter, publish the same product and store facts where crawlers are allowed.
On your own domain you set robots.txt yourself; define search crawlers, AI search crawlers and AI training crawlers as separate user-agent groups.

Methodology

Evaluated with RFC 9309: the group naming the user agent applies if present, otherwise the * group; the longest matching rule wins.
Paths follow each platform's listing URL pattern with placeholder IDs. We did not fetch any listing page.
Coupang, Gmarket, Ohouse (오늘의집) and Olive Young returned a 403 block page for robots.txt, so they are excluded rather than counted as allowed. Ably was excluded because its listing URLs follow two patterns with different results.
Fetched on 2026-09-14 with the user agent MarketResearchBot/0.1, robots.txt and llms.txt only.
robots.txt is a set of crawl directives, not an access control. An allowed cell is not a record that the crawler actually fetched a page.