Why Can't Googlebot Fetch Its Own Map on Our Site?

[KempRugeLawGroup]

I created a custom map using google maps creator and I embedded it on our site. However, when I ran the fetch and render through Search Console, it said it was blocked by our robots.txt file. I read in the Search Console Help section that: 'For resources blocked by robots.txt files that you don't own, reach out to the resource site owners and ask them to unblock those resources to Googlebot."

I did not setup our robtos.txt file. However, I can't imagine it would be setup to block google from crawling a map. i will look into that, but before I go messing with it (since I'm not familiar with it) does google automatically block their maps from their own googlebot? Has anyone encountered this before?

Here is what the robot.txt file says in Search Console:

User-agent: *
Allow: /maps/api/js?
Allow: /maps/api/js/DirectionsService.Route
Allow: /maps/api/js/DistanceMatrixService.GetDistanceMatrix
Allow: /maps/api/js/ElevationService.GetElevationForLine
Allow: /maps/api/js/GeocodeService.Search
Allow: /maps/api/js/KmlOverlayService.GetFeature
Allow: /maps/api/js/KmlOverlayService.GetOverlays
Allow: /maps/api/js/LayersService.GetFeature
Disallow: /

Any assistance would be greatly appreciated.

Thanks,

Ruben

[KempRugeLawGroup]

Awesome, Croy. That was extremely helpful.

Thanks,

Ruben

[Charles-O]

Hello Ruben!

Not too long ago, I saw a hangout with John Mueller who got the same question and gives a logical answer. Here's the URL at timestamp 46:06 https://www.youtube.com/watch?v=i_xnKznRNCc&feature=player_embedded#t=2766

I'd transcript if I had the time, but it will take you a couple of minutes. It essentially says that even though it is embedded, it won't access that block of content because maps.googleapis.com/robots.txt blocks it. If I understood wrong, please let me know!

I had the same problem where part of my content was coming from the pins INSIDE the map, but since it was blocked, it couldn't reach the content itself. Had to do a workaround

Hope it helps!

EDIT: Just jumping back in to say that you should look back at your site. Seems like Google can correctly crawl its own map and the content served through it. Our site has no more problems related to this. November the 15th I still had issues, but today I've tested multiple times and it looked okay.

Croy