Hi,

We have had roughly 1000+ requests per 24 hours from Google-adsbot to our confirmation pages. This generates an error as the confirmation page cannot be viewed after closing or by anyone who didn't complete the order.

1. How is google-adsbot finding pages to crawl that are not linked to anywhere on the site, in the sitemap or linked to anywhere else?

2. Is there any harm in a google crawler receiving a higher percentage of errors - even though the pages are not supposed to be requested.

3. Is there anything we can do to prevent the errors for the benefit of our network team and what are the possible risks of any measures we can take?

This bot seems to be for evaluating the quality of landing pages used in for Adwords so why is it trying to access confirmation pages when they have not been set for any of our adverts?

We included "Disallow: /confirmation" in the robots.txt but it has continued to request these pages, generating a 403 page and an error in the log files so it seems Adsbot doesn't follow robots.txt.

Thanks in advance for any help,

Sam