Google Adsbot crawling order confirmation pages?
-
Hi,
We have had roughly 1000+ requests per 24 hours from Google-adsbot to our confirmation pages. This generates an error as the confirmation page cannot be viewed after closing or by anyone who didn't complete the order.
-
How is google-adsbot finding pages to crawl that are not linked to anywhere on the site, in the sitemap or linked to anywhere else?
-
Is there any harm in a google crawler receiving a higher percentage of errors - even though the pages are not supposed to be requested.
-
Is there anything we can do to prevent the errors for the benefit of our network team and what are the possible risks of any measures we can take?
This bot seems to be for evaluating the quality of landing pages used in for Adwords so why is it trying to access confirmation pages when they have not been set for any of our adverts?
We included "Disallow: /confirmation" in the robots.txt but it has continued to request these pages, generating a 403 page and an error in the log files so it seems Adsbot doesn't follow robots.txt.
Thanks in advance for any help,
Sam
-
-
Hi Sam,
I can see how this might be concerning. Without knowing your site, I can't confirm anything but answers to your questions:
- Bots have been known to "fill out forms" before and at least Googlebot has been known to find pages through the use of Chrome (a user using Chrome). There are many ways, but if you are sure that there is no link to it anywhere, I wouldn't worry about it.
- No. That is what header codes are there for, to let the bots know what is there, what is forbidden, etc.
- Other than robots.txt, there isn't any way to stop them from sending in requests. If it gets out of hand, you can try talking to AdWords directly, but more than likely, this is not causing an issue.
Overall, I'd just let it happen. Let them get the 403 error and they'll figure it out. As long as this isn't showing in the organic index, you should be fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My product category pages are not being indexed on google can someone help?
My website has been indexed on google and all of its pages can be found on google except for the product category pages - which are where we want our traffic heading to, so this is a big problem for us. Our website is www.skirtinguk.com And an example of a page that isn't being indexed is https://www.skirtinguk.com/product-category/mdf-skirting-board/
Intermediate & Advanced SEO | | chelseaskirtinguk0 -
Why doesn't my website crawl by Google?
Hi mozzers and members, I am having issues, why my website: http://profilecosmeticsurgery.com/ crawl by Google? let me share more clearly when this starts happening. A month or around 45 days back our website is being indexed and crawled quite well without any issues with having .html extension pages with static built website.
Intermediate & Advanced SEO | | SEOOOOOoooooooo
We finally thought to change to .php version and make whole website and its pages to be treated dynamically.
Once we changed all changes, thereafter this issues started. It has been more than 45 days, our website isn't being crawled since then. I didn't know what are the things preventing this to? Please help. Thanks in Advance Capture1.PNG0 -
Why does Google rank a product page rather than a category page?
Hi, everybody In the Moz ranking tool for one of our client's (the client sells sport equipment) account, there is a trend where more and more of their landing pages are product pages instead of category pages. The optimal landing page for the term "sleeping bag" is of course the sleeping bag category page, but Google is sending them to a product page for a specific sleeping bag.. What could be the critical factors that makes the product page more relevant than the category page as the landing page?
Intermediate & Advanced SEO | | Inevo0 -
Weird Page switch for a keyword in Google Rankings
Over this past weekend Google switched the page which usually showed in search results for keyword benchmarking. It went from from http://www.apqc.org/benchmarking to http://www.apqc.org/benchmarking-portal/osb. Also on Google the Rankings for the keyword 'benchmarking' sank from 15 to 47 for http://www.apqc.org/benchmarking Just looking for some theories or ideas or anyone that has had this happen to them.
Intermediate & Advanced SEO | | inhouseninja0 -
Can you appear in both Google local and universal results on the page?
I have a client that ranks 2nd for a google local result, and the local results are the first set of results being displayed, making it in fact the 2nd result on the page for the keyword. There aren't any blended results on this page, it's straightforward local results, then universal results directly after. They are concerned that they are not appearing in the universal results high enough (they're on page 5 with a different url than the homepage). I'm not sure why their #2 local ranking isn't good enough since those are consistently the same results, but I have to ask; Can you appear in both local and universal results on the same page?
Intermediate & Advanced SEO | | MichaelWeisbaum0 -
Rel=author, google plus, picture in Article page SERP
Hello, Could someone explain the easiest way to use Google Plus and rel="author" to claim our articles written by us and get our picture beside them in the Google SERPS site: nlpca(dot)com
Intermediate & Advanced SEO | | BobGW0 -
Duplicate page content and Duplicate page title errors
Hi, I'm new to SeoMoz and to this forum. I've started a new campaign on my site and got back loads of error. Most of them are Duplicate page content and Duplicate page title errors. I know I have some duplicate titles but I don't have any duplicate content. I'm not a web developer and not so expert but I have the impression that the crawler is following all my internal links (Infact I have also plenty of warnings saying "Too many on-page links". Do you think this is the cause of my errors? Should I implement the nofollow on all internal links? I'm working with Joomla. Thanks a lot for your help Marco
Intermediate & Advanced SEO | | marcodublin0 -
What would cause a drastic drop in pages crawled per day?
The site didn't go down. There were no drop in rankings, or traffic. But we went from averaging 150,000 pages crawled per day, to ~1000 pages crawled per day. We're now back up to ~100,000 crawled per day, but we went more than a week with only 1000 pages being crawled daily. The question is, what could cause this drastic (but temporary) reduction in pages crawled?
Intermediate & Advanced SEO | | Fatwallet0