Googlebot soon to be executing javascript - Should I change my robots.txt?
-
This question came to mind as I was pursuing an unrelated issue and reviewing a site's robots/txt file.
Currently this is a line item in the file:
Disallow: https://* According to a recent post in the Google Webmasters Central Blog: [http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") Googlebot is getting much closer to being able to properly render javascript. Pardon some ignorance on my part because I am not a developer, but wouldn't this require Googlebot be able to execute javascript? If so, I am concerned that disallowing Googlebot from the https:// versions of our pages could interfere with crawling and indexation because as soon as an end-user clicks the "checkout" button on our view cart page, everything on the site flips to https:// - If this were disallowed then would Googlebot stop crawling at that point and simply leave because all pages were now https:// ??? Or am I just waaayyyy over thinking it?...wouldn't be the first time! Thanks all! [](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better")
-
Excellent answer. Thanks so much Doug. I really appreciate it! Adding a "nofollow" attribute to the Checkout button is a good suggestion and should be fairly easy to implement. I realize that internal nofollows are not normally recommended, but in this instance, may not be a bad idea.
-
Hi Dana,
When you click on the checkout button - what's the mechanism for taking people to the https:// site. Is it just that the checkout link uses https:// in it's link? Is there some javascript wizardry you're particularly concerned about?
Even though googlebot follows this one link to the https version of the cart, it will still have all the other links on the previous page queued up to follow (non-https) so I don't think this will stop the crawl at that point. It would be a nightmare if googlebot stopped crawling hte entire site everytime it went down a rabbit hole!
That's not to say that you wouldn't want to consider no-following your checkout button. I'm sure neither you, nor google want to the innards of the cart pages to be indexed? There's probably other pages you'd rather Googlebot spent it's time finding right?
My take on the Google blog about understanding Javascript is that the aim is to try and do a better job discovering content that might be hidden by Javascript/Ajax. It's a problem for google when the raw html that they're crawling doesn't accurately reflect the content that is displayed in front of a real visitor.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Indexed, though blocked by robots.txt: Need to bother?
Hi, We have intentionally blocked some of the website files which were indexed for years. Now we receive a message "Indexed, though blocked by robots.txt" in GSC. We can ignore as per my knowledge? Are any actions required about this? We thought of blocking them with meta tags but these are PDF files. Thanks
Algorithm Updates | | vtmoz1 -
Have you ever changed the logo anchor text from "logo" to "keyword"? How Google considers?
Hi all, We know that generally logo with the website homepage link is the first link crawled by Google and other search engines. Can we change the anchor text from "logo" to "keyword"? Have any one tried or seen others doing? How Google considers it? Thanks
Algorithm Updates | | vtmoz1 -
Is it stil a rule that Google will only index pages up to three tiers deep? Or has this changed?
I haven't looked into this in a while, it used to be that you didn't want to bury pages beyond three clicks from the main page. What is the rule now in order to have deep pages indexed?
Algorithm Updates | | seoessentials0 -
Webmaster Guidelines Change History
Does any one have the dates of changes to Googles Webmaster Guidelines?
Algorithm Updates | | MiroAsh0 -
Google Algorithm change? - Brand name now overwriting title tag?
Anyone else noticing this happening? In Google search results, many of my sites are now showing up in the following fashion... "Site name: page title" I read a few articles in the past few days that state that Google may be playing with the algo but have not read anything from Google directly. I should add that I first noticed this on Feb. 21 and have seen it rolling out more and more since. I have only noticed it on a few competitor websites thus far. Edit:Some links talking about the subject http://www.seroundtable.com/google-brand-title-appending-16432.html http://semandseo.blogspot.ca/2013/03/google-brand-title-in-search.html http://www.designbigger.com/blog/seo/google-rewrites-page-titles-to-push-brand-over-keywords/
Algorithm Updates | | mattylac0 -
Title of home page is changed to domain name in SERPs
Hi, We have a unique problem, we are getting a totally different title in Google serps for a large site. When we search with domain name with space in google.com. We are getting title as domain name with space. We don't have any Open Directory listing. We don't have any cannonical issues and other pages with title as domain name. Can you please tell us what we have to do get our original title back in SERP ? Thanks, With Regards,
Algorithm Updates | | semshah1430 -
Changes in Sitemap Indexation in GWT?
I've noticed some significant changes in the number and percentage of indexed URLs for the sitemaps we've been submitting to Google. I've been tracking these numbers directly from Google Webmaster Tools>Site Configuration>Sitemaps. We've made some changes that could be causing the changes we're seeing, but I want to confirm that this wasn't just a change in the way Google reports the indexation. Has anyone else noticed major changes, greater than a 30% change, in the indexation of your sitemaps in the past week? Thanks, Joe
Algorithm Updates | | JoeAmadon0