Googlebot soon to be executing javascript - Should I change my robots.txt?
-
This question came to mind as I was pursuing an unrelated issue and reviewing a site's robots/txt file.
Currently this is a line item in the file:
Disallow: https://* According to a recent post in the Google Webmasters Central Blog: [http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") Googlebot is getting much closer to being able to properly render javascript. Pardon some ignorance on my part because I am not a developer, but wouldn't this require Googlebot be able to execute javascript? If so, I am concerned that disallowing Googlebot from the https:// versions of our pages could interfere with crawling and indexation because as soon as an end-user clicks the "checkout" button on our view cart page, everything on the site flips to https:// - If this were disallowed then would Googlebot stop crawling at that point and simply leave because all pages were now https:// ??? Or am I just waaayyyy over thinking it?...wouldn't be the first time! Thanks all! [](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better")
-
Excellent answer. Thanks so much Doug. I really appreciate it! Adding a "nofollow" attribute to the Checkout button is a good suggestion and should be fairly easy to implement. I realize that internal nofollows are not normally recommended, but in this instance, may not be a bad idea.
-
Hi Dana,
When you click on the checkout button - what's the mechanism for taking people to the https:// site. Is it just that the checkout link uses https:// in it's link? Is there some javascript wizardry you're particularly concerned about?
Even though googlebot follows this one link to the https version of the cart, it will still have all the other links on the previous page queued up to follow (non-https) so I don't think this will stop the crawl at that point. It would be a nightmare if googlebot stopped crawling hte entire site everytime it went down a rabbit hole!
That's not to say that you wouldn't want to consider no-following your checkout button. I'm sure neither you, nor google want to the innards of the cart pages to be indexed? There's probably other pages you'd rather Googlebot spent it's time finding right?
My take on the Google blog about understanding Javascript is that the aim is to try and do a better job discovering content that might be hidden by Javascript/Ajax. It's a problem for google when the raw html that they're crawling doesn't accurately reflect the content that is displayed in front of a real visitor.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does changing text content on a site affects seo?
HI, i have changed some h1 and h2 , changed and added paragraphs,fixed plagiarism,grammar and added some pics with alt text, I have just done it today, I am ranking on second page QUESTION-1 is it gonna affect my 2 months SEO efforts? QUESTION -2 Do I have to submit sitemap to google again? QUESTION-3 does changing content on the site frequently hurts SEO?
Algorithm Updates | | Sam09schulz0 -
Google SERPs changes
Hi I wonder if anyone knew of any changes to the Google SERPs appearance in August 2015? We dropped in over a thousand visits to the homepage on brand so I wanted to find out why. Also, our DA went from 36 to 34 - does Google panda affect domain authority at all? Thank you
Algorithm Updates | | BeckyKey0 -
How soon before author rank becomes a major ranking factor?
Hi, I wanted to pose a question How soon do guys think itll be before author rank becomes a one of Googles major ranking factors? From what I can see the way they have designed it signals that it is only matter of time, before they start using it as a major ranking factor... And I have a question on Author ranks impact on the ability to sell a blog/site in the future. Surely if the blog is tied to an author(s) and the ranking of the site in the search engine is somewhat based on this authors author rank who is a part of the site/blog, then it becomes harder to sell the property if the author is not going to be a part of the property after the sale?. I look forward to your responses on this,
Algorithm Updates | | sanj50501 -
Google cant read my robots.txt from past 10 days
http://awesomescreenshot.com/08d1s6aybc hi, my robots.txt is http://wallpaperzoo.com/robots.txt google says it cant read and has postponed the crawl.. its been 10 days and no crawl.. please help me in solving this issue.. this is save with http://hdwallpaperzones.com/robots.txt
Algorithm Updates | | toxicpls0 -
Decline in traffic but no change in rankings
I'm comparing our best search traffic month in 2011 (March) with our current traffic (April)and have seen significant declines in traffic, despite no change in our rankings or even improved rankings for the same terms. Trying to sort out an explanation. We have been a white-hat SEO site since our inception over 10 years ago. Our SEO consultant doesn't think we've been affected by any algo changes, at least not to any significant degree. My only explanation for this possibly anomaly is: decrease in the use of the KW terms in search over time (how to determine?) generalized increase in PPC instead of organic search driving traffic possibility that Adv Web Rankings is no longer accurately collecting SERP rankings Does anyone have any other thoughts or considerations that might explain the decline in traffic, despite maintenance or improvement in rankings? Thanks.
Algorithm Updates | | ahw0 -
Javascript hidden divs, links to anchor content
Hello, I am working on a web project that breaks up its sections by utilizing hidden divs shown via javascript activated through anchor links. http://www.janandtom.com/ First question: Is this SEO suicide? I have confirmed that the content is being indexed by searching for specific text but have been led to believe that hidden div content will be afforded a lower 'importance'. One suggestion has having the text as display:block and then hiding it on page load. Will this make a difference? Second: Is there any way to have Google index the anchored content by the specific anchor text? An example for the second question: If you search google right now for: buyers like to look at floorplans Tom & Jan You will get a link to: http://www.janandtom.com but I would rather it be: [http://www.janandtom.com/#Interactive Floorplans](http://www.janandtom.com/#Interactive Floorplans) Sorry if this is redundant or addressed before. I tried searching the questions but wasn't getting and definitive direction to go and this project is a little unique for me. Also, I'm just getting my feet we into this 'high-end' seo (new member of SEOMoz) so please bear with me. Any help would be greatly appreciated. Thanks!
Algorithm Updates | | MASSProductions0 -
SEO updates and rank changes
We have been updating page titles and meta descriptions for a client (not changing ANY links and the content we are replacing is "fluff," no major keywords or any relevant information) yet in the past few weeks, rankings have plummeted. I used the SEOMoz grader to check and make sure we have the keywords in there, in the right places for the updated page source info, and we're getting A's yet for those same keywords, the website is nowhere to be found. For example for the phrase "organic t shirts," we get an A for this page: http://greenpromotionalitems.com/organic-t-shirts.htm but when searching organic t shirts, no Green Promotional Items... Ideas?
Algorithm Updates | | laidlawseo0