Googlebot soon to be executing javascript - Should I change my robots.txt?
-
This question came to mind as I was pursuing an unrelated issue and reviewing a site's robots/txt file.
Currently this is a line item in the file:
Disallow: https://* According to a recent post in the Google Webmasters Central Blog: [http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") Googlebot is getting much closer to being able to properly render javascript. Pardon some ignorance on my part because I am not a developer, but wouldn't this require Googlebot be able to execute javascript? If so, I am concerned that disallowing Googlebot from the https:// versions of our pages could interfere with crawling and indexation because as soon as an end-user clicks the "checkout" button on our view cart page, everything on the site flips to https:// - If this were disallowed then would Googlebot stop crawling at that point and simply leave because all pages were now https:// ??? Or am I just waaayyyy over thinking it?...wouldn't be the first time! Thanks all! [](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") -
Excellent answer. Thanks so much Doug. I really appreciate it! Adding a "nofollow" attribute to the Checkout button is a good suggestion and should be fairly easy to implement. I realize that internal nofollows are not normally recommended, but in this instance, may not be a bad idea.
-
Hi Dana,
When you click on the checkout button - what's the mechanism for taking people to the https:// site. Is it just that the checkout link uses https:// in it's link? Is there some javascript wizardry you're particularly concerned about?
Even though googlebot follows this one link to the https version of the cart, it will still have all the other links on the previous page queued up to follow (non-https) so I don't think this will stop the crawl at that point. It would be a nightmare if googlebot stopped crawling hte entire site everytime it went down a rabbit hole!
That's not to say that you wouldn't want to consider no-following your checkout button. I'm sure neither you, nor google want to the innards of the cart pages to be indexed? There's probably other pages you'd rather Googlebot spent it's time finding right?
My take on the Google blog about understanding Javascript is that the aim is to try and do a better job discovering content that might be hidden by Javascript/Ajax. It's a problem for google when the raw html that they're crawling doesn't accurately reflect the content that is displayed in front of a real visitor.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
50% drop in search, no changes to site over 2 days, no notifications, A rank...
My URL is: http://applianceassistant.com
Algorithm Updates | | applianceassistant
With no changes to my site, I suddenly experienced a huge drop in search queries on Aug1. Your company has still given me an overall rating of A. I just thought you may be able to help or be interested in my case due to it's strange nature. Due to some suggestions on the webmaster forums, I have disavowed all low quality back links to the site, and I am currently working through each page trying to make the key words a little less spammy. Here are some screen shots of the action...
https://lh6.googleusercontent.com/-WgXUf-lvUyg/U-nrWNgspPI/AAAAAAAAAEI/imoI190LUns/s1600/Analytics_081214.tiff
https://lh4.googleusercontent.com/-srmvn288rr0/U-pxlwoycVI/AAAAAAAAAEg/ckmyX_2Sl_Y/s1600/PAGES_AUG.tiff
https://lh3.googleusercontent.com/-DVCYxhkutbQ/U-pxpQVfYfI/AAAAAAAAAEo/MN9PiLFT-zs/s1600/pages_july.tiff This appears to be almost a 50% 2 year set back. Any ideas or suggestions are greatly appreciated0 -
Changes in Google "Site:" Search Algorithm Over Time?
I was wondering if anyone has noticed changes in how Google returns 'site:' searches over the past few years or months. I remember being able to do a search such as "site:example.com" and Google would return a list of webpages where the order may have shown the higher page rank pages (due to link building, etc) first and/or parent category pages higher up in the list of the first page (if relevant) first (as they could have higher PR naturally, anyways). It seems that these days I can hardly find quality / target pages that have higher page rank on the first page of Google's site: search results. Is this just me... or has Google perhaps purposely scrambled the SERPS somewhat for site: searches to not give away their page ranking secrets?
Algorithm Updates | | OrionGroup1 -
Drop in Traffic from Google, However no change in the rankings
I have seen a 20% drop in traffic from google last week (After April 29th). However when I try to analyze the rank of the keywords in the google results that send me traffic they seem to be the same. Today (6th March) Traffic has fallen further again with not much/any visible change in the rankings. Any ideas on what the reason for this could be? I have not made any changes to the website recently.
Algorithm Updates | | raghavkapur0 -
Javascript hidden divs, links to anchor content
Hello, I am working on a web project that breaks up its sections by utilizing hidden divs shown via javascript activated through anchor links. http://www.janandtom.com/ First question: Is this SEO suicide? I have confirmed that the content is being indexed by searching for specific text but have been led to believe that hidden div content will be afforded a lower 'importance'. One suggestion has having the text as display:block and then hiding it on page load. Will this make a difference? Second: Is there any way to have Google index the anchored content by the specific anchor text? An example for the second question: If you search google right now for: buyers like to look at floorplans Tom & Jan You will get a link to: http://www.janandtom.com but I would rather it be: [http://www.janandtom.com/#Interactive Floorplans](http://www.janandtom.com/#Interactive Floorplans) Sorry if this is redundant or addressed before. I tried searching the questions but wasn't getting and definitive direction to go and this project is a little unique for me. Also, I'm just getting my feet we into this 'high-end' seo (new member of SEOMoz) so please bear with me. Any help would be greatly appreciated. Thanks!
Algorithm Updates | | MASSProductions0 -
Google and Content at Top of Page Change?
We always hear about how Google made this change or that change this month to their algorithm. Sometimes it's true and other times it's just a rumor. So this week I was speaking with someone in the SEO field who said that this week a change occurred at Google and is going to become more prevalent where content placed at the "top of the fold" on merchant sites with products are going to get better placement, rather than if you have your products at top with some content beneath them at the bottom of the page. Any comments on this?
Algorithm Updates | | applesofgold0 -
SEO updates and rank changes
We have been updating page titles and meta descriptions for a client (not changing ANY links and the content we are replacing is "fluff," no major keywords or any relevant information) yet in the past few weeks, rankings have plummeted. I used the SEOMoz grader to check and make sure we have the keywords in there, in the right places for the updated page source info, and we're getting A's yet for those same keywords, the website is nowhere to be found. For example for the phrase "organic t shirts," we get an A for this page: http://greenpromotionalitems.com/organic-t-shirts.htm but when searching organic t shirts, no Green Promotional Items... Ideas?
Algorithm Updates | | laidlawseo0 -
Changes in Sitemap Indexation in GWT?
I've noticed some significant changes in the number and percentage of indexed URLs for the sitemaps we've been submitting to Google. I've been tracking these numbers directly from Google Webmaster Tools>Site Configuration>Sitemaps. We've made some changes that could be causing the changes we're seeing, but I want to confirm that this wasn't just a change in the way Google reports the indexation. Has anyone else noticed major changes, greater than a 30% change, in the indexation of your sitemaps in the past week? Thanks, Joe
Algorithm Updates | | JoeAmadon0 -
Rankings changing based on location within a country... normal?
I recently had a satellite office across the country come to me and say that they couldn't find us on Google, based on a number of keywords they were searching on. I thought that isn't right... I know we rank for those terms. So, I did a search here, and there we were for those very terms, and ranking quite nicely. Sooo, what's going on there? I know there are variations from Google.com to Google.ca in terms of ranking. But within Google.ca I've not seen this before. Can anyone shed some light on that?
Algorithm Updates | | atcosl0