Googlebot soon to be executing javascript - Should I change my robots.txt?
-
This question came to mind as I was pursuing an unrelated issue and reviewing a site's robots/txt file.
Currently this is a line item in the file:
Disallow: https://* According to a recent post in the Google Webmasters Central Blog: [http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") Googlebot is getting much closer to being able to properly render javascript. Pardon some ignorance on my part because I am not a developer, but wouldn't this require Googlebot be able to execute javascript? If so, I am concerned that disallowing Googlebot from the https:// versions of our pages could interfere with crawling and indexation because as soon as an end-user clicks the "checkout" button on our view cart page, everything on the site flips to https:// - If this were disallowed then would Googlebot stop crawling at that point and simply leave because all pages were now https:// ??? Or am I just waaayyyy over thinking it?...wouldn't be the first time! Thanks all! [](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better")
-
Excellent answer. Thanks so much Doug. I really appreciate it! Adding a "nofollow" attribute to the Checkout button is a good suggestion and should be fairly easy to implement. I realize that internal nofollows are not normally recommended, but in this instance, may not be a bad idea.
-
Hi Dana,
When you click on the checkout button - what's the mechanism for taking people to the https:// site. Is it just that the checkout link uses https:// in it's link? Is there some javascript wizardry you're particularly concerned about?
Even though googlebot follows this one link to the https version of the cart, it will still have all the other links on the previous page queued up to follow (non-https) so I don't think this will stop the crawl at that point. It would be a nightmare if googlebot stopped crawling hte entire site everytime it went down a rabbit hole!
That's not to say that you wouldn't want to consider no-following your checkout button. I'm sure neither you, nor google want to the innards of the cart pages to be indexed? There's probably other pages you'd rather Googlebot spent it's time finding right?
My take on the Google blog about understanding Javascript is that the aim is to try and do a better job discovering content that might be hidden by Javascript/Ajax. It's a problem for google when the raw html that they're crawling doesn't accurately reflect the content that is displayed in front of a real visitor.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can Yoast plugin change back old title automatically?
I have a problem with the first page on my website. Im using Wordpress and Yoast SEO plugin. After I have changed and updated some information on the page, I also wanted to change and make a better page title. That I understand is to be changed under pages, by scrolling down to the Yoast setting for the specific page. Is that right? However, I have changed the title over and over again, and asked Google to re-index the page. Everything is fine directly after that. But when I check again after 24-48 hours, the page have automatically changed back to the old title? How is that possible? I´ve tried about 5-10 times, but it does the same thing after 24-48 hours every time. Hope you people with great knowledge can help me out here. 🙂
Algorithm Updates | | Masse0 -
Have you ever seen or experienced a page indexed which is actually from a website which is blocked by robots.txt?
Hi all, We use robots file and meta robots tags for blocking website or website pages to block bots from crawling. Mostly robots.txt will be used for website and expect all the pages to not getting indexed. But there is a condition here that any page from website can be indexed by Google even the site is blocked from robots.txt; because crawler may find the page link somewhere on internet as stated here at last paragraph. I wonder if this really the case where some webpages have got indexed. And even we use meta tags at page level; do we need to block from robots.txt file? Can we use both techniques at a time? Thanks
Algorithm Updates | | vtmoz0 -
Reasons for change in organic traffic year-over-year
Need some help in answering the concern of my client. My client is concerned that organic visits from the 1<sup>st</sup> and 2<sup>nd</sup> quarter of 2016 are LESS than q1 & q2 of 2015. Overall there was about a 15% variance. Basically, I am trying to explain why the 2015 quarters have more organic traffic than 2016 (over the same time frame)? I have explained that to the client: There are external factors that affect SEO such an increase in competitor presence That we have been focusing on generating quality traffic to the site rather than just quantity That Google’s new SERPs layout is pushing organic rankings down Unfortunately, my client does not believe the above points to be valid, despite the fact he is still receiving the same amount of conversions, with less traffic. Why would his traffic be less in 2016? What else could cause this? Any help would be appreciated so I can help him.
Algorithm Updates | | TowerMarketing0 -
How much content is it safe to change?
I have read that it is unsafe to change more than 20% of your site’s content in any update. The rationale is that "Changing too much at once can flag your site within the Google algorithm as having something suspicious going on." Is this true, has anyone had any direct experiences of this or similar?
Algorithm Updates | | GrouchyKids0 -
Impact of recent On Page Optimisation changes had negative impact !
Hi I recently updated some page titles, H1 tags & on page content which overall has seen search results slip down following the first site crawl by google I assume. My question is, should I try to get back the rankings and test and change one thing at a time to see the impact right now or should i wait for a period of time for it to settle down once goggle has crawled the site a few times or will the subsequent crawls have no impact? Thanks Ash
Algorithm Updates | | AshShep10 -
Changes in Sitemap Indexation in GWT?
I've noticed some significant changes in the number and percentage of indexed URLs for the sitemaps we've been submitting to Google. I've been tracking these numbers directly from Google Webmaster Tools>Site Configuration>Sitemaps. We've made some changes that could be causing the changes we're seeing, but I want to confirm that this wasn't just a change in the way Google reports the indexation. Has anyone else noticed major changes, greater than a 30% change, in the indexation of your sitemaps in the past week? Thanks, Joe
Algorithm Updates | | JoeAmadon0 -
Did google change their algorithm over the past week?
I did some home page optimization with the seo moz on page key word optimization tool and we are now back in the top three in the past week (after dropping to page 3 a month or so ago). It seems that google has gone back to combining google places with organic searches. Has anyone else noticed this type of change? I did read some posts about panda 2.2, which seems to explain some of these findings. I am wondering if things are in flux or they may be more stable this way? Thanks for the insights.
Algorithm Updates | | fertilityhealth0