Prevent Google from crawling Ajax
-
With Google figuring out how to make Ajax and JS more searchable/indexable, I am curious on thoughts or techniques to prevent this.
Here's my Situation, we have a page that we do not ever want to be indexed/crawled or other. Currently we have the nofollow/noindex command, but due to technical changes for our site the method in which this information is being implemented if it is ever displayed it will not have the ability to block the content from search. It is also the decision of the business to not list the file in robots.txt due to the sensitivity of the content. Basically, this content doesn't exist unless something super important happens, and even if something super important happens, we do not want Google to know of its existence.
Since the Dev team is planning on using Ajax/JS to pull in this content if the business turns it on, the concern is that it will be on the homepage and Google could index it. So the questions that I was asked; if Google can/does index, how long would that piece of content potentially appear in the SERPs? Can we block Google from caring about and indexing this section of content on the homepage?
Sorry for the vagueness of this question, it's very sensitive in nature and I am trying to avoid too many specifics. I am able to discuss this in a more private way if necessary.
Thanks!
-
Toby, thanks for the suggestion! I believe that this will help accomplish what we need. My Dev gave the "oh S" I should've thought of that response.
-
You may find that you have to wrap the code that gets called when Ajax fires in something to catch the user agent. I.e. if your making an Ajax request to a php script in order to return data, you could wrap that php code in something like this (please excuse the Sudo code):
if(in_array($_SERVER['HTTP_USER_AGENT'], $knownagents){
//known webspider, or blocked agent, return nothing.
return "";
} else {
//not a known spider so continue.
}
?>
Thats very generalised but you get the idea. I put a short list together in JSON format a while back, you can find it here if its of any use: https://www.source-control.co.uk/knownspiders/spiders.php
PM me if you need any more specific help than that with development, hopefully someone else will have a slightly easier way of dealing with this though heh
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Different snippet in Google for same page
Hello, I have a question regarding the snippet of a specific case: When i search the homepage by searching the business name, i find the correct snippet of the homepage (with the meta description that was entered). If i search it via site:www. it still show the default meta description. Has anybody had experience with this? Is there a way to change the snippet of site:www.? Does it influence SEO? Thank you!
Intermediate & Advanced SEO | | conversal0 -
How to leverage Google Images?
My Google search rankings are improving rapidly at the moment, but a lot of my rankings are for images (presume that means the images are appearing near the top in Google Images). How do I capitalise on that? It's not really much help to me that my images are popular unless it results in traffic to the pages where those images are used. I am running Wordpress so I have the option to have images embed as "no link", "link to attachment page", "link to original image", etc. Is there any advantage of using one of these over the other? I'd really like to set it up so that when a Google Images user clicks "View Image" it loads the attachment page or the host content page rather than the image. Bad SEO? I'm not sure if the fact that I'm using Jetpack Photon CDN image hosting will make this more complicated or not. Tony
Intermediate & Advanced SEO | | Gavin.Atkinson0 -
Is WordPress a Blog in the eyes of Google?
Hi, My online Shop is based on WordPress with the WooCommerce plugin. Now, I have met a SEO guy who told me that's bad in the eyes of Google: Because Google apparently sees my website as a blog and not as a E-Commerce site. Wow, this statement really confused my, since I am working so hard on content and good rankings. Any opinions on this would be appreciated. Best, Robin
Intermediate & Advanced SEO | | soralsokal0 -
Google Places - Advanced Question
Hi All, Ive found multiple threads about previous issues but I haven't found any tailored to my specific question.I know there are a large amount of factors so I wanted to see if any other individuals had ran into this previously. We are currently in a centralized position in a major city. We are discussing moving the main office about 15 miles away into another city, moving us out of the main city where we have been for the past 3 years. The city where we are currently located has a lot more GEO search volume compared to the new city search terms and variants of. If we move will our local rankings drop when someone searches in the city where we were previously? How long would it take for this ranking to fall? Or would we still rank because we are moving a short distance away and have a large amount of citations there? I know we would need to change over all our online directories, on page etc..Any other suggestions on a smooth transition? I know there are many factors that go into this and any past experience, guidance and/or assistance is greatly appreciated. Cheers!
Intermediate & Advanced SEO | | PRKEL0 -
Google + under Google business domain email account
Hello there, I have a quick and straight question and I am hoping to find answer here. What do we do with a G+ profile that was set up through a business domain's email account that is used by more than one person? We want to use the company name, but we can't as it is considered personal email account although it is under business domain verified by Google. Is there a way that we ask Google to change it and allow us to use the name of the company or should we just deactivate it? Thanks in advance!
Intermediate & Advanced SEO | | montauto0 -
Google Experiment
Hi there, are there any implications for organic seo for implementing google experiment. Where you send users to a new page?
Intermediate & Advanced SEO | | pauledwards0 -
Why will google not index my pages?
About 6 weeks ago we moved a subcategory out to becomne a main category using all the same content. We also removed 100's of old products and replaced these with new variation listings to remove duplicate content issues. The problem is google will not index 12 critcal pages and our ranking have slumped for the keywords in the categories. What can i do to entice google to index these pages?
Intermediate & Advanced SEO | | Towelsrus0 -
Google indexing flash content
Hi Would googles indexing of flash content count towards page content? for example I have over 7000 flash files, with 1 unique flash file per page followed by a short 2 paragraph snippet, would google count the flash as content towards the overall page? Because at the moment I've x-tagged the roberts with noindex, nofollow and no archive to prevent them from appearing in the search engines. I'm just wondering if the google bot visits and accesses the flash file it'll get the x-tag noindex, nofollow and then stop processing. I think this may be why the panda update also had an effect. thanks
Intermediate & Advanced SEO | | Flapjack0