Prevent Google from crawling Ajax
-
With Google figuring out how to make Ajax and JS more searchable/indexable, I am curious on thoughts or techniques to prevent this.
Here's my Situation, we have a page that we do not ever want to be indexed/crawled or other. Currently we have the nofollow/noindex command, but due to technical changes for our site the method in which this information is being implemented if it is ever displayed it will not have the ability to block the content from search. It is also the decision of the business to not list the file in robots.txt due to the sensitivity of the content. Basically, this content doesn't exist unless something super important happens, and even if something super important happens, we do not want Google to know of its existence.
Since the Dev team is planning on using Ajax/JS to pull in this content if the business turns it on, the concern is that it will be on the homepage and Google could index it. So the questions that I was asked; if Google can/does index, how long would that piece of content potentially appear in the SERPs? Can we block Google from caring about and indexing this section of content on the homepage?
Sorry for the vagueness of this question, it's very sensitive in nature and I am trying to avoid too many specifics. I am able to discuss this in a more private way if necessary.
Thanks!
-
Toby, thanks for the suggestion! I believe that this will help accomplish what we need. My Dev gave the "oh S" I should've thought of that response.
-
You may find that you have to wrap the code that gets called when Ajax fires in something to catch the user agent. I.e. if your making an Ajax request to a php script in order to return data, you could wrap that php code in something like this (please excuse the Sudo code):
if(in_array($_SERVER['HTTP_USER_AGENT'], $knownagents){
//known webspider, or blocked agent, return nothing.
return "";
} else {
//not a known spider so continue.
}
?>
Thats very generalised but you get the idea. I put a short list together in JSON format a while back, you can find it here if its of any use: https://www.source-control.co.uk/knownspiders/spiders.php
PM me if you need any more specific help than that with development, hopefully someone else will have a slightly easier way of dealing with this though heh
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
HTTPS Google Search Console Verification
Hi we have clients http version of our site verified in our search console but for some reason the https version is not verified, do you usually have to install another HTML tag to do this? Cheers.
Intermediate & Advanced SEO | | bridhard80 -
Recovery after recent Google update
Hi guys. This is somewhat a continuation for this topic: https://moz.com/community/q/january-2016-massive-rankings-fluctuations After that update, several of our clients and our website as well have experienced high fluctuation rankings period, which ended up in huge drops - 10-20 spots. Which, obviously, made everyone unhappy. Anybody knows what exactly the change was about? What should we fix/take a look at, analyze again? We aren't using any shady techniques or black hat. Everything is honest. All metrics, number of backlinks etc are going up, no major changes have been recently made. Please, help!
Intermediate & Advanced SEO | | DmitriiK0 -
Website not coming up properly on Google
Hello, our website (http://www.roguevalleymicro.com/index.php) is not coming up properly on Google search (for example, when you search for Rogue Valley Microdevices on Google). We believe that there is something wrong with the website source code, and Google cannot index it properly. However, your Crawl Test results did not indicate any such problems. Can someone help us with some advice please?
Intermediate & Advanced SEO | | medved441 -
ECommerce Google PR mystery
Dear all, Our eCommerce site has the following structure Home .. PR=5 Category .. PR=4 (linked from home) Sub-Category Linked from Category PR=Un-ranked The domain has several years and perform well the site is here: http://tinyurl.com/5v9hrql Any idea or suggestion? Thank you Claudio
Intermediate & Advanced SEO | | SharewarePros0 -
Google Plus Authorship
Situation Description: I have a website called Website A. I wish to migrate alot of the content from Website A to Website B. Website B will be on a completely different domain name and environment. Authors of Website A will act as contributing authors for Website B. It is also possible that other contributing authors of other websites C and D commit to writing content on Website B. Questions (1) Does it make sense to create a google plus profile under UserA@websiteA.com and link from content on websiteB to their google plus profile under UserA@websiteA.com? (2) Does AuthorRank affect PageRank? If yes, if I take the above approach would websiteA be effected or websiteB since the content writers of websiteA are contributing to websiteB? (3) Is it ok for userA to have a corporate google plus profile assuming he might also have another google plus profile under a different address? I always think it make sense that there exists a google plus profile at an employee level and another google plus profile at a personal level. (4) If an employee leaves the company, do I leave his/hers Google Plus profile alive? The fact that no more content would be published under that particular profile, would that negatively effect author rank over time? (5) Another interesting observation is that UsaToday, CNN etc do not use authorship? No authors link to their twitter profile or google plus profile. Shouln't they be doing this in terms of author rank or is author rank not that important? Thank you
Intermediate & Advanced SEO | | seo12120 -
Google+ Pages on Google SERP
Do you think that a Google+ Page (not profile) could appear on the Google SERP as a Rich Snippet Author? Thanks
Intermediate & Advanced SEO | | overalia0 -
How to improve Google Places Account
Hello, A friend of mine has a google places account. Her website is http://debidonner.com/ You can see her places account listed under "life coach scottsdale arizona" I've attached a screenshot of her categories. I just added "NLP Coach" and "Personal Coach" today, so results are only for the other categories. We're looking for ways to improve it. We've had 43 clicks for driving directions. 4 map clicks, and 0 website clicks (how good is that, by the way) Thanks! screenshot-google-places.jpg screenshot-google-places2.jpg
Intermediate & Advanced SEO | | BobGW0 -
Adding Millions of Products to Google
What is the best way to submit all of your product pages, millions, to Google for serps? XML, RSS, Google Product Search, etc. These are products that are updated on a daily basis, and change often.
Intermediate & Advanced SEO | | Copstead0