Prevent Google from crawling Ajax
-
With Google figuring out how to make Ajax and JS more searchable/indexable, I am curious on thoughts or techniques to prevent this.
Here's my Situation, we have a page that we do not ever want to be indexed/crawled or other. Currently we have the nofollow/noindex command, but due to technical changes for our site the method in which this information is being implemented if it is ever displayed it will not have the ability to block the content from search. It is also the decision of the business to not list the file in robots.txt due to the sensitivity of the content. Basically, this content doesn't exist unless something super important happens, and even if something super important happens, we do not want Google to know of its existence.
Since the Dev team is planning on using Ajax/JS to pull in this content if the business turns it on, the concern is that it will be on the homepage and Google could index it. So the questions that I was asked; if Google can/does index, how long would that piece of content potentially appear in the SERPs? Can we block Google from caring about and indexing this section of content on the homepage?
Sorry for the vagueness of this question, it's very sensitive in nature and I am trying to avoid too many specifics. I am able to discuss this in a more private way if necessary.
Thanks!
-
Toby, thanks for the suggestion! I believe that this will help accomplish what we need. My Dev gave the "oh S" I should've thought of that response.
-
You may find that you have to wrap the code that gets called when Ajax fires in something to catch the user agent. I.e. if your making an Ajax request to a php script in order to return data, you could wrap that php code in something like this (please excuse the Sudo code):
if(in_array($_SERVER['HTTP_USER_AGENT'], $knownagents){
//known webspider, or blocked agent, return nothing.
return "";
} else {
//not a known spider so continue.
}
?>
Thats very generalised but you get the idea. I put a short list together in JSON format a while back, you can find it here if its of any use: https://www.source-control.co.uk/knownspiders/spiders.php
PM me if you need any more specific help than that with development, hopefully someone else will have a slightly easier way of dealing with this though heh
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Silo Structure in the eye of google?
does silo structure has a positive point on Google Ranking or not, and what is the importance of internal linking, how google see the internal linking content as compared to less internal linking, I'm trying an experiment I do a lot of internal backlinking in Website Unionwell as compared to Website B (which has apparently less internal Links) so with your experience in SEO field which site will get traffic rapidly.
Intermediate & Advanced SEO | | saimkhanna0 -
Missing Google verification
I just went to check my client sites in Google search console and noticed a whole bunch of them no longer 'verified'. They were all previously verified. Why would they suddenly change status to 'not verified'? Does this affect anything (eg. search analytics data flowing through to GA)? Does this mean I have to verify all over again?
Intermediate & Advanced SEO | | muzzmoz0 -
Reclaiming Ranking positions in Google
We have a website we are working on that was ranking well in Google but since having a hosting upgrade has completely dropped in rankings. When a hosting upgrade was made, the developer added an incorrect robots.txt file that restricted the site from being found, hence resulting in lost rankings. We have since sorted out that issue so the robots.txt is OK. However, ranking results have yet to be reclaimed. We are unsure why these rankings haven't rebounded back, as it has been a while now. The site is https://www.brightonpanelworks.com.au. We have since also attempted to add a sitemap however to help the site be better crawled and to regain rankings, however, it appears that sitemap generators are having problems creating a sitemap for this site and we are not sure why. And we are not sure whether this may relate to why Google has not picked up on pages and ranking results have not be restored. If you have any ideas as to how we can reclaim rankings to the strong positions they were in previously, that would be much appreciated. We believe we may be missing something here that is not allowing webpages to be picked up and ranked by Google.
Intermediate & Advanced SEO | | Gavo0 -
Google does not index image sitemap
Hi, we put an image sitemap in the searchconsole/webmastertools http://www.sillasdepaseo.es/sillasdepaseo/sitemap-images.xml it contains only the indexed products and all images on the pages. We also claimed the CDN in the searchconsole http://media.sillasdepaseo.es/ It has been 2 weeks now, Google indexes the pages, but not the images. What can we do? Thanks in advance. Dieter Lang
Intermediate & Advanced SEO | | Storesco0 -
How to reverse declining Google rankings?
We have a long established business since 2004 and have been fortunate that having been one of the original companies in our industry, we have always enjoyed strong Google rankings. Unfortunately, these have been steadily declining over the past couple of years and a comparison of August to date against the equivalent period last year has seen a 20% drop in traffic from Google. We don't believe that it is being caused by a penalty and rather is the result of some strong players entering our market and tightening their focus which has caused us to take a dip in rankings. We are guilty of being complacent in our SEO - largely due to not knowing what to do and being scared to touch it when it was working in case we broke it! - but now it's time to fight back. We still have a strong site, good traffic levels and a strong product offering. We have knowledge of SEO and resources in house, but are not experts by any means. Our current plan is to: perform a technical site audit, fixing the issues highlighted by the Moz Pro Software put strong emphasis on our blog, writing daily about the latest news and events in our industry provide weekly content articles which are more in depth than the daily blog articles and which will be of interest to our community undertake surveys and publish infographics and statistics with the hope of being picked up in national newspapers Are there any key elements that we are missing out in this plan, or is that it in a nutshell? Any help and advice is greatly appreciated.
Intermediate & Advanced SEO | | simonukss0 -
Google News and Meta Title
Hi,
Intermediate & Advanced SEO | | JohnPalmer
1. I just read this article: https://www.seroundtable.com/google-news-titles-h1-19876.html
Google want the same title. no problem. but what about the brand? for example
POST TITLE BLA BLU | My Brand
The "post title bla blu" is the H1 and title of the article and | My Brand is my brand...
I can keep it as is with the My brand? or remove it? what about posts with long title for example "POST TITLE BLA BLU POST TITLE BLA BLU | My Brand"
What is you suggestion, I know Google doesn't show all the text and we'll see "...". it's still important to write the brand name in the title or just the post title? (without the brand). Thanks,0 -
How can I make sure Google is crawling a link from an iframe (video)?
Do they crawl backlinks from an iframe example from a Youtube video embedded in a blog post? TIA!
Intermediate & Advanced SEO | | zpm20140 -
Could this work for Google Reconsideration Request?
One of my websites has received the following message: We've reviewed your site and we still see links to your site that violate our quality guidelines. Specifically, look for possibly artificial or unnatural links pointing to your site that could be intended to manipulate PageRank. Examples of unnatural linking could include buying links to pass PageRank or participating in link schemes. We encourage you to make changes to comply with our quality guidelines. Once you've made these changes, please submit your site for reconsideration in Google's search results. If you find unnatural links to your site that you are unable to control or remove, please provide the details in your reconsideration request. I have used LinkResearchTools DTOX to locate unnatural links and remove them. So far I've been able to remove or nofollow 50/350 and that's as far as I can ever go. The rest of the websites either don't respond or don't have any contact information. I added another 300 suspicious websites to my list and I'll try to get the links manually removed. Hopefully I can get 100/650 websites (and a bit more links) removed in total - at most. That is my estimate. I've been thinking to use Google Disavow Tool for the rest and make sure to submit a nicely written report with spreadsheets to Google - when I get to the reconsideration point. What are your thoughts on this?
Intermediate & Advanced SEO | | zorsto0