Can't find X-Robots tag!
-
Hi all. I've been checking out http://www.unthankbooks.com/ as it seems to have some indexing problems. I ran a server header check, and got a 200 response. However, it also shows the following:
X-Robots-Tag:
noindex, nofollowIt's not in the page HTML though. Could it be being picked up from somewhere else?
-
Hi,
You won't find the X-Robots tags in the HTML of your page as they're send within the headers of a document. These details could be set via your server. That's also the place you'll have to look to change something like this. I tried to give you an example, but somehow I can't ping your site and it returns a 406 error.
-
Tags can also be set in the .htaccess file, Blink - though not usually x-robots. (I use it to add canonical tags to pdfs to point them to the parent page, for example).
It's a long shot, but check there in case a dev decided to block the site from indexing that way during development.
Hope ya find it!
Paul
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
A client rebranded a few years ago and doesn't want to be associated with it's old brand name. He wishes not to appear when the old brand is searched in Google, is there something we can do?
The problem is there was redirection between the old branded site and the new one, and now when you type in the name of the old brand, the new one comes up. I have desperately tried to convince this client there is nothing we can do about it, dozens of news articles crop up with the two brands together as this was a hot topic a few years ago, but just in case I missed something I thought I'd ask the community of experts here on Moz. An example for this would be Tyco Healthcare that became covidien in 2007. When you type tyco healthcare, covidien crops up here and there. Any ideas? Thanks!
Intermediate & Advanced SEO | | Netsociety0 -
Why doesn't my website crawl by Google?
Hi mozzers and members, I am having issues, why my website: http://profilecosmeticsurgery.com/ crawl by Google? let me share more clearly when this starts happening. A month or around 45 days back our website is being indexed and crawled quite well without any issues with having .html extension pages with static built website.
Intermediate & Advanced SEO | | SEOOOOOoooooooo
We finally thought to change to .php version and make whole website and its pages to be treated dynamically.
Once we changed all changes, thereafter this issues started. It has been more than 45 days, our website isn't being crawled since then. I didn't know what are the things preventing this to? Please help. Thanks in Advance Capture1.PNG0 -
How can I tell where I was penalised?
It seems for sometime now I have not been showing up the top for my keywords when I use to be ranking #1 and #2 for my main keywords, however now, I don't show up anywhere - this has been for almost a year now, can anyone tell me how I can find out (apart from google web masters tools) where I was penalised? Thanks
Intermediate & Advanced SEO | | edward-may0 -
Will blocking urls in robots.txt void out any backlink benefits? - I'll explain...
Ok... So I add tracking parameters to some of my social media campaigns but block those parameters via robots.txt. This helps avoid duplicate content issues (Yes, I do also have correct canonical tags added)... but my question is -- Does this cause me to miss out on any backlink magic coming my way from these articles, posts or links? Example url: www.mysite.com/subject/?tracking-info-goes-here-1234 Canonical tag is: www.mysite.com/subject/ I'm blocking anything with "?tracking-info-goes-here" via robots.txt The url with the tracking info of course IS NOT indexed in Google but IT IS indexed without the tracking parameters. What are your thoughts? Should I nix the robots.txt stuff since I already have the canonical tag in place? Do you think I'm getting the backlink "juice" from all the links with the tracking parameter? What would you do? Why? Are you sure? 🙂
Intermediate & Advanced SEO | | AubieJon0 -
Can google read ajax
Looking to load a one page product view instead of 10 pages of pagination. Does google read ajax and see all 10 pages as 1 page.
Intermediate & Advanced SEO | | Archers1 -
Does a 'Certified Domain' help SEO?
I see that GoDaddy offer a 'Certified Domain' option. Does this help SEO at all?
Intermediate & Advanced SEO | | Techboy0 -
Should I robots block this directory?
There's about 43k pages indexed in this directory, and while helpful to end users, I don't see it being a great source of unique content for search engines. Would you robots block or meta noindex nofollow these pages in the /blissindex/ directory? ie. http://www.careerbliss.com/blissindex/petsmart-index-980481/ http://www.careerbliss.com/blissindex/att-index-1043730/ http://www.careerbliss.com/blissindex/facebook-index-996632/
Intermediate & Advanced SEO | | CareerBliss0 -
What is the downside with having too long of a title tag?
With Google placing so much relevance on title tags, it seems to help if you mention local cities within your title tag. I'm wondering if the positive of having more keywords in a title tag outweighs the negative of having too long of one?
Intermediate & Advanced SEO | | TLSNET0