Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best blog practices for website
For my Insurance website blog, I use MOZ to help me find high DA authoritative sites, then either generate ideas from them, or rewrite the copy. If I rewrite the copy, I tend to pull from 2 - 3 top authoritative sites. Just so I don't get in trouble, but still offer the most concision information. _My question is, Is this ok to do? _ Secondly, I just read that on some .Gov sites the information is public, and that you can use it as long as you give credit. _My questions is, how do I tell which information is public? _ Thank you in advance 🙂
Moz Pro | | MissThumann0 -
Website Issues - Duplicate Content
Hello, I'm fairly new to using Moz and I logged on this morning to find Issues have been found in one of the websites - 22 High Priority and 44 Medium. I know it's due to duplicate content in the blog, but i can't figure out what is duplicated? I've only recently come on board this website so I don't know if the content has been plagiarised or what? The link to the site is here: delacyspa.co.uk Any help would be appreciated. Thanks zFxQmmd
Moz Pro | | Cowbang0 -
Whats the best way to research website relevant keywords
I wanted to know the general practices SEO marketers use when initially researching and revisiting relevant keywords for a website. I use this process Brain storm a list of keywords, between me and the client Check out competitors websites & SERP's look at Google adwords traffic analyser ( mostly as a point of reference and for increased ideas) How do others do it? what process do you find works - i'm based in the Uk would be great to get a UK perspective. Obviously you can't optimise for all keywords in your list how best is it to decide on the best value or less competitive keywords? I struggle to get stats for local keywords( not enough search data) - most my clients are local businesses and have limited surrounding service areas. thanks
Moz Pro | | Bristolweb0 -
What's Happened To OSE's External Links For My Site?
Hi, I'm just taking my first steps with Open Site Explorer. I've hit a problem that I'd be really grateful for some help with. I'm running a website (that I didn't create) and want to get a clear picture of the inbound links. When I enter the URL into the OSE search bar, there are no external domains listed under the 'Inbound Links' and 'Linking Domains' tabs. Only internal site links are registered. Setting the filters, "only external" + "pages on this root domain", under the Inbound Links tab, does bring up a list of external sites. But I haven't had to enter filters just to see a list of inbound links with any other sites. Also, the breakdown under the Linking Domains tab remains the same - only links within my site are shown. Does this ring a bell? Any ideas what I might be doing wrong, or what might be wrong with my site to cause this problem? Cheers for helping, Josh
Moz Pro | | JoshAustin460 -
Why does SEOMoz crawler ignore robots.txt?
The SEOMoz crawler ignores robots.txt It also "indexes" pages marked as noindex. That means it is filling up the reports with things that don't matter. Is there any way to stop it doing that?
Moz Pro | | loopyal0 -
Linking C Blocks - SEOMoz says its a good thing?
In the competitve analysis, one competitor have more Linking C Blocks, Seomoz has a tick by it almost like its a better thing. Surely a site with the same administrative relationship is not going to help you as much from a linking point of view.
Moz Pro | | sanchez19600 -
How do you get Mozbot to crawl your website
I trying to get the mozbot to crawl my site so I can get new crawl diagnostics info. Anyone know how this can be done?
Moz Pro | | Romancing0 -
Best way to use OSE Advanced Reports (Repost after Q&A issues)
I can see that Open Site Explorer's Advanced Reports feature is very powerful in terms of sifting through large amounts of link data, but does anyone have any useful tips on how to get the most out it? I want to use my 5 reports a month, but I want to make to maker sure I'm making the most of that allowance.
Moz Pro | | seanmccauley1