Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My website not getting organic hits
Hello Moz, I am enjoying being here i just bought moz pro, and am digging for what to change in my website, Apart from that I just wanna know why my website not getting rank in search engine, if someone can do quick audit for me it will be obligation on me. querease(dot)com
Moz Pro | | querease0 -
OSE not displaying data
Is anyone else having a problem with OSE not displaying any data except for domain and page authority?
Moz Pro | | SamCUK0 -
54 new 404 errors on my website?
Hi There In the latest report I have 54 404-errors. All last week, previously I had 2 404s that I fixed. In report say: Title404 : ErrorMeta DescriptionTraceback (most recent call last): File "build/bdist.linux- x86_64/egg/downpour/init.py", line 391, in _error failure.raiseException() File "/usr/local/lib/python2.7/site- packages/twisted/python/failure.py", line 370, in raiseException raise self.type, self.value, self.tb Error: 404 Not FoundMeta RobotsNot present/emptyMeta RefreshNot present/empty Are these normal 404 errors I have to look at and fix? Or is this some script that running on my server and causing errors? In general - what should I do to fix this? Thanks Dean
Moz Pro | | Passanger880 -
What period of time do OSE social media metrics represent?
When one hovers over the social media metrics in Open Site Explorer the popups say that the numbers represent the total number of times the URL has been posted, liked, tweeted or +1'd. Those numbers are [appear to be] more dynamic than the data in OSE's other reports and seem to get a fresh analysis each time the URL is entered. So I'm wondering what period of time is it counting in it's "total number of times" determination? If I'd like to track the data to get an idea of, say, total number of Facebook Shares for a one month period, how often should I check the numbers?
Moz Pro | | BlastAM1 -
OSE Backlink results - reported link not actually there?
Not a complaint, but a question to understand how the research tool operates: When I run backlink checks on websites, often the reported link is not only not on the page, but it's not found anywhere on the site. I use several tools to search for the link url as well as for the keyword. Why does the tool report a link is there, but I cannot find the links in some cases? Is there a lag in the information the tool is using, making it not quite up to date, or is it something else? Thanks much!
Moz Pro | | AdamThompson0 -
Difference between Seomoz Competitive Domain Analysis and OSE link data?
Any reason these numbers are so much different? SeoMoz Competitive Domain Analysis:
Moz Pro | | fibers
Followed Linking Root Romains: 288
Total Linking Root Domains: 338 Open Site Explorer
Linking ROot Romains: 1270 -
Has anyone noticed slight drops in the Domain mozRank of their sites since the OSE update?
There's been no significant changes to their link profiles, so I'm thinking how mozRank is measured may have changed. Anyone else have this issue?
Moz Pro | | RiseSEO0