Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best way to increase the DA and PA of website?
Hello, I hope everybody is fine. I am looking for a right way to increase the DA and PA of the website and its webpage. I have a lesco duplicate bill info related webpage which I want to increase the PA. Kindly guys guide me the legit way of increasing the Page Authority. Thanks
Moz Pro | | rankvizguru1 -
Can increasing website pages decrease domain authority?
Hello Mozzers! Say there is a website with 100 pages and a domain authority of 25. If the number of pages on this website increases to 10,000 can that decrease its domain authority or affect it in any way?
Moz Pro | | MozAddict0 -
Blog Post Not Picked up by OSE
How come this blog post of mine which got lots of links has not been picked up by Open Site Explorer: http://infographicdirectory.wordpress.com/2012/08/22/the-internet-a-decade-later/
Moz Pro | | Eavesy0 -
My website was hacked last Thursday
My business website was hacked (for the 2nd time in 12 months) last Thursday and all data lost. I've been rebuilding the site and database since then but I'm still getting Hacking Warnings each day. The latest warning says: Dear Colin/Administrator,
Moz Pro | | NileCruises
Someone has attempted to inject SQL into your domain:
HACK DETECTED!
PHP TYPE
IP: 94.100.17.134
Scriptname: /index.cfm
PathInfo: /index.cfm
QueryString: src=http%3A%2F%2Fpicasa.com.oprst.in%2Fshow.php%3Fid%3D16907217 My Technical advisro tells me the IP address is that of Inferno Solutions of The Netherlands. I wonder if anyone has suffered hacking like this what steps they too and what I could do about the potential hackers? Colin0 -
How come there are no links to my website according to SEOmoz Competive domain analysis, while in google webmaster i do see links.?
I dont see any links to at all when i do a Competitive Domain Analysis in SEOmoz. However i do see links in google webmaster tools. this strikes me as odd. Also when i use open site exployer my website dont seem te be found. In google im on page 9 on my focus keyword so i do think there are links to my site. I would like to know what i can do so i can analyse my links in seomoz Competitive domain analysis. Many thanks. url: http://www.sadpanda.nl
Moz Pro | | Aquive0 -
In OSE is there a way to see the total number or links coming from each domain?
Open site explorer will only show up to to 25 links per domain but in order to fully understand the link profile for the site I'm working on, I need to know how many links are coming from each domain - when I download the linking domains report, it doesn't contain this information. Is there a way to access it?
Moz Pro | | jabbate0 -
OSE Linking Domains CSV Export Blank?
For some client domains, I've tried to do a CSV export from the OSE Linking Domains page, I get a CSV that contains the typical OSE header text, but no output records. Before I click the CSV link I always choose "show domains with links to All pages on the Root Domain". It doesn't happen with every domain I've tried this with, and when I've had the time/patience, some of the time if I try hours later, I get data. The problem though is I rarely have the luxury of running reports a second time during the audit cycle. Has anyone else experienced this? Is it a bug, or just an export fail due to system resources or something?
Moz Pro | | AlanBleiweiss0