Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multi language websites need proper reporting
Common guys! It can not be THAT difficult to add 3 characters to your reports..
Moz Pro | | nans
Instead of seeing 3 times Google BE, I would like to see the language code too. Example: Google BE-NL Google BE-FR Google BE-EN This is really not helpful for me or my clients (screenshot): https://docs.google.com/file/d/0B4BB1WWdu2ncYWZ4TTRqaWR1YVk/edit?usp=drivesdk There's more in the world than the US and English... Can you do this please? edit?usp=drivesdk0 -
How to monetize a video website ?
My website is popularvid.com I want to monetize it just like youtube but i don't know how to do that. Please give me suggestions to monetize my video website. How to make my website rank page one google? Please give me brilliant ideas about new features to be added that will make my website beautiful. I'm a newbie here. Thank you
Moz Pro | | idamardyana0 -
My website was hacked last Thursday
My business website was hacked (for the 2nd time in 12 months) last Thursday and all data lost. I've been rebuilding the site and database since then but I'm still getting Hacking Warnings each day. The latest warning says: Dear Colin/Administrator,
Moz Pro | | NileCruises
Someone has attempted to inject SQL into your domain:
HACK DETECTED!
PHP TYPE
IP: 94.100.17.134
Scriptname: /index.cfm
PathInfo: /index.cfm
QueryString: src=http%3A%2F%2Fpicasa.com.oprst.in%2Fshow.php%3Fid%3D16907217 My Technical advisro tells me the IP address is that of Inferno Solutions of The Netherlands. I wonder if anyone has suffered hacking like this what steps they too and what I could do about the potential hackers? Colin0 -
Why do some sites have a higher Page Authority than Domain Authority in OSE??
Hi, I have noticed when using OSE and enter a domain you very often see a higher Page Authority than Domain Authority. If someone could explain why this would happen then I would be very grateful - its my current understanding that page authority would ALWAYS be LESS THAN Domain Authority but that is not always the case (I have seen cases where PA is more than 10 higher then DA) Here's an example where PA > DA http://www.opensiteexplorer.org/links.html?site=www.primelocation.com Thanks
Moz Pro | | James770 -
How long does it take for a campaign website crawl to be completed?
Our campaign website crawl has been 'crawling' now for 5 days. Is this a normal phenomenon or is something hanging up?
Moz Pro | | Discountvc0 -
Why is Roger crawling pages that are disallowed in my robots.txt file?
I have specified the following in my robots.txt file: Disallow: /catalog/product_compare/ Yet Roger is crawling these pages = 1,357 errors. Is this a bug or am I missing something in my robots.txt file? Here's one of the URLs that Roger pulled: <colgroup><col width="312"></colgroup>
Moz Pro | | MeltButterySpread
| example.com/catalog/product_compare/add/product/19241/uenc/aHR0cDovL2ZyZXNocHJvZHVjZWNsb3RoZXMuY29tL3RvcHMvYWxsLXRvcHM_cD02/ Please let me know if my problem is in robots.txt or if Roger spaced this one. Thanks! |0 -
Filtering OSE Results
Issue: When I export OSE Linking Pages results to .CSV, I'd like to filter only unique domains. It seems to me that there should be a way to do this with Excel, perhaps pivot table. Anyone have a quick solution for this? Alternatively, I can use Linking Domains, which gives all unique roots, but then i lose follow/nofollow filter. Thoughts?
Moz Pro | | Gyi0 -
In OSE is there a way to see the total number or links coming from each domain?
Open site explorer will only show up to to 25 links per domain but in order to fully understand the link profile for the site I'm working on, I need to know how many links are coming from each domain - when I download the linking domains report, it doesn't contain this information. Is there a way to access it?
Moz Pro | | jabbate0