Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my website internal links on moz is 0?
My Web site: https://www.basictrailers.com.au/ I have been using Moz for almost 3 months, from the start to now, the internal links are always 0. However, if I check the links on my Google Search Console, it is over 1500. So what is the problem? If Moz is trying to make its algorithm similar to Google, why that they even cannot find my internal links. That is ridiculous. 0fhTnQG
Moz Pro | | Lilycharlie2 -
/essions/essions keeps appending to 1 url on our website
Moz keeps giving us an error showing URL too long, when I investigate the offending url, I get this in the crawl. We can't work out what /essions is or why it's appending to the end of the url. Is this a Moz or website issue? <colgroup><col width="841"></colgroup>
Moz Pro | | NickWillWright
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/ |
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/essions/ |
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/essions/essions/ |
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/essions/essions/essions/ |
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/essions/essions/essions/essions/ |
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/essions/essions/essions/essions/essions/ |
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/essions/essions/essions/essions/essions/essions/ |
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/essions/essions/essions/essions/essions/essions/essions/ |
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/essions/essions/essions/essions/essions/essions/essions/essions/ |
| https://www.mywebsite/singita-lebombo-lodge/essions/essions/essions/essions/essions/essions/essions/essions/essions/essions/essions/essions/ |0 -
Moz Website Grader Export Omits Check Graphics
So, I love the Grader, but when I export to PDF the pretty little graphics go away. Is this correctable?
Moz Pro | | ClickMonsterIM0 -
OSE Report Shows HTTP Status "No Data"
Open Site Explorer shows HTTP Status "No Data" on 1,575 urls of the 3,335 urls listed in the "Top Pages" report for my site. What does this mean?
Moz Pro | | debc0 -
54 new 404 errors on my website?
Hi There In the latest report I have 54 404-errors. All last week, previously I had 2 404s that I fixed. In report say: Title404 : ErrorMeta DescriptionTraceback (most recent call last): File "build/bdist.linux- x86_64/egg/downpour/init.py", line 391, in _error failure.raiseException() File "/usr/local/lib/python2.7/site- packages/twisted/python/failure.py", line 370, in raiseException raise self.type, self.value, self.tb Error: 404 Not FoundMeta RobotsNot present/emptyMeta RefreshNot present/empty Are these normal 404 errors I have to look at and fix? Or is this some script that running on my server and causing errors? In general - what should I do to fix this? Thanks Dean
Moz Pro | | Passanger880 -
Can someone explain to me why OSE shows two different numbers for Linking Root Domains in the header and in the results area?
See Photo Below. 271 in the header, 500 plus in the results section. I just want an accurate number to chart my progress from here on out. Majestic numbers seem inflated too much. Google Webmaster Tools too. g58HX.png
Moz Pro | | DRPower0 -
Strange Website Activity
Hello, I have been building websites for about 4 months now and finally had my first real success with a website. I found a niche that I was able to get on the first page with. This site was fine and then boom it dropped to #17 around the time of the recent Google changes. So I just thought it was that, I decided to add it as a campaign on here. This is when I noticed that the pages were not being crawled. So being new at all of this, I researched that. Of course, the main things were is the privacy and robots.txt. It is running on wordpress and I know not to set the blog to private, but I wasn't familiar with how to edit the robots.txt. I found a good plugin that easily allowed me to set the text to allow all bots. It seemed to be set to the normal wordpress settings before, and I never had a problem with any of my websites not being crawled. Anyway, once I just set it to allow all bots on ALL of my website, the pages started being crawled again. My traffic went back up and I was on the first page all day yesterday. Today, so far big drop off. So, I deleted the campaign and set up a new one. Sure enough no pages crawled yet. I made some security changes using Bulletproof Security and another plugin to see if that effects it. Nothing yet. I am just really confused as to what is going on, so if any of you have any ideas that would be great. It is a simple site, and I made some changes like theme when I was trying to figure out why the pages weren't being crawled. So it is not the most beautiful design right now. Also, I try my best to put up well-written useful content, so I don't think that is the issue for the rankings drops. I don't have many if any actual backlinks yet because of the newness of my site, could be the reason for it acting strange BUT none of that explains the pages not being crawled at the same time my site drops????? Sorry so long but had to explain it all! Thanks in advance to anyone who has anything to say about this situation! Edit: I should clarify, I put the the security plugins on yesterday while I was on the first page and have deleted them to see if that allows the pages to be crawled. Sorry if I wasn't clear.
Moz Pro | | iheartkelby0 -
What is the quickest way to get OSE data for many URLs all at once?
I have over 400 URLs in a spreadsheet and I would like to get Open Site Explorer data (domain/page authority/trust etc) for each URL. Would I use the Linkscape API to do this quickly (ie not manually entering every single site into OSE)? Or is there something in OSE or a tool I am overlooking? And whatever the best process is, can you give a brief overview? Thanks!! -Dan
Moz Pro | | evolvingSEO0