Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My website was at the top of Google search for some years... suddenly I almost can't reach first page! Moz ranks my website better than the competitors... what might be going one? Could anybody help me out? Thanks!
Hello Guys! My website was at the top of Google search for some years... suddenly I almost can't reach first page! Moz ranks my website better than the competitors... what might be going one? Could anybody help me out? Moz rank us grade A... the competitors B or C .. I think we have better back links than they do... Would you need any kind of data or report to help me here? Thanks!
Moz Pro | | wesleyms0 -
New Website & DA Score
We are preparing to build a new website for a branch of our company. According to Neil Patel (whom I highly respect) our DA should come in around 30 (categorizing us as a startup, see chart below). So I ask: Why do our current sites DA Score below 30? (20 & 29 respectively) Our sites were redesigned in 2013 and are responsive, pass the Google Mobile Friendly test and I've improved load speed by over 25% by compressing images. I have used Moz as guidance for 10 months on both 7 + yo sites, HP.com is 14 yo, our pages score in the high 90's and I have disavowed bad backlinks for 6 months (now only every other month). I thought we would be in or close to the 50 range by 2017 after a solid year of improvements, but I don't see that happening. Our scores go up a point or two, then back down, we have floundered 2 points either way for 10 months. Most important question: What do I do now to raise our DA? Lastly, my plan is to take all the lessons learned from the past years and apply it to the new site, measure the success and then redesign the other two sites in accordance to the new model. But that may be a year away. KJr www.heritageprinting.com www.heritageprintingcharlotte.com | DA | Rating |
Moz Pro | | KevnJr
| 1-10 | Poor – Your site is young and weak. You have a lot of growing to do. |
| 11-20 | Decent. Your site isn’t stellar, but you’re doing better. It would be good to grow. |
| 21-30 | Fair. Your site shows signs of SEO, but there are many things you can and should do to improve. |
| 31-40 | Competitive. A lot of startups find themselves in this DA range. It’s not bad, and you’re beginning to get close to the sweet spot. |
| 41-50 | Good. Now, you’re getting somewhere. This is a nice place to be, and many good e-commerce sites find themselves squarely in this category. |
| 51-60 | Strong. As you swing out of the lower half of the scale, you’re beginning to get much healthier. This is a good place to be. |
| 61-70 | Excellent. A DA at this level represents a great site with a lot of recognition, a lot of link backs, and a considerable authority in its niche. Many .edus are in this space. |
| 71-80 | Outstanding. You’re dominating in the SERPs and owning your niche. Quick Sprout is a 73. |
| 81-90 | Very outstanding. You’re in the upper echelons of authority. You can consider yourself to have arrived. |
| 91-100 | Rare. These sites are household names — Wikipedia, Facebook, New York Times, etc. Your site will probably never attain this level. Only a miniscule fraction of a percentage of sites on the Internet ever get this high. | https://www.quicksprout.com/2014/04/09/how-to-score-your-websites-seo-in-10-minutes-or-less/0 -
Block Moz (or any other robot) from crawling pages with specific URLs
Hello! Moz reports that my site has around 380 duplicate page content. Most of them come from dynamic generated URLs that have some specific parameters. I have sorted this out for Google in webmaster tools (the new Google Search Console) by blocking the pages with these parameters. However, Moz is still reporting the same amount of duplicate content pages and, to stop it, I know I must use robots.txt. The trick is that, I don't want to block every page, but just the pages with specific parameters. I want to do this because among these 380 pages there are some other pages with no parameters (or different parameters) that I need to take care of. Basically, I need to clean this list to be able to use the feature properly in the future. I have read through Moz forums and found a few topics related to this, but there is no clear answer on how to block only pages with specific URLs. Therefore, I have done my research and come up with these lines for robots.txt: User-agent: dotbot
Moz Pro | | Blacktie
Disallow: /*numberOfStars=0 User-agent: rogerbot
Disallow: /*numberOfStars=0 My questions: 1. Are the above lines correct and would block Moz (dotbot and rogerbot) from crawling only pages that have numberOfStars=0 parameter in their URLs, leaving other pages intact? 2. Do I need to have an empty line between the two groups? (I mean between "Disallow: /*numberOfStars=0" and "User-agent: rogerbot")? (or does it even matter?) I think this would help many people as there is no clear answer on how to block crawling only pages with specific URLs. Moreover, this should be valid for any robot out there. Thank you for your help!0 -
How to monetize a video website ?
My website is popularvid.com I want to monetize it just like youtube but i don't know how to do that. Please give me suggestions to monetize my video website. How to make my website rank page one google? Please give me brilliant ideas about new features to be added that will make my website beautiful. I'm a newbie here. Thank you
Moz Pro | | idamardyana0 -
OSE Problems
I'm experiencing severe delays trying to export OSE data into CSV over the past couple of days. Relatively small reports I ran 2 days ago are still pending! Has anyone else noticed this? Also my campaign rankings have not updated since 24th April - anyone else? Logged with Moz Helpdesk but no reply as yet...
Moz Pro | | PeterAlexLeigh2 -
OSE Link Report Question.
Hi, when i export a link report as a csv file via the export tool on OSE, there is a column in the exported table called "Numder of Link". Please explain me, what does this column says exactly. Its located between "Domain Authorithy" and "Number of Domains Linking to Domain". Thanx, Seb.
Moz Pro | | TheHecksler0