Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Potential spam websites with high DA linking back to us
Hey everybody, I'm going through all my sites and disavowing crap links. However, I'm having trouble distinguishing which high DA sites to disavow. What would you do? For example:
Moz Pro | | MEllsworth
https://moz.com/researchtools/ose/spam-analysis?site=busca.starmedia.com&target=domain&source=subdomain&page=1&sort=spam_score and https://moz.com/researchtools/ose/spam-analysis?site=cc879fe.activerain.com&target=domain&source=subdomain&page=1&sort=spam_score They both have tons of backlinks - both good and crap. The first has a DA of 72 and a Moz spam score of 4/17 and the second has a DA of 86 and a Moz spam score of 9/171 -
A tool to tell a websites estimated traffic
I am new to Moz (as a member), so I am not sure if Moz has a tool that I need. I don't want this post to be about self promotion, so I will keep it short. Our business helps increase conversions and sales for online businesses. Our ideal prospects belongs to some key categories of businesses like ecommerce, saas etc. However, I would like to know the estimated volume of traffic for a website before approaching them and introducing our service. So if there was a tool I could use to estimate the volume of visitors a specific website receives on average a day or month, it would be hugely beneficial.Obviously, these are prospective clients, so we do not have access to their system or their analytics. I just want to get an estimate. So for example, if I entered the domain abc.com into the system, I would hope it could tell me, that abc.com gets an average of 900 unique visitors a day. I don't need too much detail like geographic locations etc, but it would be a bonus having that additional information. I also don't mind paying for a tool that's quality. So it doesn't have to be free.
Moz Pro | | RyanShahed0 -
Adding new categories to my website and getting them ranked
Hi, I currently run a website and have added a couple of new categories lately. The website is Planes Trains Automobiles Co UK. For example I want to add "Planes for sale" as a new category. What is the best way to go about this? Will each aircraft added add content therefore ranking? I don't want to approach an SEO company yet as the service would be free for the time being? Thanks
Moz Pro | | PlanesTrainsAutos0 -
Can we add sites to the crawl queue for OSE?
Is it possible to request that Open Site Explorer crawls a new URL on its next run? This tool is the first place I go to when working on a new site, and when there is "No Data Available" this is a little frustrating. I fully appreciate that this lack of data is usually a signal that the website is either very new or of low quality, however that if often the reason that I am brought in and would very much like to benchmark and provide initial analysis using this tool. It would make sense that OSE crawls the sites that Moz members are working on wouldnt it? Scott.
Moz Pro | | eseyo0 -
OSE Linking Domains CSV Export Blank?
For some client domains, I've tried to do a CSV export from the OSE Linking Domains page, I get a CSV that contains the typical OSE header text, but no output records. Before I click the CSV link I always choose "show domains with links to All pages on the Root Domain". It doesn't happen with every domain I've tried this with, and when I've had the time/patience, some of the time if I try hours later, I get data. The problem though is I rarely have the luxury of running reports a second time during the audit cycle. Has anyone else experienced this? Is it a bug, or just an export fail due to system resources or something?
Moz Pro | | AlanBleiweiss0 -
How to track with SEOMOZ a website in several language
Hi, We have a customer with a website in EN, FR and ES. They used Joomfish, so each language is in a subdirectory : sitename/en sitename/fr sitename/es and they want their website to be well placed on the web for all that languages and countries: English, French, Spanish, German and Italian. It is a website for specific affiliation, that's why there is no barriers. What I need to do to use the best way SEOmoz. For the moment I created one campaign following Google US, google Germany and Google France. To go deeper, I would need to create different campaigns in my account? And also, your robot will be able to recognize the different subdirectories and languages? And to improve the SEO of this website, it wouldn't be better to have 3 domains name, one for each country? Thanks a lot in advance for your answer, Anne
Moz Pro | | ahernoux1 -
Historical Linkscape/OSE data available?
Hey Mozzerati, I'm curious if there's a way to expose and analyze historical data from each previous Linkscape/OpenSiteExplorer (OSE) update. Essentially, I'm looking for something similar to MajesticSEO's Backlink History tool that can show you both daily links over a short time as well as cumulative links over a long period. I think the utility of such a tool is to see which competitors have gained many links over a short period, signalling that they're engaging in some form of link-building/-buying/-baiting activity. Is there any way to yield this data (other than to record it manually after each Linkscape/OSE update) from SEOmoz PRO tools?
Moz Pro | | jcolman0 -
Incorrect domain authority result on SEO tool bar and OSE
The SEO tool bar is returning what I believe to be an incorrect domain authority of 71 and showing 24,356,141 lins from 153,051 domains. The OSE is also returning 71 as domain authority. Anyone know what could be doing this? Thanks. Jason
Moz Pro | | jayderby0