Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Advanced popup for WP website
Hi guys, I'm looking for advanced popup for WP website.
Moz Pro | | EdmondHong87
Im looking something with advanced filters. For example: Filter the visitors by the operation system (For mobile).Will show to new visitors, but exclude to returns visitors.
Will exclude visitors that sign up to the website.
Filter the traffic...Popup will show only to visitors from FB, Google organic etc... Well, i believe that you understand what i need...
Anyone know something like this ? Thank you!0 -
Tools necessary for a Technical Audit of website with penalties and need remediation?
Tools necessary for a Technical Audit of website with penalties and needs remediation? I am being tested for a job interview to prove and/or disprove a website has issues. I am familiar with Moz tools but I'm not sure of the procedure for this request? I am not finding anything online. The client will be giving a website and I will be doing this audit. What tools would you use? What exactly should I be looking for? What are some obvious fixes? WHERE CAN I LEARN MORE?
Moz Pro | | Joseph.Lusso0 -
Social pages not lining up with my website/listing
Hi! Newbie here... 🙂 On my Moz report and I have been told that my social sites are not lining up with my webpage. Any tips on getting this done? Also on my Moz Analytic report, I've added my social sites but only facebook is coming up... Any tips would be appreciated!!! Thanks!
Moz Pro | | fullerton0 -
Multi language websites need proper reporting
Common guys! It can not be THAT difficult to add 3 characters to your reports..
Moz Pro | | nans
Instead of seeing 3 times Google BE, I would like to see the language code too. Example: Google BE-NL Google BE-FR Google BE-EN This is really not helpful for me or my clients (screenshot): https://docs.google.com/file/d/0B4BB1WWdu2ncYWZ4TTRqaWR1YVk/edit?usp=drivesdk There's more in the world than the US and English... Can you do this please? edit?usp=drivesdk0 -
How can i extract back link report in OSE of HTTPS domain?
Hello, Recently i was reviewing backlinks of one of the HTTPS domain and was socked to see the difference of backlinks in Majestic SEO and OSE tool. I tried a lot to find the option to explore the backlink of HTTPS domain but i think OSE is not providing that. Can anyone suggest the tool or method to find link report of HTTPS domains? Thank you, Sophie
Moz Pro | | sophiep0 -
Best Way to Include Social Media in Website?
Hi, how are you doing? I am new to MOZ, totally love it. I recently developed the social media for my page (www.aceromart.com) in Facebook, Google + and Twitter. (I am from Mexico, so the website is in Spanish) I am no expert in SEO whatsoever, but i like to engage my customers with great content both in my page and social media. My question is: **What the best way to include your social media links or icons on my page. Is there a program or a way to include the links. I want the people that visit ** **Should you include them in every page?, in a footer?, with icons or links. ** Thanks in advance for your advices, they are greatly appreciated. Best Regards, Jesus D
Moz Pro | | JesusD0 -
Why is blocking the SEOmoz crawler considered a red "error?"
Why is blocking the SEOmoz crawler considered a red "error?" Please see attached image... Y3Vay.png
Moz Pro | | vkernel0