Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Whether or not to remove a link from a website with high spam score on Open Site Explorer
Hello Moz! I just subscribed for your Moz Pro program. Amazing stuff! On open site explorer, I found a number of links to my site from a page called with a very high page authority and high domain authority, but also a high spam score (8 or 9, one with a 10). I say multiple spam scores, because it's strange, there are what appears variations of the same url, and each one is considered a link. For instance, there's an abc.linkstomysite.com and xyz.linktomysite.com, and 123.linktomysite.com... there are about 15 of these (all with the spam scores mentioned above)! This must have been some old SEO work done I payed for back in the prehistoric SEO days. However, my fear is the following: Removing these links, and then losing some potentially strong link juice. I don't have many high DA or PA links to my site, and these are some major ones. The domain in question "linktomysite.com", when entered into OSE, only has a spam score of 4, and it has a domain authority of 45 and page authority of 37. My site has a spam score of 2 and no messages from google regarding a penalty, but an overall reduction in google traffic over the years (just keeps slowly dropping... as if a weight is pulling me down?) What do you think, should I leave, or remove? The linkstomysite page is just a LONG page full of links, with short descriptions, nothing of value, but with a an old domain age (relatively). Most important for me is keeping at least some ranking/visibility, while I personally work on building quality links and helpful content. thanks!
Moz Pro | | DavidC.0 -
I have a new website and would like to use the keywords my competitors are using and was wondering the best way to take a look using Moz Pro?
The website is www.fapwrap.com - it's a new adult wall decals website. [editor's note: may not be safe for some workplaces] The wall decals we have are obviously a bit more niche than the bigger competitors, but I want to get into the first page to see how it helps us. I'm slowly taking over some kws like "xbox wraps" and "ps4 wraps" but those have low search volume and I feel like "wall decals" will help - though it's highly competitive. Suggestions? I just kinda want to plug in my competition and pick from their list of existing kws..... Ha.
Moz Pro | | JohnnyRoq0 -
Link report that is broken down by C Block?
I've tried to do this in the advanced reports are of Moz, but to no avail. I just want to be able to see all the links (and anchor text would be nice too) for each CBlock.
Moz Pro | | DeluxeCorp0 -
How long before new links show up in OSE?
Hi - How long should it be before a brand new external link to our site shows up in OSE?
Moz Pro | | tcolling0 -
Strange Website Activity
Hello, I have been building websites for about 4 months now and finally had my first real success with a website. I found a niche that I was able to get on the first page with. This site was fine and then boom it dropped to #17 around the time of the recent Google changes. So I just thought it was that, I decided to add it as a campaign on here. This is when I noticed that the pages were not being crawled. So being new at all of this, I researched that. Of course, the main things were is the privacy and robots.txt. It is running on wordpress and I know not to set the blog to private, but I wasn't familiar with how to edit the robots.txt. I found a good plugin that easily allowed me to set the text to allow all bots. It seemed to be set to the normal wordpress settings before, and I never had a problem with any of my websites not being crawled. Anyway, once I just set it to allow all bots on ALL of my website, the pages started being crawled again. My traffic went back up and I was on the first page all day yesterday. Today, so far big drop off. So, I deleted the campaign and set up a new one. Sure enough no pages crawled yet. I made some security changes using Bulletproof Security and another plugin to see if that effects it. Nothing yet. I am just really confused as to what is going on, so if any of you have any ideas that would be great. It is a simple site, and I made some changes like theme when I was trying to figure out why the pages weren't being crawled. So it is not the most beautiful design right now. Also, I try my best to put up well-written useful content, so I don't think that is the issue for the rankings drops. I don't have many if any actual backlinks yet because of the newness of my site, could be the reason for it acting strange BUT none of that explains the pages not being crawled at the same time my site drops????? Sorry so long but had to explain it all! Thanks in advance to anyone who has anything to say about this situation! Edit: I should clarify, I put the the security plugins on yesterday while I was on the first page and have deleted them to see if that allows the pages to be crawled. Sorry if I wasn't clear.
Moz Pro | | iheartkelby0 -
OSE shows 118 FB shares but we show more, whats wrong here?
I ran www.lauriloewenberg.com/ in OSE and it shows 118 Facebook shares while we see more than that. Maybe we are doing something wrong or a setting isn't correct, I'm not sure. The site was created in WordPress. Here are 3 posts that add up to 224 shares: Post 1 114 shares Post 2 64 shares Post 3 46 shares I'm sure there are some people that are sharing multiple posts, Will that make a difference? Even if that was true the number still seems very low. Thanks 🙂
Moz Pro | | Mike-Dream0 -
Tons of Crappy links in new OSE (Open Site Explorer)
I am starting to miss the old OSE. I've found that for a lot of the pages on our site, the new OSE is showing WAY more links and most of them are garbage nonsense links from China, Russia, and the rest of the internet Wild West. For instance, in the old OSE, this page used to show 9 linking domains: http://www.uncommongoods.com/gifts/by-recipient/gifts-for-him It now shows 454 links. Some of the new links (about 5 of them) are legitimate. The other 400+ are garbage. Some are porn sites, most of them don't even open a web page, they just initiate some shady download. I've seen this for other sites as well (like Urban Outfitters) This is making it much harder for me to do backlink analysis on bc I have no clue how many "Normal" links they have. Is anyone else having this problem ? Any way to filter all this crap out ? See attached screenshot of the list of links I'm getting from OSE. NHXnn
Moz Pro | | znotes1 -
Does OSE Anchor Text Distribution CSV include internal links or external only?
Quite often when I'm using OSE and perform a CSV export, it looks like the total inbound link count and many phrases come from internal links for the domain in question. I haven't had the time to try and figure this out and don't know exactly how to go about figuring it out. So the initial question is does OSE's anchor text distribution CSV include internal links or are they external only? And alternately, how do I test for this? I deal with sites that have tens or hundreds of thousands of pages and hundreds of thousands or millions of links...
Moz Pro | | AlanBleiweiss0