Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Search Engine blocked by robots.txt

Reporting & Analytics

1112

MirandaP last edited by

I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Hi Judith,

If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

I am useing the Bing SEO Toolkit, it worked fine for me,

try removeing robots for a test
1 Reply Last reply
Reply Quote 0
MirandaP @MirandaP last edited by

the SEOmoz crawler
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

What are you iusing to crawl?
1 Reply Last reply
Reply Quote 0
MirandaP @AlanMosley last edited by

I'm sure why it let you scroll and not me?
3 Replies Last reply
Reply Quote 0
AlanMosley last edited by
I just scroled it ok.

but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowed
```
User-agent: *
Disallow: */wo/
Disallow: */ajax/
```
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Moz reports way fewer backlinks than google search?

My site is only 11 months old but has steadily (if not slowly) been gaining backlinks. My question, is why Moz shows me at 303 backlinks and Google search console is showing at 1,237? I am more than a little suspicious that this could highlight the reason Moz shows such an unfavorable DA ranking for our site at a DA12. Other competitors that rank for similar keywords to mine are DA 42, DA 65, DA 73, etc. If the largest ranking factor is links, and they have mine reported incorrectly - is this the issue with DA as it relates to sites like mine? Any answer from someone who has experienced similar, or has a definitive answer is more than welcome to chime in! Thanks, Kevin
Reporting & Analytics | | kvncrll

0
Site property is verified for new version of search console, but same property is unverified in the old version

Hi all! This is a weird one that I've never encountered before. So basically, an admin granted me search console access as an "owner" to the site in search console, and everything worked fine. I inspected some URL's with the new tool and had access to everything. Then, when I realized I had to remove certain pages from the index it directed me to the old search console, as there's no tool for that yet in the new version. However, the old version doesn't even list the site under the "property" dropdown as either verified or unverified, and if I try to add it it makes me undergo the verification process, which fails (I also have analytics and GTM access, so verification shouldn't fail). Has anyone experienced something similar or have any ideas for a fix? Thanks so much for any help!
Reporting & Analytics | | TukTown

1
Google Webmaster indicates robots.text access error

Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent. A quick test shows I can access the file. This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8% Note the above message says 'over the last 24 hours', however the date is Jan-17 This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us. Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal. Thanks!
Reporting & Analytics | | jmueller0823

0
Finding an Explanation for a Massive Spike in Organic Search Traffic

Hi, I watch analytics on a website (for a friend's business) that is reasonably stagnant, which just experienced a massive spike in search traffic for no explainable reason. The organic search engine traffic had always been steady, but about two months ago, organic search traffic started rising slowly. I checked OSE & a few other tools, but couldn't find any massive source of gained links or other explanations - just the usual occasional blog post about the company. I got in touch with my friend to see if maybe they'd gone with a competitor or something else, but he also had no idea (and even if he wasn't being honest with me, we still should've been able to spot links or social metrics or something!) Then, yesterday, their organic search traffic just tripled. The crazy thing is, it's not from one keyword: Every search term, and (not provided) essentially went up 200-400%. And I have no freaking idea why. No large gain of links. No website editing. The only possible explanation I thought up is maybe one of their competitors got knocked out, but I doubt that would cause such a stratospheric rise. So figured I'd turn to y'all. Any ideas on what might be causing such wonderful results? Anyone have any good tips on figuring out why a website could all of a sudden be doing incredibly? Analytics chart is below for the curious, and thanks in advance for any ideas / tips! nQHrscw.png
Reporting & Analytics | | FlynnZaiger

0
Why are plus signs (+) suddenly showing up in Google Analytics organic search keywords reports?

Since June 13, 2013, the number of organic search queries containing a plus sign (+) has gone up over 1,000% compared to the previous period on my site in Google Analytics. These plus signs appear to be taking the place of spaces in these search queries (i.e. "word1+word2+word3"). This appears to be almost (or completely) Google organic traffic, not other search engines. Since I highly doubt searcher behavior would change so suddenly, I'm trying to figure out why Google is replacing spaces with plus signs. Is anyone else seeing this? Any ideas?
Reporting & Analytics | | RCF

0
A switch from Search to Direct traffic from Feb 2012

I have read about the Google algorithm updates galore in 2012. I would therefore be inclined to suspect a sudden sharp drop in Search traffic this year to be caused by such updates. However, I am seeing what appears to be a direct switch from 'Search' to 'Direct' traffic as of about February 16th this year. There are no specific brand-building exercises or general advertising campaigns happening around this time for the site in question. It seems very UN-explainable? I wondered if there are any thoughts from the various lurking gurus? I have attached GA screenshot.. DjTu1.png
Reporting & Analytics | | MikiP

0
(Not provided) organic search results in Analytics

Hi, So from what I've read, (not provided) organic search results in Analytics is a result of the user being logged into Google. The problem I have is about 20% of the organic search results are (Not provided), so what is mainly left are branded search terms. I think I'm clutching at straws, but is there anything I can do to see these organic traffic sources?
Reporting & Analytics | | JuiceBoxOM

0
Track image searches on google analytics

How can i track image searches on google analytics? To filter the refferers by image.google or /imgres does not work in new google analytics (and brings an impossibly small number on old GA). also, this would not cover local google image refferers, that don't come from image.google.com. anyone knows how to find image search sources? i haven't found anything helpful in google's documentation or forums. thanks!
Reporting & Analytics | | zeepartner

0