Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Search Engine blocked by robots.txt

Reporting & Analytics

1112

MirandaP last edited by

I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Hi Judith,

If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

I am useing the Bing SEO Toolkit, it worked fine for me,

try removeing robots for a test
1 Reply Last reply
Reply Quote 0
MirandaP @MirandaP last edited by

the SEOmoz crawler
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

What are you iusing to crawl?
1 Reply Last reply
Reply Quote 0
MirandaP @AlanMosley last edited by

I'm sure why it let you scroll and not me?
3 Replies Last reply
Reply Quote 0
AlanMosley last edited by
I just scroled it ok.

but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowed
```
User-agent: *
Disallow: */wo/
Disallow: */ajax/
```
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Are these Search Console crawl errors a major concern to new client site?

We recently (4/1) went live with a new site for a client of ours. The client site was originally Point2 before they made the switch to a template site with Real Estate Webmasters. Now when I look into the Search Console I am getting the following Crawl Errors: 111 Server Errors (photos) 104 Soft 404s (blogs, archives, tags) 6,229 Not Found (listings) I have a few questions. The server errors I know not a lot about so I generally ignore. My main concerns are the 404s and not found. The 404s are mostly tags and blog archives which I wonder if I should leave alone or do 301s for each to /blog. For not found, these are all the previous listings from the IDX. My assumption is these will naturally fall away after some time, as the new ones have already indexed. But I wonder what I should be doing here and which will be affecting me. When we launched the new site there was a large spike in clicks ( 250% increase) which has now tapered off to an average of ~85 clicks versus ~160 at time of launch. Not sure if the Crawl Errors have any effect, I'm guessing not so much right now. I'd appreciate your insights Mozzers!
Reporting & Analytics | | localwork

0
Should I block Ukraine, China and Russian Federation for better SEO?

My statistics show that a large percentage of hits to my site are from Ukraine, China and the Russian Federation. Now I know for a fact that most of my demographic is from the US with a bit of Western Europe thrown in. So obviously these are spam bots. Also I have many linked pages with the .ru extension that are spammy. Should I block Ukraine, China and the Russian Federation in my htaccess? I would think bounce rate is very high with spam bots so would that improve my SEO? Then of course there is the extremely annoying buttons-for-website.com and semalt-semalt.com!
Reporting & Analytics | | ijb

0
How many users completely block Google Analytics cookies ?

Hello everyone! In your experience, how many of your visitors' browsers completely block cookies including those of Google Analytics ?
Reporting & Analytics | | Masoko-T

0
What's the final word on Image Search tracking in Google Analytics?

Sorry if this has been answered but I can't seem to get a straight answer to my questions by searching around. How is traffic referred by Google Images counted in Google Analytics? I know it used to be referral traffic from google.com/imgres. A lot of things I have read say that it should all be under google/organic now, but my site still gets referral traffic from google.com/imgres, so that can't be. However I also get traffic as google/organic that I am pretty sure is from image search, because we don't rank for the keyword normally, but we do for image search. What's the deal? How is traffic from an embedded image in a regular result page counted? How can I segment my image search traffic better? It would be great to see image search traffic as it's own medium. I found a script here -- http://jrom.net/google-images-in-google-analytics -- that looks promising, has anyone used it or can recommend another way? I haven't used the GA API very much so I want to make sure the script is kosher and won't screw up my numbers.
Reporting & Analytics | | tact

0
Google Analytics internal Site Search - Destination pages dispaly Search results

Hi, Im having a bit of an issue with Google Analytics internal site search, I am able to currently track the search terms through my website internal search but when I click onto destination pages I just get the search result page. When clicking destination pages I would expect to get the pages on which the user ended up after the results page, instead I just get the results page which is pretty much useless ?submitsearchXXXXXX hope you can help, look forward to your response. Thanks,
Reporting & Analytics | | Tug-Agency

1
How can I track search engine optimization data in Google analytics?

My website is linked to a Google Analytics web property. But, I am not able to track search engine optimization data in Google Analytics. So, How can I get it done?
Reporting & Analytics | | CommercePundit

0
Google Secure Search Announcement: How do you think this will play out?

So, I read this post on the Google Blog and then this commentary about it. Wondering what some of you veterans think about how this will impact results. I always remain logged out for queries and use several different software tools to check when I run an analysis. Since social influences are becoming a bigger factor, do you think this is going to create some issues in data? What are your thoughts?
Reporting & Analytics | | TheARKlady

0
Google Analytics - April Search Report Data Gone For All Clients - Anyone Else Seeing This?

Hey everyone, In the last hour I was doing some SEO referral reportingin Google Analytics and discovered that my April data was completely gone (flat lined to zero). Specifically, if i select a date range for any point in april and select search from the advanced segments area I am seeing this issue. This is happening for ALL of my active clients... anyone else seeing this?
Reporting & Analytics | | dpeddle

0