Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Search Engine blocked by robots.txt

Reporting & Analytics

1107

MirandaP last edited by

I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Hi Judith,

If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

I am useing the Bing SEO Toolkit, it worked fine for me,

try removeing robots for a test
1 Reply Last reply
Reply Quote 0
MirandaP @MirandaP last edited by

the SEOmoz crawler
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

What are you iusing to crawl?
1 Reply Last reply
Reply Quote 0
MirandaP @AlanMosley last edited by

I'm sure why it let you scroll and not me?
3 Replies Last reply
Reply Quote 0
AlanMosley last edited by
I just scroled it ok.

but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowed
```
User-agent: *
Disallow: */wo/
Disallow: */ajax/
```
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Search Console Crawl Errors/Not Found - Strange URLs

Hello, In Google Search Console under Crawl > Crawl Errors > Not found I have strange URLs like the following: https://www.domain.com//UbaOZ/
https://www.domain.com//UPhXZ/
https://www.domain.com//KaUpZ/WYdhZ/SnQZZ/MOcUZ/ There is no info in Linked From tab. Have you seen this type of error??
Does anyone know whats causing it??
How should it be fixed?? Thanks for reading and the help!
Reporting & Analytics | | chuck-layton

0
Google Search Console - Click Count Inconsistency

In Google's search console I see a discrepancy between click counts. At top I see this here and then beneath I see these kinds of numbers for click counts here. So the top click count says 252 and the bottom section appears to only shows less than 40. Probably a simple explanation here that I'm just not seeing. Thanks!
Reporting & Analytics | | a_toohill

0
Google Search Console - > Google Search Analytic gives figure for google organic or adwords or combine of both?

Hi All, In Google Search Console -> In Search Analytics. I can see Clicks, Impressions, CTR and Position. I want to know all these 4 - Clicks, Impressions, CTR and Position gives information related to google organic only? or combine or google organic and google adwords? Thanks!
Reporting & Analytics | | wright335

0
In Google Search Console: Total Clicks differ from the sum clicks on search queries

Hi.This may be a google technical question, but I've searched a lot and I couldn't find any certain information about that.The problem appears when you look at two stats in Search Console ( WebMasters Tools, some months ago), particulary in the Search Analysis.1- Total clicks2- Sum of clicks at every search query.I attached an image to make me clear.8IZsxs6.png
Reporting & Analytics | | NachoRetta

2
Google Search Console - Why is my average mobile position better than my average desktop position?

I'm wondering why my average mobile position is much better than my average desktop position. I'm wondering if Google is comparing the same queries for both mobile vs desktop or if they're only showing me the top ranked for each type of search. Is it example 1 or 2? Example 1: Desktop may have 5,000 ranking queries that average to 21.6
Mobile may have 1,500 ranking queries that average to 8.5 OR Example 2: example.com has 5,000 total ranking keywords, those queries' average ranking in a desktop search 21.6 and mobile search 8.5. I'm curious to know exactly what I'm seeing in Google's Search Console. https://08875344305734164866.googlegroups.com/attach/777ae98664ed418f/Mobile%20VS%20Desktop.png?part=0.1&view=1&vt=ANaJVrEHOjRLlPH43i00NnC8PxaG3ct7bsHum_TWnUoa7xVamCpRp8jrvRQJL-gz4n7Q0otqKcKxcAJA5z1VySs2naQU_Zy5tDps6bJhUSZsLRQq4uU-tJQ
Reporting & Analytics | | Pauly_Gigs

0
Google Analytics and Bounce Rates Query - Should I block access from foreign countries ?

Hi , When I look at my google analytics for my UK Website, I can see alot of visits come from outside the UK , i.e Brazil and USA. Both of which give me almost 100% bounce rates from people visiting from there. I am wondering, if google looks at bounce rates with regards to ranking factors and should I therefore block access to my site from visitors outside the UK ?... Would this help increase my rankings ? Given that we only serve uk customers, I cant see any benefit of allowing non uk customers the ability to see the site . what does people think ? thanks pete
Reporting & Analytics | | PeteC12

1
(Not provided) organic search results in Analytics

Hi, So from what I've read, (not provided) organic search results in Analytics is a result of the user being logged into Google. The problem I have is about 20% of the organic search results are (Not provided), so what is mainly left are branded search terms. I think I'm clutching at straws, but is there anything I can do to see these organic traffic sources?
Reporting & Analytics | | JuiceBoxOM

0
SERPS different based on location of search even with non-personalization

Hello Mozzers, Our agency's website, www.kenta.ro - ranked for a long time at #1 for "ann arbor seo" and similar keywords. For the past several (4-5) months we've been sitting around #5. My guess was that this was Google playing around with the results but I'm not sure why we have been at this position for such a long time. I have a vpn that I use for checking rankings overseas and if I connect to a server in Chicago, LA, Ontario, etc we show up as #1 - only when you search for "ann arbr seo" in our area do we get a lower ranking. All rank checking programs including seomoz show us at the #1 position because of this. What this means for us is that all of the traffic we target with this keyword sees the poor result, while the rest of the world sees the great result (should they search for it). How can we ensure that our target market finds us at #1 like the rest of the world does? Thank you in advance. weabi.png
Reporting & Analytics | | kentaro-256929

0