Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Search Engine blocked by robots.txt

Reporting & Analytics

1110

MirandaP last edited by

I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Hi Judith,

If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

I am useing the Bing SEO Toolkit, it worked fine for me,

try removeing robots for a test
1 Reply Last reply
Reply Quote 0
MirandaP @MirandaP last edited by

the SEOmoz crawler
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

What are you iusing to crawl?
1 Reply Last reply
Reply Quote 0
MirandaP @AlanMosley last edited by

I'm sure why it let you scroll and not me?
3 Replies Last reply
Reply Quote 0
AlanMosley last edited by
I just scroled it ok.

but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowed
```
User-agent: *
Disallow: */wo/
Disallow: */ajax/
```
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

GA Internal Site Search Correct Query Parameter?

Hi Guys, Recently added GA internal site search to a sub-folder: http://tinyurl.com/jhm9cyl Just want to confirm would the query parameter be: /search/ Or different because it's a sub-folder? Cheers.
Reporting & Analytics | | jayoliverwright

0
In Google Search Console: Total Clicks differ from the sum clicks on search queries

Hi.This may be a google technical question, but I've searched a lot and I couldn't find any certain information about that.The problem appears when you look at two stats in Search Console ( WebMasters Tools, some months ago), particulary in the Search Analysis.1- Total clicks2- Sum of clicks at every search query.I attached an image to make me clear.8IZsxs6.png
Reporting & Analytics | | NachoRetta

2
Why do WMT's Search Analytics & GA's Real Time Keywords Differ?

I took a screen shot of the keywords from Google Analytics Real Time Keywords for Google / Organic. Then I waited a week and filtered Google Webmaster Tools' Search Analytics for queries with those keywords. None of them showed up. Why not? What I missing? Why do some keywords show up in Google Analytics RT for Google / Organic while most others are in Not Provided? I see WMT's Search Analytics says that it doesn't always show queries that "are made a very small number of times or those that contain personal or sensitive information." But most of my queries are like that. Anybody know any patterns / criteria that leads WMT to show some but not others? Thanks
Reporting & Analytics | | GilReich

0
What does WMT measure against for search query fluctuation?

Hi guys, I always thought that when checking out search queries on Webmaster Tools that it measured change per week. However, I noticed that my impressions and clicks were actually up but WMT was reporting that they were down over a week so it can't be that. Does anyone know? Attached screenshot for clarity. Thanks in advance! XkWANTo_
Reporting & Analytics | | Whittie

0
Should I block Ukraine, China and Russian Federation for better SEO?

My statistics show that a large percentage of hits to my site are from Ukraine, China and the Russian Federation. Now I know for a fact that most of my demographic is from the US with a bit of Western Europe thrown in. So obviously these are spam bots. Also I have many linked pages with the .ru extension that are spammy. Should I block Ukraine, China and the Russian Federation in my htaccess? I would think bounce rate is very high with spam bots so would that improve my SEO? Then of course there is the extremely annoying buttons-for-website.com and semalt-semalt.com!
Reporting & Analytics | | ijb

0
Best way to measure local search keyword rank

I have several clients that have a regional or local presence and want to track their rankings for various keywords, but only in the areas they are located in. What are the best ways to track keyword rank in specific locations or regions of the US?
Reporting & Analytics | | TheURLdr

0
Differences in organic search visits and non-paid keyword visits

Hi folks, I was just wondering at the disparity between the "Organic Search Visits" total in the Traffic Data tab and the total visits from "Non Paid Keywords Sending Search Visits". Once you add up all the traffic generated from the individual keywords (including the not provided numbers) shouldn't the total match the number of organic search results? Or is there something I'm missing? At the moment the total visits from non-paid keywords is about 2,000 short of the total organic search visits.
Reporting & Analytics | | BrettCollins

0
My own brand searches stopped appearing in google

Hi everyone, I've never asked a question here before, so go easy on me 🙂 I've noticed recently that my site is not showing up for my own brand searches anymore, and that my organic google results are way down. For example, it used to be if you would search for "grouvee kings quest 6" you would either see http://www.grouvee.com/news/kings-quest-6-retro-rerun-playthrough-part-1/63/ or http://www.grouvee.com/games/kings-quest-vi-heir-today-gone-tomorrow/69/ show up first in google. Now they're nowhere to be found. Here's another example. If you would search for "grouvee mass effect 3" you'd either see http://www.grouvee.com/games/mass-effect-3/75/ or http://www.grouvee.com/news/mass-effect-3-lets-take-the-earth-back/136/ show up first in google. Once again, nowhere to be found. I could go on with several other examples, but I don't really know what I've done. I 301 redirected 2 pages a few weeks ago because I added a /reviews/ structure to my site and needed to move a couple of pages that used to be under /news/ over to the new section, but those shouldn't have had anything to do with the pages I referenced above. I haven't done any link building really. I submit some of my articles occasionally to places like reddit or n4g, but that's about it. Anyone have any tips or things I should be looking at?
Reporting & Analytics | | petecorsaro

0