Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Search Engine blocked by robots.txt

Reporting & Analytics

1110

MirandaP last edited by

I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Hi Judith,

If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

I am useing the Bing SEO Toolkit, it worked fine for me,

try removeing robots for a test
1 Reply Last reply
Reply Quote 0
MirandaP @MirandaP last edited by

the SEOmoz crawler
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

What are you iusing to crawl?
1 Reply Last reply
Reply Quote 0
MirandaP @AlanMosley last edited by

I'm sure why it let you scroll and not me?
3 Replies Last reply
Reply Quote 0
AlanMosley last edited by
I just scroled it ok.

but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowed
```
User-agent: *
Disallow: */wo/
Disallow: */ajax/
```
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

How to interpret search "clicks" in GA?

I am trying to make some sense of the data in the "Landing Pages" report under "Search Engine Optimization" in GA versus the data under the "Landing Pages" report in "Behavior | Site Content". For example, the SEO report says my page http://www.asiantraveltips.com/blog/bangkok-skytrain-bts-mrt-lines/ received 22,000 search impressions in the past 30 days and 900 "clicks" (12.42%). What are these "clicks" when the Content "Landing Pages" report says the same page only got 382 "sessions" in the same period??? What are these "clicks" if not clicks on the search results link that should be reflected in landing page sessions on the corresponding page?
Reporting & Analytics | | Gavin.Atkinson

0
Improving Search Click through Rate

We are having a problem on our website with click through rates. We are getting between 100-150k impressions through search but we are only getting between 500-1000 clicks to the site. What strategies have you used in the past to help improve your click through rates? Thanks!
Reporting & Analytics | | pdangermond

2
"Local Search Volume" For what period (is it a month?)

When I look at the heading "Local Search Volume" or, "Global Search Volume... Are these figures for 1 month (that's what I'm assuming) Thanks
Reporting & Analytics | | bricktech

0
Can you help me figure out what happened to my website search results in Google?

On or about the 24th of April I noticed an abrupt decrease in traffic to my website:
http://www.getyourphotosoncanvas.com Sorry this might be long but I’m trying to be as thorough as possible. I thought that I had been hacked, a virus, maybe penalized by Google I don’t know what ? I submitted a reconsideration request to Google and they responded with the following: Reconsideration request for http://www.getyourphotosoncanvas.com/: No manual spam actions found
May 10, 2012
Dear site owner or webmaster of http://www.getyourphotosoncanvas.com/,
We received a request from a site owner to reconsider http://www.getyourphotosoncanvas.com/ for compliance with Google's Webmaster Guidelines. - - - - - -
We reviewed your site and found no manual actions by the webspam team that might affect your site's ranking in Google. There's no need to file a reconsideration request for your site, because any ranking issues you may be experiencing are not related to a manual action taken by the webspam team.
Google Search Quality Team I have ran all kinds of web crawl tests, Google webmaster, talked with SEO “Experts” and still can not figure out what is happening. I decided to use a couple of SEOmoz tools to try to help me explain what is happening. I figured that if I could take a very specific and unique KeyPhrase and run it on a specific page that I might be able to better explain what is happening. Basically, We appear to be no longer searchable by key words or phrases on google?
Here is an example:
Key Phrase: Free Services to Help Improve Your Photos on Canvas
Website: http://www.getyourphotosoncanvas.com/free-photo-canvas-retouching
Attached are some screen shots of the actual search results on Bing, Yahoo and Google along with the ranking tool results from SEOmoz and the on page grade for the key phrase.
Anybody got any Ideas? I am hurting; the internet and Google search is about 40% of by business. http://www.getyourphotosoncanvas.com/wp-content/uploads/2012/05/Bing-Free-Services.jpg http://www.getyourphotosoncanvas.com/wp-content/uploads/2012/05/Yahoo-Free-Services.jpg http://www.getyourphotosoncanvas.com/wp-content/uploads/2012/05/Google-Free-Services.jpg http://www.getyourphotosoncanvas.com/wp-content/uploads/2012/05/SEOmoz-Ranking.jpg http://www.getyourphotosoncanvas.com/wp-content/uploads/2012/05/SEOmoz-Report-Card.jpg [" target="_blank">iframe>](<iframe class=) Bing-Free-Services.jpg Yahoo-Free-Services.jpg Google-Free-Services.jpg SEOmoz-Ranking.jpg SEOmoz-Report-Card.jpg
Reporting & Analytics | | rdominey

0
My own brand searches stopped appearing in google

Hi everyone, I've never asked a question here before, so go easy on me 🙂 I've noticed recently that my site is not showing up for my own brand searches anymore, and that my organic google results are way down. For example, it used to be if you would search for "grouvee kings quest 6" you would either see http://www.grouvee.com/news/kings-quest-6-retro-rerun-playthrough-part-1/63/ or http://www.grouvee.com/games/kings-quest-vi-heir-today-gone-tomorrow/69/ show up first in google. Now they're nowhere to be found. Here's another example. If you would search for "grouvee mass effect 3" you'd either see http://www.grouvee.com/games/mass-effect-3/75/ or http://www.grouvee.com/news/mass-effect-3-lets-take-the-earth-back/136/ show up first in google. Once again, nowhere to be found. I could go on with several other examples, but I don't really know what I've done. I 301 redirected 2 pages a few weeks ago because I added a /reviews/ structure to my site and needed to move a couple of pages that used to be under /news/ over to the new section, but those shouldn't have had anything to do with the pages I referenced above. I haven't done any link building really. I submit some of my articles occasionally to places like reddit or n4g, but that's about it. Anyone have any tips or things I should be looking at?
Reporting & Analytics | | petecorsaro

0
Nofollow page is being reported as a landing page for organic search in Google Analytics

One of my client's websites includes a series of pages for an enrollment process. All of these pages are blocked by robots.txt. In Google Analytics these pages are showing data as landing pages for organic search traffic, and have been for quite some time. There was recently a surge of organic search traffic landing on one of these pages, coming from multiple search engines. The pages appear to be blocked and I'm not finding any of them in the search results for the keywords that are being reported in GA or by searching for the url. Does anyone have any insights into why this might be happening?
Reporting & Analytics | | rgibson100

0
Google encryption of search results

Hello Is there any definitive information on whether the recent changes to the way Google encrypts search results for people logged into Google affects the traffic present via the Google Adwords Keyword tool? Plus, how does SEOmoz ensure/minimise the affect of personalisation/localisation etc. on the rankings data provided via the pro tool. Thanks in advance for your assistance. Kind regards Neil
Reporting & Analytics | | mccormackmorrison

0
Standard % of nonbranded organic search traffic?

I am working on developing goals for 2012. I will be doing several SEO projects, including one on a website for study abroad programs. Currently, nonbranded organic search is 15.49% of our visits and in 2010, it was 15.02%. I am wondering if anyone has any advice or knows of a benchmark percentage that could help me set an attainable goal. Thank you!!
Reporting & Analytics | | CIEEwebTeam

0