Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Search Engine blocked by robots.txt

Reporting & Analytics

1112

MirandaP last edited by

I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Hi Judith,

If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

I am useing the Bing SEO Toolkit, it worked fine for me,

try removeing robots for a test
1 Reply Last reply
Reply Quote 0
MirandaP @MirandaP last edited by

the SEOmoz crawler
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

What are you iusing to crawl?
1 Reply Last reply
Reply Quote 0
MirandaP @AlanMosley last edited by

I'm sure why it let you scroll and not me?
3 Replies Last reply
Reply Quote 0
AlanMosley last edited by
I just scroled it ok.

but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowed
```
User-agent: *
Disallow: */wo/
Disallow: */ajax/
```
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Google Search Console All Verfication Methods Failed

Hi Guys, We are trying to get google webmaster tools verified for this site: https://www.nookie.com.au/ We have implemented the recommended: HTML tag from Search Console which is: Which is installed see: http://prntscr.com/g66cle However, verification still failed. We then tried Google Analytics (failed), alternative methods (HTML file upload failed) Google Tag Manager. All failed. I have no idea why this has happened, any ideas? Cheers.
Reporting & Analytics | | jaynamarino

0
The difference between organic searches in Acquisition and organic searches in Default Channel Grouping

Hi guys, We have a question. In Google Analytics, there are 2 types of identifying organic searches: through Acquisition and through Default Channel Grouping. On our website, we have some differences between the number of organic sessions. Which one do you think is more relevant? Which one do you use? Many thanks in advance!
Reporting & Analytics | | RIDGID_Europe

0
Did Analytics change the way to handle Google images searches on Dec 12?

Dear all, One of the sites I'm monitoring receives a lot of traffic from image searches or images that appear in universal search results. On Dec 12th, 2015, the bounce rate for these sessions went from around 30% the day before to around 87%. See screen shot below. Did anybody notice similar bounces in the bounce rate? Did Google change something in the way that image search is handled? Looking forward to your ideas! large?v=mpbl-1&px=999
Reporting & Analytics | | AABMarketing_Frank

0
Help Blocking Crawlers. Huge Spike in "Direct Visits" with 96% Bounce Rate & Low Pages/Visit.

Hello, I'm hoping one of you search geniuses can help me. We have a successful client who started seeing a HUGE spike in direct visits as reported by Google Analytics. This traffic now represents approximately 70% of all website traffic. These "direct visits" have a bounce rate of 96%+ and only 1-2 pages/visit. This is skewing our analytics in a big way and rendering them pretty much useless. I suspect this is some sort of crawler activity but we have no access to the server log files to verify this or identify the culprit. The client's site is on a GoDaddy Managed WordPress hosting account. The way I see it, there are a couple of possibilities.
1.) Our client's competitors are scraping the site on a regular basis to stay on top of site modifications, keyword emphasis, etc. It seems like whenever we make meaningful changes to the site, one of their competitors does a knock-off a few days later. Hmmm. 2.) Our client's competitors have this crawler hitting the site thousands of times a day to raise bounce rates and decrease the average time on site, which could like have an negative impact on SEO. Correct me if I'm wrong but I don't believe Google is going to reward sites with 90% bounce rates, 1-2 pages/visit and an 18 second average time on site. The bottom line is that we need to identify these bogus "direct visits" and find a way to block them. I've seen several WordPress plugins that claim to help with this but I certainly don't want to block valid crawlers, especially Google, from accessing the site. If someone out there could please weigh in on this and help us resolve the issue, I'd really appreciate it. Heck, I'll even name my third-born after you. Thanks for your help. Eric
Reporting & Analytics | | EricFish

0
Need help understanding what happened to our organic search.

I help run an ecommerce business that mainly runs on Google organic search (yes, I realize this is not a good place to be). Last April, we saw around 25-30% of our organic search cut. I’m pretty sure it was due to the mobile update but we had some changes in the past month or two before that. I’m looking for someone to review my analytics account and see what happened. Possibly this is called an audit? Also, we’re looking to ask some questions about long term strategy as we are thinking about a redesign and switching to a new platform. Maybe more content? Maybe more social?
Reporting & Analytics | | kirbyf

0
Hi - I have a beginner question about organic search results dropping to zero

Really confused about a site I'm getting going on SEO with... I'm new to SEO, but I've found that the organic search results for princessdesign.co.uk have dropped to zero dramatically. I'm concerned that I've missed something and hope that somebody out there might be able to help? Any input greatly appreciated 🙂
Reporting & Analytics | | Gokart

0
Track image searches on google analytics

How can i track image searches on google analytics? To filter the refferers by image.google or /imgres does not work in new google analytics (and brings an impossibly small number on old GA). also, this would not cover local google image refferers, that don't come from image.google.com. anyone knows how to find image search sources? i haven't found anything helpful in google's documentation or forums. thanks!
Reporting & Analytics | | zeepartner

0
What impact will Google's 10/18/2011 announcement of 'Making Search More Secure' have on the ability to track specific keyword queries via Analytics?

The full announcement is here: http://googleblog.blogspot.com/2011/10/making-search-more-secure.html My concern is that the ability for Google Analytics to parse information on specific keyword queries will be diminished. The article hints that Google Webmaster Tools will be exempt from the problem, and I've never relied on Webmaster tools as a go-to for tying specific keyword queries to Goal Tracking (form submissions and sales). The community's thoughts on this one are appreciated. 🙂
Reporting & Analytics | | MKR_Agency

0