Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Search Engine blocked by robots.txt

Reporting & Analytics

1107

MirandaP last edited by

I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Hi Judith,

If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

I am useing the Bing SEO Toolkit, it worked fine for me,

try removeing robots for a test
1 Reply Last reply
Reply Quote 0
MirandaP @MirandaP last edited by

the SEOmoz crawler
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

What are you iusing to crawl?
1 Reply Last reply
Reply Quote 0
MirandaP @AlanMosley last edited by

I'm sure why it let you scroll and not me?
3 Replies Last reply
Reply Quote 0
AlanMosley last edited by
I just scroled it ok.

but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowed
```
User-agent: *
Disallow: */wo/
Disallow: */ajax/
```
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Rogerbot directives in robots.txt

rogerbot reporting

I feel like I spend a lot of time setting false positives in my reports to ignore. Can I prevent Rogerbot from crawling pages I don't care about with robots.txt directives? For example., I have some page types with meta noindex and it reports these to me. Theoretically, I can block Rogerbot from these with a robots,txt directive and not have to deal with false positives.
Reporting & Analytics | | awilliams_kingston

0
Organic reports showing a URL that isn't in Search Ask Question

In the image I've attached you can see that I have pulled a source/medium > google organic report. I've also made "landing page" my secondary dimension. The first landing page that is showing up is /v3/?slug=fnl, that is this page (https://orders.freshnlean.com/v3/?slug=fnl). You can see that the page has 230 sessions from Sep 3 - 9 and 17 transactions during that same time frame. The only thing is, that landing page is nowhere to be found in the SERPs. So how is it showing up in this report as having received google organic visitors that converted if it's not even in search? 05OclDp
Reporting & Analytics | | tdastru

0
Search traffic hit after switching magazine to subdomain

Hi there, I'm fairly new at all of this and would appreciate any help with understanding why our website has taken a hit in traffic. We curate an online magazine, which was previously accessible through: forensicoutreach.com. It was receiving about 2,000+ unique visitors per day up until a week ago, when we changed a few things. However, the magazine doesn't reflect what our business does, so we created a product-focused web presence on forensicoutreach.com, and moved the magazine (which everyone loved) to library.forensicoutreach.com (DA 37, PA 1). We thought separating the properties was a good idea, but now I'm not so sure. Our traffic on library.forensicoutreach.com is 1,500 (so 500 less than usual!) and our main property has about 56 unique visits a day. It's pretty substantial. A few questions: 1. If we move the library to forensicoutreach.com/library, will that make any difference? 2. Where did we go wrong here and how can we fix it? Any help would be appreciated. Thanks.
Reporting & Analytics | | shivaniseos

1
Looking for a GA developer to block internal traffic and integrate with Snipcart JS API - paid

Hi There, We're looking for someone to help us setup: Blocking internal traffic using a custom dimension/query URL as described here: http://www.simoahava.com/analytics/block-internal-traffic-gtm/ Integrate Google Analytics ecommerce tracking with the Snipcart API as described here: https://snipcart.com/blog/integrating-snipcart-with-google-analytics-ecommerce-tracking Discuss Kiss Metrics and possibly integrate with Snipcart We're a small, 2 person startup so are ideally looking for a freelancer or small firm to help - Any recommendations or pointers in the right direction would be much appreciated. Cheers Ben
Reporting & Analytics | | cmscss

0
Are organic search visitors always seen as organic in origin, even if their return to the site is direct?

Many of our conversions occur in a customers second visit to the site. Often, a customer will arrive at our site, submit a finance application, leave, and return at a later date to checkout. We are interested in tracking how many of our checkouts come from customers who originally found our site through an organic search result. If a customer enters the site through organic search, leaves, and returns later through an email link or directly entering our URL, will G analytics show that customer as direct or organic origin? Cheers, Ben
Reporting & Analytics | | WSPL

0
Is there any way to find out how many people blocked our site in Google?

With it being a factor in Panda, I would like to know what the number is and whether it's high / low compared to other sites etc. I kind of doubt a lot of people would have blocked us, but I'd like to at least be aware of whether that's a problem for us.
Reporting & Analytics | | rango

0
Google Analytics Organic search queries aren't being updated, even though I'm still seeing results in all our typical results pages.

We pushed some new changes to the site and Google Analytics is no longer updating the Organic Search queries listing, even though traffic is consistent and and we're still landing results in all our typical keyword searches. Any ideas?
Reporting & Analytics | | unclekaos

0
Google Bot blocked

Google says that they are blocked from crawling my site. I have tried to get my web masters to look into this and they say the site apears fine. However, my SERP results have been falling and the errors are increasing. http://www.RealEstateMarketLeaders.com https://support.google.com/webmasters/bin/answer.py?hl=en&answer=2409441 I have sent an e mail into Google asking for help and I have been looking into every possible channel to find a solution for this problem. The site does have a 301 re direct pointing in from my old site http:///www.TriCitiesRealEstateAgent.com into my new site http://www.RealEstateMarketLeaders.com. However, I am not sure if that is the problem or if something else happend to make me see such a huge drop in SERP results.
Reporting & Analytics | | Brandon_Patton

0