Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Search Engine blocked by robots.txt

Reporting & Analytics

1112

MirandaP last edited by

I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Hi Judith,

If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

I am useing the Bing SEO Toolkit, it worked fine for me,

try removeing robots for a test
1 Reply Last reply
Reply Quote 0
MirandaP @MirandaP last edited by

the SEOmoz crawler
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

What are you iusing to crawl?
1 Reply Last reply
Reply Quote 0
MirandaP @AlanMosley last edited by

I'm sure why it let you scroll and not me?
3 Replies Last reply
Reply Quote 0
AlanMosley last edited by
I just scroled it ok.

but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowed
```
User-agent: *
Disallow: */wo/
Disallow: */ajax/
```
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Search Console API script aggregating over date range

Hi I was recently reading this article https://moz.com/blog/how-to-get-search-console-data-api-python and found it really helpful to get started with the search console API. However, it makes note that it is designed to return data for each day individually in a date range. I don't know python very well and was wondering if anyone knew how to edit the script to aggregate over the given date range for each url in a list of urls.
Reporting & Analytics | | aranzato

0
Are these Search Console crawl errors a major concern to new client site?

We recently (4/1) went live with a new site for a client of ours. The client site was originally Point2 before they made the switch to a template site with Real Estate Webmasters. Now when I look into the Search Console I am getting the following Crawl Errors: 111 Server Errors (photos) 104 Soft 404s (blogs, archives, tags) 6,229 Not Found (listings) I have a few questions. The server errors I know not a lot about so I generally ignore. My main concerns are the 404s and not found. The 404s are mostly tags and blog archives which I wonder if I should leave alone or do 301s for each to /blog. For not found, these are all the previous listings from the IDX. My assumption is these will naturally fall away after some time, as the new ones have already indexed. But I wonder what I should be doing here and which will be affecting me. When we launched the new site there was a large spike in clicks ( 250% increase) which has now tapered off to an average of ~85 clicks versus ~160 at time of launch. Not sure if the Crawl Errors have any effect, I'm guessing not so much right now. I'd appreciate your insights Mozzers!
Reporting & Analytics | | localwork

0
GA Internal Site Search Correct Query Parameter?

Hi Guys, Recently added GA internal site search to a sub-folder: http://tinyurl.com/jhm9cyl Just want to confirm would the query parameter be: /search/ Or different because it's a sub-folder? Cheers.
Reporting & Analytics | | jayoliverwright

0
Google Search Console (new GWT) - Does a language specific sub folder need its own GSC profile

HI I've got a clients site set which targets 3 language/countries: English via the main site on the domain.com Turkish via a Turkish language site on a subfolder domain.com/tr/ And German via domain.de The devs have set up .com and .de in GSC and is reporting data in both However there's no data in the domain/com/tr GSC profile ! Is that because its on a subfolder so data pertaining to it is being reported in the main domain.com GSC account ? Or does something more need to be done to set up the Turkish subfolder in GSC ? If so what ? All Best Dan
Reporting & Analytics | | Dan-Lawrence

0
Google Search Console - Why is my average mobile position better than my average desktop position?

I'm wondering why my average mobile position is much better than my average desktop position. I'm wondering if Google is comparing the same queries for both mobile vs desktop or if they're only showing me the top ranked for each type of search. Is it example 1 or 2? Example 1: Desktop may have 5,000 ranking queries that average to 21.6
Mobile may have 1,500 ranking queries that average to 8.5 OR Example 2: example.com has 5,000 total ranking keywords, those queries' average ranking in a desktop search 21.6 and mobile search 8.5. I'm curious to know exactly what I'm seeing in Google's Search Console. https://08875344305734164866.googlegroups.com/attach/777ae98664ed418f/Mobile%20VS%20Desktop.png?part=0.1&view=1&vt=ANaJVrEHOjRLlPH43i00NnC8PxaG3ct7bsHum_TWnUoa7xVamCpRp8jrvRQJL-gz4n7Q0otqKcKxcAJA5z1VySs2naQU_Zy5tDps6bJhUSZsLRQq4uU-tJQ
Reporting & Analytics | | Pauly_Gigs

0
Why my WMT is showing more clicks than my Google Analytics Organic Search Traffic?

Hello Everybody, Can somebody help me figure out this puzzle: My WMT is showing 6000 clicks, while my Organic Traffic in Google Analytics is showing only 3500 daily ... how is that possible? Best Regards
Reporting & Analytics | | Muhammad_Jabali

0
Paid Search Referral

I have a brand new site with a paid search referral in my G&A, but we are not running any adwords or any paid marketing for it. The referral is "not set" so I do not know where it is coming from.
Reporting & Analytics | | KJ-Rodgers

0
Having Issue with Site Search in Analytics

Hi Mozzers, We launched a website in October 2012 and have added in the settings(Google analytics) of that profile "Do Track Site Search" since we have a search box on the website. The site search report worked for 10 days and it was over(from end of december till beginning of January 2013). Since then I have been trying to understand this issue. I have added all the query search terms possible, but still not showing any signs of life. At this point I am not sure what to do? Some Help would be appreciated! Search URL= subdomain.example.com**/search/node/**.... Thanks! z93cGUZ.png
Reporting & Analytics | | Ideas-Money-Art

0