Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Search Engine blocked by robots.txt

Reporting & Analytics

1110

MirandaP last edited by

I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Hi Judith,

If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

I am useing the Bing SEO Toolkit, it worked fine for me,

try removeing robots for a test
1 Reply Last reply
Reply Quote 0
MirandaP @MirandaP last edited by

the SEOmoz crawler
1 Reply Last reply
Reply Quote 0
AlanMosley @MirandaP last edited by

What are you iusing to crawl?
1 Reply Last reply
Reply Quote 0
MirandaP @AlanMosley last edited by

I'm sure why it let you scroll and not me?
3 Replies Last reply
Reply Quote 0
AlanMosley last edited by
I just scroled it ok.

but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowed
```
User-agent: *
Disallow: */wo/
Disallow: */ajax/
```
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Can Anyone help me with Google search console soft 404 error?

Hello everyone, I just build one site on WordPress and submitted it to the Google search console along with the sitemap. Some URLs got indexed but one of my URL is showing error in Search console. My post https://hotpass.site/create-subdomain-godaddy/ is showing soft error 404 webpage not available in Google search console. I have rechecked the site and seo settings. Nothing is there. I have even checked it with LIVE URL test. But still Search console is not accepting this URL. Can anyone help me with this? Help will be appriciated.
Reporting & Analytics | | Pauline21

0
How to track google auto search suggestion click?

Hello Guys, In google.co.uk when I search SEL and google gives me option of different different sites and when I click on any one site then that click tracking I need. I have attached the screenshot to understand easily. Is it possible to track such things or possible via server logs etc? TV99h
Reporting & Analytics | | micey123

1
Lots of missing main keywords from Search Console

Hello Moz Community, A while back I noticed in Google Search Console that the volume of impressions and clicks dropped off a cliff. I also noticed that this was for primary head keywords that generally had a decent volume of impression share. The natural initial reaction was 'oh, I must have lost those rankings' but upon checking I realized these rankings are still in existence. Admittedly most are page 2 or 3, but still within the confines of being captured in GSC. Is there a logical reason why these keywords have just gone from search console? An example keyword would be something like 'online football management game' and the website is https://www.worldelitesoccer.com There is 0 queries in my search console data that includes the word 'football'. Thanks, Ben
Reporting & Analytics | | melaniedsg

0
Did Analytics change the way to handle Google images searches on Dec 12?

Dear all, One of the sites I'm monitoring receives a lot of traffic from image searches or images that appear in universal search results. On Dec 12th, 2015, the bounce rate for these sessions went from around 30% the day before to around 87%. See screen shot below. Did anybody notice similar bounces in the bounce rate? Did Google change something in the way that image search is handled? Looking forward to your ideas! large?v=mpbl-1&px=999
Reporting & Analytics | | AABMarketing_Frank

0
Changes to cookies, August 24 or 25? (organic search not being cookied?)

Hi folks. We've seen a precipitous change in the referral data that we're able to gather from cookies. Specifically: On August 24 or 25, traffic to our site that was cookied as coming from organic search dropped by ~25%, and traffic coming in with no referrer data at all (i.e., it appears to be "direct") rose by roughly the same amount. As far as we can see, we haven't changed anything in our systems that would have caused this (we're not just mis-reading the cookie info), so I'm looking for external reasons. Has anyone else seen this? Or have any ideas why it would happen?
Reporting & Analytics | | RobM416

0
How to safely exclude search result pages from Google's index?

Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blah I wanted to block everything of that sort, but how do I do it without blocking /index.php ? Thanks in advance and have a great day everyone!
Reporting & Analytics | | llamb

0
What is the difference between "Organic Traffic" and the "Non-Paid Search Traffic" default segment in Google Analytics?

These two filtering options ("organic traffic" in the left sidebar and "non-paid search traffic" in the advanced segments) give me slightly different numbers. Any idea why this would be the case?
Reporting & Analytics | | FPD_NYC

1
Google Analytics Organic search queries aren't being updated, even though I'm still seeing results in all our typical results pages.

We pushed some new changes to the site and Google Analytics is no longer updating the Organic Search queries listing, even though traffic is consistent and and we're still landing results in all our typical keyword searches. Any ideas?
Reporting & Analytics | | unclekaos

0