Search Engine blocked by robots.txt
-
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
-
Hi Judith,
If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
-
I am useing the Bing SEO Toolkit, it worked fine for me,
try removeing robots for a test
-
the SEOmoz crawler
-
What are you iusing to crawl?
-
I'm sure why it let you scroll and not me?
-
I just scroled it ok.
but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowedUser-agent: * Disallow: */wo/ Disallow: */ajax/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can't see backlinks in Search Console
Hello, We have 7 sites and have noticed that for one site, we don't see any backlink info in Search Console even if it's been linked for over 8 months. Other tools show thousands of backlinks, but Search Console is still pending. I also see very little info in regards to Performance even if we are getting thousands of hits a day. Could this be a sign of a bigger problem? In summary, the site is up and running, getting hits, getting backlinks, but Search Console still looks like it's a new site with no activity several months after being installed.
Reporting & Analytics | | CJolicoeur0 -
Search Console API script aggregating over date range
Hi I was recently reading this article https://moz.com/blog/how-to-get-search-console-data-api-python and found it really helpful to get started with the search console API. However, it makes note that it is designed to return data for each day individually in a date range. I don't know python very well and was wondering if anyone knew how to edit the script to aggregate over the given date range for each url in a list of urls.
Reporting & Analytics | | aranzato0 -
Search Console "Not found" errors
The Google Search Console is showing recent "Not found" errors for pages that haven't been on my site for years. The pages are not in my sitemap and there don't appear to be any internal links to the pages. Is this normal? Should I be concerned? (Google Search Console > Crawl > Crawl Errors > URL Errors)
Reporting & Analytics | | nkolson1 -
How to Diagnose "Crawled - Currently Not Indexed" in Google Search Console
The new Google Search Console gives a ton of information about which pages were excluded and why, but one that I'm struggling with is "crawled - currently not indexed". I have some clients that have fallen into this pit and I've identified one reason why it's occurring on some of them - they have multiple websites covering the same information (local businesses) - but others I'm completely flummoxed. Does anyone have any experience figuring this one out?
Reporting & Analytics | | brettmandoes2 -
Delays in Search Console Data
We are seeing big delays in search console data this month. In the past we have seen delays of a few days, but I have never noticed it being this long before. At the moment we have no data since 23rd February. Have I just never noticed this before or is this particularly long? Is anyone else seeing the same thing?
Reporting & Analytics | | Wagada0 -
Any advice for better results for local search in the UK ?
Hi there, I have tried the tools here, and they are BRILLIANT but maybe to much for me. I just wondered if anyone in the UK had time to give me some advice. A lot of my customers are small businesses (mainly trades, like plumbers, builders etc etc), and I am trying to find best practices for getting them more quality traffic amidst a lot of competition. One customer I am working with currently has 50+ competitiors for his small plumbing business that are within 5 miles of him and have an effective web prescence. In those circumstances (take it as read that I have killed myself creating quality content that I am proud of for this client), does anyone have any killer advice for me that could make my sites more effective ? If anyone does have the time I am more than willing to PM the URL I am working with at the moment. Thanks for your time..
Reporting & Analytics | | nicklemonpromotions0 -
Google encryption of search results
Hello Is there any definitive information on whether the recent changes to the way Google encrypts search results for people logged into Google affects the traffic present via the Google Adwords Keyword tool? Plus, how does SEOmoz ensure/minimise the affect of personalisation/localisation etc. on the rankings data provided via the pro tool. Thanks in advance for your assistance. Kind regards Neil
Reporting & Analytics | | mccormackmorrison0 -
Get Stats on Search Box Usage, Using Google Analytics?
Let me preface this by first saying I am NOT using the Google Adsense Search feature. The question I am asking is pertaining to the standard search box that comes already coded in most .php blogs. In the Google Analytics, is there a way to tell how many people are searching using the search box? I've looked through the navigation and pathway options in analytics and I don't seem to see anything that definitively points to the number of people using the search box but, perhaps, I am overlooking it. Any insight?
Reporting & Analytics | | ChristineCadena0