Search Engine blocked by robots.txt
-
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
-
Hi Judith,
If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
-
I am useing the Bing SEO Toolkit, it worked fine for me,
try removeing robots for a test
-
the SEOmoz crawler
-
What are you iusing to crawl?
-
I'm sure why it let you scroll and not me?
-
I just scroled it ok.
but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowedUser-agent: * Disallow: */wo/ Disallow: */ajax/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Console Crawl Errors/Not Found - Strange URLs
Hello, In Google Search Console under Crawl > Crawl Errors > Not found I have strange URLs like the following: https://www.domain.com//UbaOZ/
Reporting & Analytics | | chuck-layton
https://www.domain.com//UPhXZ/
https://www.domain.com//KaUpZ/WYdhZ/SnQZZ/MOcUZ/ There is no info in Linked From tab. Have you seen this type of error??
Does anyone know whats causing it??
How should it be fixed?? Thanks for reading and the help!0 -
GA Internal Site Search Correct Query Parameter?
Hi Guys, Recently added GA internal site search to a sub-folder: http://tinyurl.com/jhm9cyl Just want to confirm would the query parameter be: /search/ Or different because it's a sub-folder? Cheers.
Reporting & Analytics | | jayoliverwright0 -
Exact Match in Google Search (Not Adwords)
I was going throught the list of keywords that have sent traffic to my site over the last 7 years and cam across one "A516 grade 70" that had hundreds of variants. Now in a lot of cases search volumes were different as were SERPS. We've tested a few variants with reworked pages (70% similar to original but optimised for variant keyword) and see good SERPS and traffic results. Theres obviously some diminishing returns here for us but the interesting question is when to these variants become an exact match and when not? In some cases the variants are unique because of the spacig, periods and hyphens used. there isn't a clear correlation with exact matc though. Insight appreciated. (Sorry for spelling errors. Form doesn't play nicely with iPad)
Reporting & Analytics | | Zippy-Bungle0 -
Massive decrease in search volumes from Keyword planner
The search volumes that I'm getting out of Keyword planner are approximately 10% (some as low as 1%) of the search volumes for the same keywords back in April. Is there something I missed in this time? I'm using the same account, same language and location settings. Anyone have an explanation for this? To be honest these new search volumes sound a lot more plausible to me - I'm just intrigued by what might have happened!
Reporting & Analytics | | a-champ0 -
"Local Search Volume" For what period (is it a month?)
When I look at the heading "Local Search Volume" or, "Global Search Volume... Are these figures for 1 month (that's what I'm assuming) Thanks
Reporting & Analytics | | bricktech0 -
Why are plus signs (+) suddenly showing up in Google Analytics organic search keywords reports?
Since June 13, 2013, the number of organic search queries containing a plus sign (+) has gone up over 1,000% compared to the previous period on my site in Google Analytics. These plus signs appear to be taking the place of spaces in these search queries (i.e. "word1+word2+word3"). This appears to be almost (or completely) Google organic traffic, not other search engines. Since I highly doubt searcher behavior would change so suddenly, I'm trying to figure out why Google is replacing spaces with plus signs. Is anyone else seeing this? Any ideas?
Reporting & Analytics | | RCF0 -
What is the difference between "Organic Traffic" and the "Non-Paid Search Traffic" default segment in Google Analytics?
These two filtering options ("organic traffic" in the left sidebar and "non-paid search traffic" in the advanced segments) give me slightly different numbers. Any idea why this would be the case?
Reporting & Analytics | | FPD_NYC1 -
Will Bing/Google's engine index a page that has only been on social media?
Will Bing's engine index and rank a page that has only been seen on social media and has no inbound links? Will Google's? Are inbound links absolutely required to get a page indexed and ranking and getting traffic? If unknown, how would you go about testing this?
Reporting & Analytics | | SarahGoliger0