Search Engine blocked by robots.txt
-
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
-
Hi Judith,
If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
-
I am useing the Bing SEO Toolkit, it worked fine for me,
try removeing robots for a test
-
the SEOmoz crawler
-
What are you iusing to crawl?
-
I'm sure why it let you scroll and not me?
-
I just scroled it ok.
but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowedUser-agent: * Disallow: */wo/ Disallow: */ajax/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Keywords, Meta Keywords and Meta Descriptions; Keeping Webmaster Tools Current
What are Search Keywords, Meta Keywords and Meta Descriptions? What exactly is the difference between them and which one is more important? In regards to Webmaster Tools, if we delete a page or a product, it still shows up in Search Analytics. How can we update Webmaster Tools so as to keep it current with our website? Lastly, again in regards to Webmaster Tools, in Search Analytics. At the moment we put relevant queries into the Meta Description of low ranking pages, in order to raise the position of the page. Is this the right way to handle queries? Should we be putting the queries into the Meta Description or the Meta Keywords?
Reporting & Analytics | | CostumeD0 -
No Search Data in Google Search Console (Search Analytics)
Wondering if anyone has experienced 0 data in their Google Search Console (Search Analytics) and found possible solutions to retrieve data? We have a few clients who, prior to the update to Google Search Console, were getting data regularly in terms of the Search Queries report, but ever since the update to Google Search Console, they are no longer receiving data. As an FYI, both the www and non-www versions of the website are verified in Search Console and the XML Sitemaps and Robots.txt files are clean, tested and working fine. Any insights or experience of sites showing 0 data in Search Analytics? Any possible solutions would be greatly appreciated. Thanks!
Reporting & Analytics | | SEO5Team0 -
Google Web Master Tools show that my site has been crawled, but search results show old title tags, ect.
Does the index report reflect what will be displayed on Google results? Google seems to be indexing my site every Sunday...
Reporting & Analytics | | 928shopper0 -
Linked my adwords account to GA and vice versa and still paid search is getting recorded into organic traffic??
Hi Mozzers, I have linked properly my adwords account to GA and vice versa and somehow I can see 3/4 of this paid traffic recorded to organic search. The most confusing part is that I can see 1/4 of the paid traffic under the "paid" metric. At this point I don't know really what should I do? Thank you guys in advance!
Reporting & Analytics | | Ideas-Money-Art0 -
Block IP from analytics
How can I block an ip from analytics? basically so I do not see test orders from a certain ip in the sales performance section? Thanks for any help.
Reporting & Analytics | | tdigital0 -
Paid Search Referral
I have a brand new site with a paid search referral in my G&A, but we are not running any adwords or any paid marketing for it. The referral is "not set" so I do not know where it is coming from.
Reporting & Analytics | | KJ-Rodgers0 -
How to get a list of robots.txt file
This is my site. http://muslim-academy.com/ Its in wordpress.I just want to know is there any way I can get the list of blocked URL by Robots.txt In Google Webmaster its not showing up.Just giving the number of blocked URL's. Any plugin or Software to extract the list of blocked URL's.
Reporting & Analytics | | csfarnsworth0 -
Problem when searching for "link:www.mysite.com" vs "link: www.mysite.com"
Why does a search for "link:www.mysite.com" show no results, but when there is a space before www.mysite.com it shows results? The same happens for "link:www.mysite.com" (nothing shows up), but when I search for "link:www.mysite.com/index.php" it returns results. Is there a problem I am missing? Thanks so much!
Reporting & Analytics | | EmilyP0