Search Engine blocked by robots.txt
-
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
-
Hi Judith,
If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
-
I am useing the Bing SEO Toolkit, it worked fine for me,
try removeing robots for a test
-
the SEOmoz crawler
-
What are you iusing to crawl?
-
I'm sure why it let you scroll and not me?
-
I just scroled it ok.
but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowedUser-agent: * Disallow: */wo/ Disallow: */ajax/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Search Console data include GMB traffic? Branded CTR is 37.8%- Good or Bad?
Hey all, Per Search Console our branded keyword CTR is 37.8%. But when that keyword is searched our GMB listing shows up on top of the #1 result. For the same 90 day period GMB shows another 35% visits to our GMB (based on the number of impressions and visits to our GMB page) listing when the same keyword is searched. My question is this. Does Search console data include clicks that came from our GMB listing or not? My thinking is like this: If GMB traffic is not calculated in search console then it means that 72.8% of people looking for our brand will end up on our site on way or another 9organic #1 result plus GMB listing visits) We are also doing PPC for this very keyword that has gets almost 20% of the remaining traffic. So after adding all up we are loosing about 8% of our branded traffic to people who are doing adwords. When you search our brand you normally see 2, 3 competitor's adwords ads. Does anyone know how this works exactly? And if you don't mind sharing your branded keyword CTR's, so I can compare to ours please. I would love to compare to a site that actually has a GMB listing ranking for the same keyword Thanks in advance, Davit
Reporting & Analytics | | Davit19850 -
What is the "UPDATE" indicate in the Google Search Console Query Reports?
We recently noticed an update note in the Google Search Console that happened on April 27th. Does this denote an algorithm update? Any feedback or article would really be helpful. Thanks! gfQ8FG9.jpg
Reporting & Analytics | | RosemaryB0 -
How to get multiple pages to appear under main url in search - photo attached
How do you get a site to have an organized site map under the main url when it is searched as in the example photo? SIte-map.png
Reporting & Analytics | | marketingmediamanagement0 -
800,000 pages blocked by robots...
We made some mods to our robots.txt file. Added in many php and html pages that should not have been indexed. Well, not sure what happened or if there was some type of dynamic conflict with our CMS and one of these pages, but in a few weeks we checked webmaster tools and to our great surprise and dismay, the number of blocked pages we had by robots.txt was up to about 800,000 pages out of the 900,000 or so we have indexed. 1. So, first question is, has anyone experienced this before? I removed the files from robots.txt and the number of blocked files has still been climbing. Changed the robots.txt file on the 27th. It is the 29th and the new robots.txt file has been downloaded, but the blocked pages count has been rising in spite of it. 2. I understand that even if a page is blocked by robots.txt, it still shows up in the index, but does anyone know how the blocked page affects the ranking? i.e. while it might still show up even though it has been blocked will google show it at a lower rank because it was blocked by robots.txt? Our current robots.txt just says: User-agent: *
Reporting & Analytics | | TheCraig
Disallow: Sitemap: oursitemap Any thoughts? Thanks! Craig0 -
How to get a list of robots.txt file
This is my site. http://muslim-academy.com/ Its in wordpress.I just want to know is there any way I can get the list of blocked URL by Robots.txt In Google Webmaster its not showing up.Just giving the number of blocked URL's. Any plugin or Software to extract the list of blocked URL's.
Reporting & Analytics | | csfarnsworth0 -
(Not provided) organic search results in Analytics
Hi, So from what I've read, (not provided) organic search results in Analytics is a result of the user being logged into Google. The problem I have is about 20% of the organic search results are (Not provided), so what is mainly left are branded search terms. I think I'm clutching at straws, but is there anything I can do to see these organic traffic sources?
Reporting & Analytics | | JuiceBoxOM0 -
Google Analytics internal Site Search - Destination pages dispaly Search results
Hi, Im having a bit of an issue with Google Analytics internal site search, I am able to currently track the search terms through my website internal search but when I click onto destination pages I just get the search result page. When clicking destination pages I would expect to get the pages on which the user ended up after the results page, instead I just get the results page which is pretty much useless ?submitsearchXXXXXX hope you can help, look forward to your response. Thanks,
Reporting & Analytics | | Tug-Agency1 -
Will Bing/Google's engine index a page that has only been on social media?
Will Bing's engine index and rank a page that has only been seen on social media and has no inbound links? Will Google's? Are inbound links absolutely required to get a page indexed and ranking and getting traffic? If unknown, how would you go about testing this?
Reporting & Analytics | | SarahGoliger0