Search Engine blocked by robots.txt
-
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
-
Hi Judith,
If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
-
I am useing the Bing SEO Toolkit, it worked fine for me,
try removeing robots for a test
-
the SEOmoz crawler
-
What are you iusing to crawl?
-
I'm sure why it let you scroll and not me?
-
I just scroled it ok.
but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowedUser-agent: * Disallow: */wo/ Disallow: */ajax/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are these Search Console crawl errors a major concern to new client site?
We recently (4/1) went live with a new site for a client of ours. The client site was originally Point2 before they made the switch to a template site with Real Estate Webmasters. Now when I look into the Search Console I am getting the following Crawl Errors: 111 Server Errors (photos) 104 Soft 404s (blogs, archives, tags) 6,229 Not Found (listings) I have a few questions. The server errors I know not a lot about so I generally ignore. My main concerns are the 404s and not found. The 404s are mostly tags and blog archives which I wonder if I should leave alone or do 301s for each to /blog. For not found, these are all the previous listings from the IDX. My assumption is these will naturally fall away after some time, as the new ones have already indexed. But I wonder what I should be doing here and which will be affecting me. When we launched the new site there was a large spike in clicks ( 250% increase) which has now tapered off to an average of ~85 clicks versus ~160 at time of launch. Not sure if the Crawl Errors have any effect, I'm guessing not so much right now. I'd appreciate your insights Mozzers!
Reporting & Analytics | | localwork0 -
Why my WMT is showing more clicks than my Google Analytics Organic Search Traffic?
Hello Everybody, Can somebody help me figure out this puzzle: My WMT is showing 6000 clicks, while my Organic Traffic in Google Analytics is showing only 3500 daily ... how is that possible? Best Regards
Reporting & Analytics | | Muhammad_Jabali0 -
Google Internal Search Tracking Kaput :-(
Buongiorno from cloudy & overcast Wetherby UK... On this site http://www.dartexcoatings.com/ I configured Google to tracdk internal search & heres how - http://i216.photobucket.com/albums/cc53/zymurgy_bucket/internal-search-jinx_zpscf86b49d.jpg But internal search data is not pulling through 😞 How can i fix this please.
Reporting & Analytics | | Nightwing
Thanks,
David0 -
What's the final word on Image Search tracking in Google Analytics?
Sorry if this has been answered but I can't seem to get a straight answer to my questions by searching around. How is traffic referred by Google Images counted in Google Analytics? I know it used to be referral traffic from google.com/imgres. A lot of things I have read say that it should all be under google/organic now, but my site still gets referral traffic from google.com/imgres, so that can't be. However I also get traffic as google/organic that I am pretty sure is from image search, because we don't rank for the keyword normally, but we do for image search. What's the deal? How is traffic from an embedded image in a regular result page counted? How can I segment my image search traffic better? It would be great to see image search traffic as it's own medium. I found a script here -- http://jrom.net/google-images-in-google-analytics -- that looks promising, has anyone used it or can recommend another way? I haven't used the GA API very much so I want to make sure the script is kosher and won't screw up my numbers.
Reporting & Analytics | | tact0 -
Ways to analyze a 1M rows dataset of search queries
Hi, I have this large dataset, about 1 million search queries with visits, bounce rate and a few other metrics. I'm trying to explore this data to find keyword "buckets" (such as include product name, location name, transactional objective, informational, etc.), as well as explore the density of certain keywords (keywords as in instances of a single word amongst all queries) My idea was to use Excel and a macro to split all queries in separate words (also clearing punctuation and uppercase/lowercase), then storing this word in a new worksheet, adding to another column the visit counts from the row where the word was extracted (as to give a sense of weight). Before adding the word to the new worksheet, the script will look if the word already existed, if so it would just add the current value of visits to the existing visit counts etc. In the end it will create sort of a "dictionary" of all the keywords in all search queries ranked by weight (= visits from search query including this keyword) This would help me get started I believe, because I can't segment and analyze 1M raw search queries... My issue is: this VBA has been running on my (fast) PC for the last 24hr and it doesn't seem to get to an end. Obviously excel+VBA is not the best way to do text mining and manipulation in such a large dataset (although it's just a 30mb file) What would you do if you had this dataset and would like to mine the text/semantic as I am doing? Any idea of tools? process? I'm considering dumping this data into a MySQL db and doing the processing through PHP (the only backend language I'm versed in), and getting the "summified" data stored into another table, which I'll then be able to export to a Excel for analysis. But I'm afraid that I'll be facing memory limit issues and such... In the meantime, I'm definitely interested into knowing what you guys would do if you had this data and wanted to simply start exploring its constituencies Thanks!
Reporting & Analytics | | briacg0 -
Blocking our IP's but wondering if Google still uses our search data?
The company owner here has our (company) website as his home page. I excluded our static IP’s on Google Analytics, but is that good enough to keep Google from using his search traffic as an indicator of anything negative. Does Google still take into account his activity, but simply block it from my reporting? Finally, does one person actually have that kind of influence as far as time on site, bounce rates, etc. Should I convince him to find a new home page?
Reporting & Analytics | | Ticket_King0 -
Get Stats on Search Box Usage, Using Google Analytics?
Let me preface this by first saying I am NOT using the Google Adsense Search feature. The question I am asking is pertaining to the standard search box that comes already coded in most .php blogs. In the Google Analytics, is there a way to tell how many people are searching using the search box? I've looked through the navigation and pathway options in analytics and I don't seem to see anything that definitively points to the number of people using the search box but, perhaps, I am overlooking it. Any insight?
Reporting & Analytics | | ChristineCadena0 -
Can you get local search numbers/traffic out of Google Analytics?
With Google's new local search I am more curious as to market penetration on keywords that are now localized to my different US cities. I understand that you can separate out Google traffic based on regional Google domains, but I am curious if there is an effective way to separate out searches and keywords based on a my local US Metros? If google cannot do this, any recommendations on products that can? Thanks.
Reporting & Analytics | | Thos0031