How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html) -
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do we find keyword opportunities - but not just against 3 competitors...
Hello everyone! How can I find out what might be a keyword opportunity against a competitor but make sure it's still a realistic opportunity in SERPS generally?* After all, we are not just competing with 2 or 3 competitors. We are competing against everyone! Perhaps there is a part of the keyword research process that helps? I can't help but think there's a wasted effort to just beat a couple of competitors unless you can rank well generally - especially if it's a competitive sector. (*I'm not in the position of making lots of pages as part of a long-tail keyword/ content strategy so I need to focus on just a few and make them count) Thanks in advance. Mp
Competitive Research | | mark_seomoz10 -
Cant find any keyword for my site
I searched for keyword rank for shoreloop.com but couldn't find any. What could be the problem?
Competitive Research | | killerseo20200 -
What's the best SEO way to benefit from your competitor's shutting down?
Hi ! Our main competitor is shutting down (website will be offline) at the end of the month and we are negotiating with them to buy their domain name: the idea would be to take advantage of their good rankings in the SERPs to redirect traffic to our pages (we're planning to crawl their site or get their sitemap and redirect th category / product pages to ours). The question is: for how long this strategy will be useful: days / weeks / months? (= for how long their pages will continue to appear in the SERPs from the day we enable the 301 redirections to our site?) Thanks in advance for your help! And if you have better suggestions, we're up to hear them of course.. 😉 Cheers
Competitive Research | | Kuantokusta0 -
Free tools to find country of origin of backlinks/urls
Hey are there any free tools out there which can allow me to insert a large list of urls, and it determines the country of origin of the domain. I know the paid version of majestic does, but i was wondering if theres any free tools? Cheers, Chris
Competitive Research | | monster990 -
How to track competitor who uses another website on the url?
Hi everyone, The thing is pretty simple: I have a competitor who uses a different format of url, very similar to this: www.example.com - but he started to use a thing like this: example.mtv.com The problem is that every analysis that I make tell me something about mtv.com instead of the site I want. The url is redirected. Sorry about my english, but I think it is very clear. I want to know how to track something like this, because even in competitor analysis I'm getting info about the big site, and not the one I need. Thank you very much.
Competitive Research | | bluehelmet0 -
Sites With Duplicate Content Ranking Way Higher
The site I am writing about is easendtorontohomes.com I don't get it. I see that some of my competitors have more inbound links and have been around longer. Their SEOmoz trust etc is also higher. But I don't get how many could possibly be ranking higher since they're sites are all template based site via Web Tech Design and the content is duplicated from one to the other. The other weird thing is that they do everything wrong - they stuff their keywords tag, some of them use the same keywords/meta data on every single page, some of them don't even have meta tags, some of them don't even have a blog with ANY original content. Apart from listings EastEndTorontoHomes.com has TONS of original, well-researched, and keyword rich content. I just don't get it. Is it possible that google has some sort of relationship with that company? Or am I totally missing something? Some of these sites include: tinasmith.ca, mikeclarke.com, teamkassen.com
Competitive Research | | annasus0 -
Is There Video Instruction on How To Use SEOMoz and Open Site Explorer?
Hi I have just signed up to use this software and would like to know how to use it properly. Is there a video tutorial on how to get the most out of the tools? I want to get the competition's backlinks but for some reason it's not openiing the page. I also want to know how the backlinks are assessed to find the real quality ones that I can link to. How are the backlinks assesed and what standards are used to asses backlink quality? Thank you Sunil.
Competitive Research | | sunilmuse0 -
Government Sites Cluttering Results?
Hi Guys, Have you ever come across government sites that are cluttering up SERP's that you're trying to rank for? For a new site that I'm working on one of the keyword terms is "driving test cancellations" and is in the UK. 3 of the top 4 results are government related sites which have verry little (if not nothing) to do with the keywords. Whilst these government sites are (very) loosley related to the keyword terms, and understandably have high pa/da, what would be the best way to try and rrank higher than these sites. I'm in the process of building links and social profiles - I'm really just wondering if there's something I'm missing that is an "easy fix" for jumping ahead of these sites - or getting them removed due to their lack of relevence. Gary...
Competitive Research | | perfectweb0