How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do we find keyword opportunities - but not just against 3 competitors...
Hello everyone! How can I find out what might be a keyword opportunity against a competitor but make sure it's still a realistic opportunity in SERPS generally?* After all, we are not just competing with 2 or 3 competitors. We are competing against everyone! Perhaps there is a part of the keyword research process that helps? I can't help but think there's a wasted effort to just beat a couple of competitors unless you can rank well generally - especially if it's a competitive sector. (*I'm not in the position of making lots of pages as part of a long-tail keyword/ content strategy so I need to focus on just a few and make them count) Thanks in advance. Mp
Competitive Research | | mark_seomoz10 -
I can see competitors ranking for certain long-tail keywords but cannot find them on web pages. What am I missing?
Hi there. I'm pretty new to SEO and I've been doing a fair bit of training but there is one aspect I have yet to grasp. When I carry out keyword research, I get all these results and I understand the metrics. What I'm not getting is, when a competitor is ranking highly for say "where can I buy fresh turkeys", I assume that that phrase must appear somewhere on the page, but it doesn't. I realise I'm just not thinking about this in the right way. Can anyone offer clarification, please? Kind regards, Bruce
Competitive Research | | BruceBarbour0 -
What to do about one site dominating search results? (multiple pages ranking)?
Anybody have thoughts on dealing with search results where the same site gets listed multiple times? "weebly vs wix" is one example (same site #1-3, repetitive articles, not crazy high authority), but I see this now and then. I know Google likes variety, so it's weird for me to see results like this dominating search results. Thoughts? What gets these sites to take over the top rankings for a specific term? Any way to rise up in this situation, outside of the usual? Any tips on duplicating this kind of success?
Competitive Research | | davidwaring0 -
Why is my competitor's site ranking #1?
I'm about to work for a local business website that offers cleaning services and products. The keyword they want the most is ruled by a very odd site; My client's competitor's site has been around for 7 years. (Less than the average of it's competitors. Less than my client's) Has 1 backlink. Lower PA MR MT & DA than any other in the SERP. It's a 1 page site made with Flash. They do not have FB or Twitter accounts. So I thought maybe they were ranking so well because of their traffic. But neither my client, me or my coworkers have ever heard of this company. And yet, they are ranking #1. And the only thing I notice that might have helped is that the title of their page is the exact keyword and nothing more. Any ideas?
Competitive Research | | Eblan0 -
Why is this site ranking higher?!
I'm trying to figure out why www.lvimmigrationattorney.com would be ranking higher on several keywords than www.immigrationlawnv.com. It doesn't make sense to me because the latter site has a higher page authority and mozrank, as well as more followed links. The domain authority from the linking sites to each site are pretty close. I'm sure I'm missing something, but I can't put my finger on it. Any idea?
Competitive Research | | atstickel120 -
Why would a site with virtually no external links and low authority rank very high?
Hey there: I'm not sure anyone's going to be game to answer this but... I have designed, done on-page optimization for and done ongoing content marketing and linkbuilding for a local fitness site. The site is about nine months old and I am attempting to achieve rank for the keyword "personal trainers in philadelphia." We've made progress; having taken a new site from +100 to 19 in about four or five months worth of content creation and linkbuilding on a very modest budget. I'm not complaining. What has me confused though is that there are one or two sites that are ranking very high for this keyword string with considerably shoddier metrics than our site and I'm wondering how a stock WP theme site with very few inbound links can rank top five for a reasonably competitive keyword? Our site is http://vtrainersinc.com. The site I've noticed that ranks top five and seems to have an inferior SEOMoz profile is bodycrafters.net. Anybody got any ideas as to why these things seem to occur? I have this feeling this site has stumbled upon a magic SEO elixir which I'd like to get my hands on!
Competitive Research | | splatdude0 -
Wondering why my site ranks well for one page and not another.
Customers mainly find our site from searching very specific part numbers on google. My site is launch3telecom.com. Take for example these two parts:
Competitive Research | | launch3
MT500A-81015 MR050-81045 Search only those parts in google - we rank 3rd for one of them and don't rank at all for another. Is there any way to check backlinks pointing to those pages? We even show up before big sites like ebay and such on one of them. Can anyone help me understand this?0 -
How to check the Pr and Da of a thousand sites
Hi, I have a list of a thousand sites. I want to qualify the sites based on Pr and Da so that I can do my outreach. Pls advice how this can be done without manually putting each site in ose. Cheers, Vishal
Competitive Research | | vishalkhialani0