How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
2 sites with low DA and PA and unoptimised <title>outranking everyone for competitive search term - How ?</title>
Im doing some keyword research for a new client, Ive found 2 anomalies for competitive search terms where sites with unoptimised <title>tags, poor backlinks and other stats, (compared to the other sites listed in the SERP) are out ranking everyone and claiming the #1 spot</p> <p> The 2 search terms are :</p> <ul> <li dir="ltr"> <p dir="ltr"><a href="https://www.google.co.uk/search?q=interior+designers+london&oq=interior+designers+london&aqs=chrome..69i57j69i65j69i60j69i65l2j69i60.2784j0j1&sourceid=chrome&ie=UTF-8">Interior designers London</a>- #1 new-id.co.uk - <a href="https://moz.com/researchtools/ose/links?site=new-id.co.uk&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a></p> </li> <li dir="ltr"> <p dir="ltr"><a href="https://www.google.co.uk/search?q=interior+designers+london&oq=interior+designers+london&aqs=chrome..69i57j69i65j69i60j69i65l2j69i60.2784j0j1&sourceid=chrome&ie=UTF-8#q=interior+designer+london">Interior designer London</a>- #1 nstudio.co.uk - <a href="https://moz.com/researchtools/ose/links?site=http%3A%2F%2Fnstudio.com%2F&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a></p> </li> </ul> <p> (interestingly they hardly rank at all for any modification of those search terms)</p> <p> The only thing i could think of is that they both had some really good links using that exact anchor text, but ive been checking OSE for the past month both in the backlinks and the just discovered but couldnt see anything like that.</p> <p>I conducted the SERP checking from a laptop with a broadband connection based in London, although im not very local to either of these locations im about an 8th of the cities distance away from one of those location and about half a cities distance away from the other. So i dont think its a local skew thats causing the SERP rankings.</p> <p>For both sites i can see they had old domains which now redirect to the them :</p> <ul> <li dir="ltr"> <p dir="ltr"><span data-mce-mark="1">New-id.co.uk = </span><span data-mce-mark="1"><a href="http://newidinteriors.com/">newidinteriors.com</a> - <a href="https://moz.com/researchtools/ose/links?site=http%3A%2F%2Fnewidinteriors.com&no_redirects=1&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a> - PA1, DA1</span></p> </li> <li dir="ltr"> <p dir="ltr"><span data-mce-mark="1">Nstudio.co.uk = intarya.com - <a href="https://moz.com/researchtools/ose/links?site=www.intarya.com&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a> - Da 32, PA43</span></p> </li> </ul> <p> Although the intarya domain was quite strong with good backlinks (but nothing above par comapred to the other sites the nstudio.co.uk site is now outranking, the anchor text was mostly natural with only some keyword anchor text from PA1 pages. </p> <p>The <a href="http://newidinteriors.com/">newidinteriors.com</a> didnt seem to have a backlink profile, both PA1 and DA1.</p> <p> Ive taken screenshots of the SERP pages below, but feel free to dig further</p> <p>Any idea what's going on here ?</p> <a download="HWkjmDf.png" class="imported-anchor-tag" href="http://i.imgur.com/HWkjmDf.png" target="_blank">HWkjmDf.png</a> <a download="A70jXse.png" class="imported-anchor-tag" href="http://i.imgur.com/A70jXse.png" target="_blank">A70jXse.png</a></title>
Competitive Research | | mike8780 -
Is my client's site penalized and if not, why are other lesser quality sites ranking with lower metrics than my client's?
Hello, I'm trying to explain to a client, as to why certain websites that appear in the top 50 listings for a keyword are there when they don't have very good backlink profiles, and for this keyword, seems you need a high PA above 20 or DA above 25, just to page for page 2 - 3. Running a full SERPs report, I have seen that backlinks to the page that's ranking seems to be as high of a factor as PA, followed by DA and overall backlink profile. Having a high score in one of these 4 areas seems to increase your ranking. However, going into page 3 and beyond, seems that things kinda take a weird turn. After comparing the client's site to those ranking on page 2 I could see that all of them had better PA and more links to the page ranking for that keyword, so page 2 I understand why we don't rank on yet. However, looking at page 3, I see that the client's site in question has better PA, DA, links to page and overall links to domain. Which leaves me to think we have a penalty that's keeping us ranking 50 or beyond. Since our metrics across the boards for what MOZ is showing as ranking factors. The more odd issue is that looking at the line graph, shows right at the time we implemented sitewide https our ranking for this one keyword dropped, while rankings for our other keywords increased. Am I to assume it's a keyword density penalty? What penalties are there with a outcome of ranking limits?
Competitive Research | | Deacyde0 -
Site Ranking for keywords that they haven't targeted in content
There is a site that I am constantly battling for the #1 spot for a particular keyword and I can't see that they are doing any link building, they are not using any anchor text for the keyword "at all" just their company name (not exact match) and their content doesn't even contain the keyword. I used Open site explorer to analyze their activity, but they are doing something I can't figure out from that data. Any other tools to use? I have higher quality links than them, post content nearly 5 times per week to my blog and their blog hasn't been updated in ages, I kill them in social media, there isn't one instance that they are better than my site and I only build quality driven links, no blog comment crap and get featured on lots of industry blogs for our work. I distribute my content very effectively, I just can't figure it out. They were no where about 5 months ago now they are tearing it up for lots of keywords in the industry top spots. I can build a few links and surpass them, but I have to do it every week or so and I think they are doing something fishy. I just want to figure out what they are doing and bury them. I don't want to post their url and mine here as I don't want them to see this post in search results.
Competitive Research | | photoseo10 -
SEO's done, 301s in place, old site STILL outranks new site. What to do?
Since Sep 2010 I have had a site up with minimal SEO optimization (www.chrisbrushmusic.com). Oct 29, 2012, I launched a new site on a new domain (www.chrisbrushdrums.com) with more content and tons of SEO work behind it. The content of the new site is significantly different from the old site, and I wish to keep the old site around. I have 301's in place for specific URLs on the old site that point to the new site. I have submitted xml sitemaps for the new site. As of now, the old site still outranks the new site (i.e. Google search for "nashville session drummer" and my old site is #9 - my new site is nowhere). What should I do? Thanks.
Competitive Research | | cbrush0 -
Where do we find a Keyword Discovery Tool in SEOmoz?
I've been looking for a way to compare keywords amongst my competition websites.
Competitive Research | | homesonthesound0 -
Can I write guest blogs on competitor's blogs?
I have been doing some guest blogging and its hard to find guest blogs opportunities. I have found some but they are direct competitors. Is it safe to write guest post? by the way the are not local competitors. Any suggestion please. thanks
Competitive Research | | conversiontactics0 -
Open Site Explorer is not showing dmT. Where do I go to get this metric.
Just trying to understand the video I watched on Link Building w/ Open Site Explorer. I'm not seeing the dmT metric in the live version. Has it been replaced by something else. Or is there someplace else I need to look for it?
Competitive Research | | AhmadS0 -
Identify keywords certain sites target
Hello, I've been asked to identify what keywords a group of blogs/communities might target so that we might gain some insight for our own SEM purposes. It's been several years since I used SEOmoz's tools, so they have changed a lot. I couldn't remember if you had such a tool, but it appears you do not--or at least not that I am finding. I was wondering if anyone could point me in the direction of such a tool where I can input a subdomain, and have a report stating which keywords are the ones the domain rates highly for. To be clear: I'm not looking for a traffic estimation or anything; simply a breakdown of what keywords a specific page or domain support most. P.S. I did a search on here for certain terms like "competitive keyword targets" and so forth, and saw no matching QA. I'm sorry if there is a duplicate! Thanks,
Competitive Research | | krisgosser
Kris0