How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html) -
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do we find keyword opportunities - but not just against 3 competitors...
Hello everyone! How can I find out what might be a keyword opportunity against a competitor but make sure it's still a realistic opportunity in SERPS generally?* After all, we are not just competing with 2 or 3 competitors. We are competing against everyone! Perhaps there is a part of the keyword research process that helps? I can't help but think there's a wasted effort to just beat a couple of competitors unless you can rank well generally - especially if it's a competitive sector. (*I'm not in the position of making lots of pages as part of a long-tail keyword/ content strategy so I need to focus on just a few and make them count) Thanks in advance. Mp
Competitive Research | | mark_seomoz10 -
Best way to find the highest ranking pages for a message board?
What is the best tool or method to find the highest ranking posts and content on a message board or forum? What is the best tool or method for finding the highest ranking keywords for a message board or forum?
Competitive Research | | Hondaspeder0 -
Site Ranking for keywords that they haven't targeted in content
There is a site that I am constantly battling for the #1 spot for a particular keyword and I can't see that they are doing any link building, they are not using any anchor text for the keyword "at all" just their company name (not exact match) and their content doesn't even contain the keyword. I used Open site explorer to analyze their activity, but they are doing something I can't figure out from that data. Any other tools to use? I have higher quality links than them, post content nearly 5 times per week to my blog and their blog hasn't been updated in ages, I kill them in social media, there isn't one instance that they are better than my site and I only build quality driven links, no blog comment crap and get featured on lots of industry blogs for our work. I distribute my content very effectively, I just can't figure it out. They were no where about 5 months ago now they are tearing it up for lots of keywords in the industry top spots. I can build a few links and surpass them, but I have to do it every week or so and I think they are doing something fishy. I just want to figure out what they are doing and bury them. I don't want to post their url and mine here as I don't want them to see this post in search results.
Competitive Research | | photoseo10 -
Competitor's Ranking's Reports are Showing Blank Values for Rankings
When selecting competitors to see how they rank for specific terms in the ranking's report page here, it loads the page normally, but without any actual rankings. Where values should be, it lists all values as our competition not ranking whatsoever (for all competitors). I'm not sure if there is something we're doing wrong, or perhaps we just need to wait for a refresh, but if anybody could help out, that would be wonderful!
Competitive Research | | ACLens0 -
Good tools to compare site speed on different sites?
Google tells me that my site is slower than 81% of the internet.Now this is a very large news website, so I would like to compare this against other very large news websites. Are there any tools that will be able to tell me the speed of a list of websites? I am looking for something a little more detailed than 'this site loaded in 1 second'. Also, do any of you fellow SEOs know of any good tools to analyse what is causing your site to slow down? Kind regards
Competitive Research | | MirandaP0 -
How a site only with Frame can be at #1 google in a competitive keyword?
Guys, I need help. I have a company in Brasil and due to lack of budget, I'm doing the SEO for my self. And for two years Im reading everything that I can to do my best in my website. I worked very hard, but the #1 in the most important keyword for my company is a very bad website only with frames. What they are doing? Could anyone give help? Keyword: "acampamento"
Competitive Research | | Naghirniac
Google Brazil (www.google.com.br)
#1 website: www.acampar.net My website: www.acampamentoaguiasdaserra.com.br Thank you in advance Miguel0 -
Should I link to this site or is it too spammy?
Hi all, I've been asked if I want to exchange links with http://www.myteetimesonline.com/ and I'm tempted but fearful that its link profile looks spammy. What do you guys think? I'm basically starting from scratch with my own site http://www.golfgear4you.com and have a Page Rank 0 and Domain authority 32. The guys that want to link to me are Page Rank 3 but Domain authority 35. Their content would be of interest to my site's users and vice versa but my biggest concern is their link profile looks awful because the sites that link to them don't appear to be very relevant. I've done a lot of reading on this and have found the resources on this site to be a goldmine. The generally message is avoid easy links to spammy sites because they could hurt credibility. I just want to check I've actually learnt something and my instincts about this site are correct. i.e leave it alone!! Thanks, Andy
Competitive Research | | getzen560 -
Government Sites Cluttering Results?
Hi Guys, Have you ever come across government sites that are cluttering up SERP's that you're trying to rank for? For a new site that I'm working on one of the keyword terms is "driving test cancellations" and is in the UK. 3 of the top 4 results are government related sites which have verry little (if not nothing) to do with the keywords. Whilst these government sites are (very) loosley related to the keyword terms, and understandably have high pa/da, what would be the best way to try and rrank higher than these sites. I'm in the process of building links and social profiles - I'm really just wondering if there's something I'm missing that is an "easy fix" for jumping ahead of these sites - or getting them removed due to their lack of relevence. Gary...
Competitive Research | | perfectweb0