How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My (properly optimised) webpage outscores page#1 ranked competitors on page/domain authority ... but I'm only on page#2\. Huh?
I'm puzzled. I've optimised a particular page for a particular search term, and the SEOMoz tool gives me an A for on-page optimisation. So no problem there. I can understand why my webpage/site is being outranked by pages from (for example) the Guardian and Oxford University, but there are several sites that Google is ranking on page #1 though their page and domain scores are well below ours. Specifically: my page/domain authority scores are 46/52, compared with 22/46 for the competitor that Google is ranking #5 - yet we only rank a lowly #12. And it's not as though the particular page in question isn't an obvious and appropriate part of our site. We work with new writers and the page in question offers a selection of creative writing courses. It's not like we're a writing-related site that suddenly has a page advertising fake rolexes. It's not a timing issue either, as most of our links have been in place for a couple of years at least. So I'm puzzled. And concerned. This page of ours was a reliable revenue generator for us and it's dying out there on the page#2 wilderness. If anyone can help, I'd be massively grateful. I don't know if this is helpful, but the page in question is http://www.writersworkshop.co.uk/Creative-Writing-Courses.html and the search term is ... well, heck, you take a wild guess. We're a British firm, so the only search engine that really matters to us is google.co.uk
Competitive Research | | harrybingham0 -
Why is this site at no.1?
In the last 2 months, a previously unranked/unheard of website jumped straight to no.1 position for a particular keyword when searched from USA and is even ranking over many other big brands or stronger DA/PA sites. I did notice some very strange JavaScript popup links that are heavily keyword linked from the first mention of the keyword on the page content. What are they trying to do here? Otherwise they don't seem to have much PA/DA or seo going on. Why are they doing so well?!
Competitive Research | | emerald0 -
Can I write guest blogs on competitor's blogs?
I have been doing some guest blogging and its hard to find guest blogs opportunities. I have found some but they are direct competitors. Is it safe to write guest post? by the way the are not local competitors. Any suggestion please. thanks
Competitive Research | | conversiontactics0 -
What is the best tool to monitor your competitor's backlinks?
I am looking for a tool that not only accurately tells me how many links are being built by a competitor, I want to know the specific URLs for specific periods. If I can filter/search by time periods, i.e which links were added for the month of August, 2012, that would be great. Thanks for the help.
Competitive Research | | inhouseseo0 -
Is open site explorer running on your side
I am facing trouble to open the website http://www.opensiteexplorer.org/ please check on your side and let me know
Competitive Research | | 247Email0 -
IP Address Of Site
Hi there, really new to seo and loving being able to research everything on seo moz. I have a question about the ip address of my site: I use a company that sells templated real estate websites (I'm a realtor) and I'm noticing all of their websites show as having the same ip address. There are probably 100's of sites on their server targeting the same keywords as I am, would this affect my ranking at all? I'm just considering switching to a Wordpress site with a unique IP address, so I'm curious to find out how the seomoz members would answer this question. Cheers, Adrian
Competitive Research | | adrianaldous0 -
Keyword Ranks reported by SEOMOZ don't match actual searches
SEO Moz reported that various keywords I was tracking were not in the top 50 results on Yahoo or Bing, yet when I did a Yahoo or Bing search for those keywords (from multiple computers and not logged into any Yahoo or MSN/Bing account) I found the keywords easily. What's the scoop?
Competitive Research | | Cybernautic0 -
Site redirect
Hi, Our site www.suncamp.nl has, for language reasons, 2 redirects So : www.suncamp.nl is redirected with a 301 to www.suncamp.nl/nl/nl and is than redirected 301 to http://www.suncamp.nl/nl/nl/home/uc19-l1-n804/ My question is; is this bad for our SEO? Recently we've been doing a lot of linkbuilding and SEO copywriting, but in comparison with our competitors were lagging behind. So I'm looking for other bottlenecks. Kind regards, Dennis Overbeek My email: dennis@acsi.eu ACSI
Competitive Research | | SEO_ACSI0