How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best way to find the highest ranking pages for a message board?
What is the best tool or method to find the highest ranking posts and content on a message board or forum? What is the best tool or method for finding the highest ranking keywords for a message board or forum?
Competitive Research | | Hondaspeder0 -
What's the value of Exact Match Keyword Domains vs. Company Name Domains?
Hey Mozers, I was in a discussion this morning about the value of Exact Match Keyword domains vs. a company name domain and wanted to get a little more clarification. Let's say we are doing a site for a company called Favored Dental, and they have had the domain favoredental.com for quite a while and have their authority built up in it. Is it better to have favored-dental.com or favoreddental.co or keep its current form? The reasoning behind the alternate domains would be they have the exact match keyterm, in this case lets say "Favored Dental" is the keyterm we were going after. To my knowledge EMDs aren't as relevant as they'd use to be as Google would rather branding of companies instead of keyterm domains? Is this correct, or do EMDs of keywords you're going after hold higher authority? Thanks for the clarification!
Competitive Research | | MonsterWeb280 -
What is the best tool to monitor your competitor's backlinks?
I am looking for a tool that not only accurately tells me how many links are being built by a competitor, I want to know the specific URLs for specific periods. If I can filter/search by time periods, i.e which links were added for the month of August, 2012, that would be great. Thanks for the help.
Competitive Research | | inhouseseo0 -
Local Real Estate Site vs Larger National Sites
Hello all, I am a local realtor that created my website with an idx feed. I am on page one but at the bottom. Question is how can a local real estate website rank better in the SERPS than the larger national brands? Any tips out there from those who have worked with real estate websites and seo? I'm trying to out rank the larger sites for my main keywords. Thanks all.
Competitive Research | | bronxpad0 -
How did this little site rank #2?
Keyword: "adoption agency" Site: http://www.opendooradoption.org These guys are ranked #2 for this very competitive term in all US regions. They're beating Bethany (and my company) who are orders of magnitude larger in both company size and link profile. Their domain authority is 40, and they have less than 200 links pointing to them with "adoption agency" in the anchor according to OSE. Their on-site is pretty much nonexistent. Does anyone have any theories as to how they accomplished this?
Competitive Research | | AdoptionHelp0 -
What are the competition's Google Places pages optimised for?
I'm doing some work on a client's Googe Places page, and wondered if there's any way to see what a completitors Places page is currently optimised or categorised for? Basically, we're trying to rank for 'Bathrooms Edinburgh' and almost all of the page 1 SERP's are (unsurprisingly) full of Places results, with #1 Organic slot right down at the bottom of the page. In short - we NEED to get our Places page kicked into shape, and pronto! So, is there any way to find out how the competition's Places pages are ranking so well? e.g. What have they categorised themselves under? Cheers in advance folks, JM
Competitive Research | | JamesMio0 -
Search shows site links, but not hyperlinked?
2 of our competitors on Google have site links show in their results which brings you to that specific page. In our listing we show up with the same results a few spots behind them with our site links visible showing very similar content, but are not actually hyper linked. Is there a way to fix the issue and let people actually be able to click on our site links instead of just reading them?
Competitive Research | | sknott0 -
How can I estimate a domain's overall organic search traffic - any tools?
Most of my analysis revolves around looking at rankings for specific keyword phrases that I've identified as important/relevant. But it'd be nice to be able to look at a domain and get a sense for how much organic traffic they get overall. If they're not ranking for the keywords I'm researching but have a lot of organic traffic that would be a nice signal to me that they are probably targeting other phrases more or have a big brand presence or something. Any suggestions? Thanks! Jeff Gibson
Competitive Research | | jeff.gibson0