How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html) -
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO newbie here - I cannot seem to find the direct answers to a few queries that I have
Hi, Apologies if these are very simple queries, but I cannot seem to find the answers anywhere. I understand meta keywords are pretty much pointless nowadays, however when I run a report on another 'keyword website' - my website only ranks for around 2k keywords where my competitors rank for much more (around 20k). I was just wondering where these keywords are found, as I can only assume that these are meta keywords? The company that I'm working on the SEO for is a holiday agency that specialises in holidays abroad for disabled people, so that means that we have a niche product/service and whilst we rank fairly high for some keywords, we would like to rank higher. How is it possible to work towards ranking higher for particular keywords? When doing keyword research 'disabled holidays' has a lot more search traffic than 'accessible holidays'. We are called Enable Holidays and our main competitors have the words 'disabled' and 'accessible' in their domain's, so I would imagine that it's going to be hard to potentially rank higher than our competitors for these keywords? Could someone put me in the right direction of how you improve rankings in the SERP for certain keyword searches? For example, do we have to write more content on a certain page about a keyword? Do we have to include more long-tail keywords in the content? Any help is appreciated. Thanks
Competitive Research | | enableholidays0 -
Why would a newly created site, have ranked ahead of our site for keyword that we are optimized for?
This newly created site has a DA of 1 and PA of 1, no backlinks, no optimized urls, the keyword they ranked better than us on was listed a total of 6 times on the homepage. Our PA is 29 and DA is 18 for the page that ranks for this keyword. They really copied a few elements of our site but made sure to change a few things, but also list at the bottom of their site 5 keywords that are crucial to our niche industry but they're all linking to the same page. Any ideas? It's an SEO guy running the site, we've watched them toy with adwords trying to be number 1, but not liking the price, so they are here and there with it. Mainly I don't see why they'd rank better for this keyword, we our site have prolly 500% more content that's both of quality and relevance to our customers, in the form of Pdfs, infographics, help sections, and video. Very baffled here, any advice would rock!
Competitive Research | | Deacyde1 -
How to top competitor's article
Hello, For bobweikel.com I'm going to work on getting some natural backlinks by making a better article than http://www.medicinenet.com/weight_loss/article.htm even though they're 30 times more of an authority than me. If you think this is a good idea, how do you recommend I top this thing? I'm just brainstorming
Competitive Research | | BobGW0 -
Where do we find a Keyword Discovery Tool in SEOmoz?
I've been looking for a way to compare keywords amongst my competition websites.
Competitive Research | | homesonthesound0 -
Tool for finding what keywords a competitor ranks for?
Does anyone know of any good tools that display what keywords a competitor ranks for? I have many competitors that I know get a lot of traffic, but I'm not entirely sure where the traffic comes from so it would be nice to plug in their url and get a general overview of what keywords they rank for and what positions.
Competitive Research | | shawn810 -
Site Ranks Better than Comp...but why?
My site is ranking better than some of the competition sites even though the Moz Rank and Moz Trust are lower, anyone know why?
Competitive Research | | CalgaryRealtor390 -
How due I take the advanced report that i have exported as a excel file and recreate your online color coded version with the data.
In the keyword difficulty screen how do I recreate your beutiful color coded spreadsheet in the exported excel file based on the advanced reports.
Competitive Research | | bmcfarlane0 -
In Open Site Explorer, what does it mean when a linking page does not contain any reference to the URl entered?
When running Open Site Explorer on a particular URL, I get a list of linking pages. Many of these pages have a high Page Authority. I am assuming that this is a list of pages that presumably link to the URL I entered. First, is this correct? Next, when I click on an entry in the list I don't see any reference to the URL on the page, even viewing the page source. What does this mean and why is the link in the list?
Competitive Research | | jkenyon1