How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it possible to find out how many indexed pages a website had at particular dates in the past?
I've found all kinds of websites and tools that can tell me how many pages a site has indexed currently, but how about historically? Does such a tool exist?
Competitive Research | | bigvibes9090 -
How to top competitor's article
Hello, For bobweikel.com I'm going to work on getting some natural backlinks by making a better article than http://www.medicinenet.com/weight_loss/article.htm even though they're 30 times more of an authority than me. If you think this is a good idea, how do you recommend I top this thing? I'm just brainstorming
Competitive Research | | BobGW0 -
What's causing this site to rank better than another?
As I work on getting a grasp for what causes sites to rank better than others, I'm a bit stumped with two of our competitors. When you search Google for "essay editing" you get these results: 1. Ivyeyesediting.com 2. papercheck.com 3. gradesaver.com 4. scribendi.com 5. essayedge.com When you put these in OSE, it seems crazy that Ivyeyes is anywhere near the top. For example, essayedge creams these guys on EVERY single category. As a new SEO here, what am I missing. This makes absolutely no sense to me. Not only that, it's pretty discouraging seeing essayedge so far down the SERPS with what appears to be some really great numbers. Can anyone shed some light on this?
Competitive Research | | Kibin0 -
SEO's done, 301s in place, old site STILL outranks new site. What to do?
Since Sep 2010 I have had a site up with minimal SEO optimization (www.chrisbrushmusic.com). Oct 29, 2012, I launched a new site on a new domain (www.chrisbrushdrums.com) with more content and tons of SEO work behind it. The content of the new site is significantly different from the old site, and I wish to keep the old site around. I have 301's in place for specific URLs on the old site that point to the new site. I have submitted xml sitemaps for the new site. As of now, the old site still outranks the new site (i.e. Google search for "nashville session drummer" and my old site is #9 - my new site is nowhere). What should I do? Thanks.
Competitive Research | | cbrush0 -
How do I find my client's real competition?
A recently acquired client keeps insisting that his major competitors are online fashion magazines, fashion news sites and insider bloggers but the site he owns is basically a fashion encyclopedia. It contains facts and catalogues and even a dictionary of fashion concepts. According to me, these latter are his benchmarks and not his competition His major competitors, according to me. are people competing on the first SERP pages for terms like "fashion history", "fashion icons", "fashion biographies", etc. Am I right? How do I convince him that these are his real competition. This is really going to determine the direction of my keyword research.
Competitive Research | | amit20760 -
Free SEO tools appearing in SERPS for a site
Hi there, I've just taken over a site and it doesn't have a huge amount of shelfspace on the SERPs for it's brand name. Looking at the results that are there are a lot of free SEO tools that are coming up. They haven't had an SEO before so it's not coming from their side. There's such a lot of them I was wondering if anyone had come across this before - is it simply a competitor looking at the seo, could it be one of the seo analytics packages using free sites or is there anything else to be aware of. I can start knocking the sites down the SERPs but just wondering if anyone has a good solution for this. thanks Shiv
Competitive Research | | Shivvyt0 -
How can my site not ranked on easy keyword while it currently ranked 1 on difficult keyword??
The site in question is http://ikt.co.id The site is perfectly optimized (or even over optimized) for keyword "Bikin Website Murah" (in Indonesia it means "develop an affordable website") currently if you search on both google.co.id AND google.com, it ranked on #10 (on google.co.id) and it ranked on #7 (on google.com), and both term only return 1.3 million results I have A LOT OF pages that started with 'bikin website' , or 'website murah' ... there are a lot of them, and i have had a lot of backlinks on those pages too (have been manually building a lot of links, starting from creating a tutorial article and blog submission and many other way) .. the site has been perfectly optimized for that keyword For example we have http://ikt.co.id/blog-category/bikin-website-murah-promo-terbaru.html (for newest promotion) http://ikt.co.id/blog/bikin-website-seo-murah-discount-40-sampai-31-juli-2011.html (for discount 40% promo that lasted untill 31 juli 2011) AND YET, if you search on google.co.id for keyword "2000 likes facebook" , our lone page that i created when our facebook likes reached that amount, RANKED #1!! , and please let me tell you this term return 400 million results++ (on google.com this keyword ranked #19, on 500 million results++). And let me tell you it take that SINGLE page (which has absolutely 0 backlinks) 2 days to ranked #1 .... and her brother if you searched "1000 likes facebook" also show similar result (though not better then the sister) How ?!? and Why ?!? , i dont understand this at all.... my domain does not contain any keyword, they uses the same engine and yet that lone pages which have absolutely no backlink ranked so well on a very competitive keyword, while my Target Keyword ranked poorly.... And what is even more bizzare? "Bikin website murah" might not rank good, but "bikin website" ranked better x_x ... arghh i dont get it... why my site ranked poorly on easy keywords and ranked high on difficult keywords :(( .... Why I am not smiling because those easy keyword is actually my primary keywords 😞
Competitive Research | | IKT0 -
Alt image or allow /image in robot.txt
Howdy, Me and my co-workers had a discussion at lunch time. We wonder if we should allow /image in our robot.txt. We arent sure what rank best. Image whit keyword in it or simply had a Alt at each image and hide the /image in the robot.txt... or have a allow /image AND a alt source. P.S. At this point, we didnt work on any sitemap or link it anywhere. Ty.
Competitive Research | | Promoteam0