How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO newbie here - I cannot seem to find the direct answers to a few queries that I have
Hi, Apologies if these are very simple queries, but I cannot seem to find the answers anywhere. I understand meta keywords are pretty much pointless nowadays, however when I run a report on another 'keyword website' - my website only ranks for around 2k keywords where my competitors rank for much more (around 20k). I was just wondering where these keywords are found, as I can only assume that these are meta keywords? The company that I'm working on the SEO for is a holiday agency that specialises in holidays abroad for disabled people, so that means that we have a niche product/service and whilst we rank fairly high for some keywords, we would like to rank higher. How is it possible to work towards ranking higher for particular keywords? When doing keyword research 'disabled holidays' has a lot more search traffic than 'accessible holidays'. We are called Enable Holidays and our main competitors have the words 'disabled' and 'accessible' in their domain's, so I would imagine that it's going to be hard to potentially rank higher than our competitors for these keywords? Could someone put me in the right direction of how you improve rankings in the SERP for certain keyword searches? For example, do we have to write more content on a certain page about a keyword? Do we have to include more long-tail keywords in the content? Any help is appreciated. Thanks
Competitive Research | | enableholidays0 -
Which is the best tool to check number of css, js, images and it's size?
Hello Team, Can you please suggest which is the best tool to check number of css, js, images and it's size of my competitors? Also what if they are using CDN network and bundling? Will that tool consider that also? Thanks! Wrights
Competitive Research | | wright3350 -
2 sites with low DA and PA and unoptimised <title>outranking everyone for competitive search term - How ?</title>
Im doing some keyword research for a new client, Ive found 2 anomalies for competitive search terms where sites with unoptimised <title>tags, poor backlinks and other stats, (compared to the other sites listed in the SERP) are out ranking everyone and claiming the #1 spot</p> <p> The 2 search terms are :</p> <ul> <li dir="ltr"> <p dir="ltr"><a href="https://www.google.co.uk/search?q=interior+designers+london&oq=interior+designers+london&aqs=chrome..69i57j69i65j69i60j69i65l2j69i60.2784j0j1&sourceid=chrome&ie=UTF-8">Interior designers London</a>- #1 new-id.co.uk - <a href="https://moz.com/researchtools/ose/links?site=new-id.co.uk&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a></p> </li> <li dir="ltr"> <p dir="ltr"><a href="https://www.google.co.uk/search?q=interior+designers+london&oq=interior+designers+london&aqs=chrome..69i57j69i65j69i60j69i65l2j69i60.2784j0j1&sourceid=chrome&ie=UTF-8#q=interior+designer+london">Interior designer London</a>- #1 nstudio.co.uk - <a href="https://moz.com/researchtools/ose/links?site=http%3A%2F%2Fnstudio.com%2F&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a></p> </li> </ul> <p> (interestingly they hardly rank at all for any modification of those search terms)</p> <p> The only thing i could think of is that they both had some really good links using that exact anchor text, but ive been checking OSE for the past month both in the backlinks and the just discovered but couldnt see anything like that.</p> <p>I conducted the SERP checking from a laptop with a broadband connection based in London, although im not very local to either of these locations im about an 8th of the cities distance away from one of those location and about half a cities distance away from the other. So i dont think its a local skew thats causing the SERP rankings.</p> <p>For both sites i can see they had old domains which now redirect to the them :</p> <ul> <li dir="ltr"> <p dir="ltr"><span data-mce-mark="1">New-id.co.uk = </span><span data-mce-mark="1"><a href="http://newidinteriors.com/">newidinteriors.com</a> - <a href="https://moz.com/researchtools/ose/links?site=http%3A%2F%2Fnewidinteriors.com&no_redirects=1&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a> - PA1, DA1</span></p> </li> <li dir="ltr"> <p dir="ltr"><span data-mce-mark="1">Nstudio.co.uk = intarya.com - <a href="https://moz.com/researchtools/ose/links?site=www.intarya.com&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a> - Da 32, PA43</span></p> </li> </ul> <p> Although the intarya domain was quite strong with good backlinks (but nothing above par comapred to the other sites the nstudio.co.uk site is now outranking, the anchor text was mostly natural with only some keyword anchor text from PA1 pages. </p> <p>The <a href="http://newidinteriors.com/">newidinteriors.com</a> didnt seem to have a backlink profile, both PA1 and DA1.</p> <p> Ive taken screenshots of the SERP pages below, but feel free to dig further</p> <p>Any idea what's going on here ?</p> <a download="HWkjmDf.png" class="imported-anchor-tag" href="http://i.imgur.com/HWkjmDf.png" target="_blank">HWkjmDf.png</a> <a download="A70jXse.png" class="imported-anchor-tag" href="http://i.imgur.com/A70jXse.png" target="_blank">A70jXse.png</a></title>
Competitive Research | | mike8780 -
How can I find out the exact match visitors for a competitor's website?
Hi, I'm trying to find a way of finding out the estimated number of exact match visitors that a competitor's website has got. I know that SEO agencies can find this out, and there must be a way of doing this for myself. Thanks
Competitive Research | | MThacker0 -
Feed sites replacing my pages in SERP
I have a problem! Sites like:
Competitive Research | | pycckuu
http://lynk.ly/
http://www.yousaytoo.com/
http://www.stumbleupon.com/
http://friendfeed.com/ Are replacing my posts even if i am searching for an exact title of the post. For example this search query:
Christmas Party Style Sets With Black Cocktail Dresses Original post here:
http://richclubgirl.com/christmas-party/christmas-party-style-sets-with-black-cocktail-dresses/ How do i fight this issue? I want to stay on these sites somehow! Thanks in advance!0 -
Analysis tool that returns all possible phrases that a web site ranks for
Is there anything out there similar to the semrush.com's competitor research that can tell you a web sites total amount of appearances on Google that includes the phrases for which the domain ranks?
Competitive Research | | thebaldman30 -
I must be tripping. How is this site ranking for the term "SEO Company"??????
When I go to Google non-custom search results and search the term: seo company The website "greencowseo.com" is ranked 3rd. OpenSiteSearch shows nothing. And get this... the domain was registered 14 days ago by somebody in Egypt. Do I need to quit my SEO Moz membership and hire this guy? 🙂
Competitive Research | | Bo-Jangles0 -
How do i improve my site's google ranking against my competitors?
the searches that i would like to improve my ranking on are
Competitive Research | | oswaldjr
st kitts real estate
nevis real estate
st kitts and nevis real estate. my website is www.sknlistings.com0