Disallow: /sr/ and Disallow: /si/ - robots.txt
-
Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember.
If any of you know what these directives do, then please let me know.
-
Thanks Tomas and Mike - good advice - I have done that and found legacy stuff they've since moved away from - there is indeed no current use for the directives.
I wonder whether there's any resource on the web that lists all robots.txt directives - and interprets them - if not then perhaps it would an idea for Moz?
-
Have a look at your site through http://web.archive.org/. You'll be able to see what the directories were used for.
However, if there's no use for them on the current site then what's the purpose of keeping these disallows in the robots.txt?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does a similar CMS fabric/theme used by 2 colleagues, cause SEO issues?
Does using the same CMS fabric/theme with substantially different content for each website but the same business address (on google places and the websites) cause SEO issues? For example: 2 colleagues with somewhat similar services located at the same business addresses desire to use the same CMS fabric /theme but will have different content on the sites. Will this hurt their SEO / should they use a different website skin/theme?
Web Design | | toti5880 -
Has anyone ever tested to see if having an ads.txt file provided any SEO lift?
I know that the ads.txt system is designed to prevent ad fraud and technically has nothing to do with search. That said, the presence of such a file would seem to be an indicator of overall site quality because it would show that a site owner wants to participate in a fraud-free system. Has anyone ever tested that? If so, they don't seem to have published their results. Maybe it's a secret weapon that some pros are using and not sharing?
Web Design | | scodtt0 -
How to split organic traffic for A/B testing
This might be a silly questions as I may be missing something completely obvious here, but we are completely new to A/B testing. Our site doesn't receive a phenomenal amount of traffic although we are looking to set up some A/B testing for our popular products. Is there a way to split organic traffic for a specific product page. I'm aware that we need to experiment which one performs better in Analytics but I'm unsure how to redirect 50% of the organic traffic.
Web Design | | Jseddon920 -
Do I need to 301 redirect www.domain.com/index.html to www.domain.com/ ?
So, interestingly enough, the Moz crawler picked up my index.html file (homepage) and reported duplicate content, of course. But, Google hasn't seemed to index the www.domain.com/index.html version of my homepage, just the www.domain.com version. However, it looks like I do have links going specifically to www.domain.com/index.html and I want to make sure those are getting counted towards my overall domain strength. Is it necessary to 301 redirect in the scenario described above?
Web Design | | Small_Business_SEO0 -
Image with 100% width/height - bad ranking?
Hi, we have some articles like this: http://www.schicksal.com/Orakel/Freitag-13 The main image has a width of 100% and a height of 100%. Today, I've discovered that GWT Instant Preview has some troubles with rendering the page. We have CSS rules to deliver the image with the right dimensions. If a bot like google is not sending any screen height / width we assume the screen size is 2560x1440. Does this harm the ranking of the page? (Content starts below the fold/image) What is a "default" screen size for google? How do they determine if something is "above the fold"? Any tips or ideas? Best wishes, Georg.
Web Design | | GeorgFranz0 -
Nav / Sitemap Question. Using a "services" page vs just linking directly to individual service page?
Okay, so our company offers video production, web design, and web marketing services. While we do offer these services individually, our goal is to get our clients to integrate these services together. Our nav is currently like so : home - about - video - web design - web marketing - blog - contact Now I've seen businesses and agencies also use a nav with a "services" button instead of listing out their service offerings (if they have more than 1, like us). The services button usually links to a category page or has a drop down with links to the company's individual services. I'm wondering if there is any benefit to having a main services page like this and linking to the individual pages off of it (video ,web design, marketing, etc). Or if we should just keep it the way we have it now (since we've already got some page authority on the individual service pages). I know this may not be the most important aspect of our site and we may be over-thinking it but any thoughts/ideas would be greatly appreciated, thanks!
Web Design | | RenderPerfect0 -
Best way to handle Spanish/English WEBSITE
Hey guys, How are you doing? I have a website (www.aceromart.com ) in which the primary language is Spanish. The company is oriented toward the Mexican Audience. However, recently we are dealing with many U.S companies. Also, we want to be included in the Yahoo Directory and several other directories which demand a U.S version of the website. So i want to have a U.S version of the website. My ecommerce is based in the NETWORK Solutions platform and has around 1,000 products and pages. What is the best way to include an English version of the website. Ive seen some pages using the Google traslate tool, which only traslates the text. Hope to hear some of your ideas, Regards,
Web Design | | JesusD0 -
After a website redesign, what is the impact and is it a good practice to use /v2/ naming convention?
Hi mightyful SEOMoz community. We just launched a redesign of our commercial website from https://www.data-field.com to https://www.data-field.com/v2/ All URLs from previous website were 301 permanent redirect to the appropriate page in the new website, and the root domains ( /, /v2/ ) send the users to their own language content /v2/en/, /v2/fr/, /v2/zh/ Up to here everything is fine. But then I setup the usual "Share" buttons, only to find that they were displaying a "0" count. Then I realized that it was because of the root URL change from / to /v2/ My question is the following: 1. Is using /v2/ a good practice? 2. If yes, then should I link the Social tool to https://www.data-field.com/ ( only ) instead of linking it to the actual page in the address bar? Thanks for your answers.
Web Design | | NicolasE0