Disallow: /sr/ and Disallow: /si/ - robots.txt
-
Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember.
If any of you know what these directives do, then please let me know.
-
Thanks Tomas and Mike - good advice - I have done that and found legacy stuff they've since moved away from - there is indeed no current use for the directives.
I wonder whether there's any resource on the web that lists all robots.txt directives - and interprets them - if not then perhaps it would an idea for Moz?
-
Have a look at your site through http://web.archive.org/. You'll be able to see what the directories were used for.
However, if there's no use for them on the current site then what's the purpose of keeping these disallows in the robots.txt?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does a similar CMS fabric/theme used by 2 colleagues, cause SEO issues?
Does using the same CMS fabric/theme with substantially different content for each website but the same business address (on google places and the websites) cause SEO issues? For example: 2 colleagues with somewhat similar services located at the same business addresses desire to use the same CMS fabric /theme but will have different content on the sites. Will this hurt their SEO / should they use a different website skin/theme?
Web Design | | toti5880 -
Https Implementation - Weird Redirection After Putting 's' in http://
Hi Mozers, I have come across some websites with their https version going to a totally different website. For example, http://www.samplesite1.com will load fine but when the protocol is changed to https (https://www.samplesite1.com) it will go to a total different domain say, https://www.samplesite2.com How does this happen, in technical sense? In the warning from browser, it says the the security certificate is from the other website but I would like to understand how this happens and how it impacts SEO. I seem to be not able to understand the relationship of this error and SEO impact. Thanks in advance for your response. Malika
Web Design | | Malika10 -
Any alternative techniques to display tabbed content without using Javascript / JSON and be SEO Friendly?
John Mueller's input in the EGWMH hangout suggests that Google MAY ignore expandable content served by Javascript. Are there any alternative techniques to display tabbed content without using Javascript / JSON and be SEO Friendly? I do however view these as good for website interactivity and UX - and see many examples of websites performing well and ranking highly whilst using these techniques - are there any Google friendly ways to serve content on a page so that search bots can recognise and choose to crawl / consume the content as legitimate fodder?
Web Design | | Fergclaw0 -
Managing website content/keywords for wordpress site
We are in the midst of redesigning our website and have been working with freelance blog/content writers to increase the unique content on our site. We are finding it increasingly difficult to manage the topics/keywords as we continue to expand. Googledrive and google spreadsheets have been our primary tools thus far. Can anyone recommend a good tool that would allow us to manage content and blog posts for our site?
Web Design | | Tom_Carc0 -
Blog.yoursite.com or yoursite.com/blog/?
Rand made the following post in 2009. In July of 2013 I question whether this is still best practice. Obviously, much has changed in our industry in four-and-a-half years! "...if you're seeking to maximize your ranking ability for a given piece of content, it's my personal belief that you should, most of the time, keep it on 1 subdomain under 1 root domain (but feel free to use subfolders as it makes sense). Starting a blog? I almost always recommend yoursite.com/blog over blog.yoursite.com. Want to launch a new section of content? Use yoursite.com/newstuff rather than newstuff.yoursite.com." Source: http://moz.com/blog/understanding-root-domains-subdomains-vs-subfolders-microsites
Web Design | | JorgeUmana0 -
Need help to implement microdata/microformat for ecommerce site
**Can somebody please help me to implement microdata/microformats codes for our ecommerce product pages? **
Web Design | | EastEssence22
Please guide me if you have some CSS example for the same. Thanks.0 -
Site Change / CMS Change
Hi Guys I have a very good client with whom we have been working with for over 2 years. When we 1st arrived, their website was built in Wordpress. During our SEO, the dleted the site and changed it to Drupal over night. Youc an imagin i went balistic and its been a horrid year in getting all the rrors down and redirects etc. Finally done I have just been informed that they will be deleting the website from Drupal to Joomla again! This is crazy as we have finally fix all the rrors and are driving quality traffic to the site. Now its back to square one... What do i do? ps the change from Drupal to Joomla is due to MYSQL issues
Web Design | | stefanok0 -
XML Sitemap that updates daily/weekly?
Hi, I have a sitemap on my site, that updates but it isn't a XML sitemap. See here: http://www.designerboutique-online.com/sitemap/ I have used some free software to crawl the site and create a sitemap of pages, however I think that if I were to upload the sitemap, it would be out of date as soon as I listed new products on the site, so would need to rerun it. Does anyone know how I can get this to refresh daily or weekly? Or any software that can do it? I have a web firm that are willing to do one, but our relationship is at an all time low and I don't want to hand over £200 for them to do one. Anyone with any ideas or advice? Thanks Will
Web Design | | WillBlackburn0