Disallow: /sr/ and Disallow: /si/ - robots.txt
-
Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember.
If any of you know what these directives do, then please let me know.
-
Thanks Tomas and Mike - good advice - I have done that and found legacy stuff they've since moved away from - there is indeed no current use for the directives.
I wonder whether there's any resource on the web that lists all robots.txt directives - and interprets them - if not then perhaps it would an idea for Moz?
-
Have a look at your site through http://web.archive.org/. You'll be able to see what the directories were used for.
However, if there's no use for them on the current site then what's the purpose of keeping these disallows in the robots.txt?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Having a Subfolder/Subdirectory With a Different Design Than the Root Domain
Hi Everyone, I was wondering what Google thinks about having a subfolder/subdirectory with a different design than the root domain. So let's say we have MacroCorp Inc. which has been around for decades. MacroCorp has tens of thousands of backlinks and a couple thousand referring domains from quality sites in its industry and news sites. MacroCorp Inc. spins off one of its products into a new company called MicroCorp Inc., which makes CoolProduct. The new website for this company is CoolProduct.MacroCorp.com (a subdomain) which has very few backlinks and referring domains. To help MicroCorp rank better, both companies agree to place the MicroCorp content at MacroCorp.com/CoolProduct/. The root domain (MacroCorp.com) links to the subfolder from its navigation and MicroCorp does the same, but the MacroCorp.com/CoolProduct/ subfolder has an entirely different design than the root domain. Will MacroCorp.com/CoolProduct/ be crawled, indexed, and rank better as both companies think it would? Or would Google still treat the subfolder like a subdomain or even a separate root domain in this case? Are there any studies, documentation, or links to good or bad examples of this practice? When LinkedIn purchased Lynda.com, for instance, what if they kept the https://www.lynda.com/ design as is and placed it at https://www.linkedin.com/learning/. Would the pre-purchase (yellow/black design) https://www.linkedin.com/learning/ rank any worse than it does now with the root domain (LinkedIn) aligned design? Thanks! Andy
Web Design | | AndyRCWRCM1 -
What are the downsides and/or challenges to putting page paths (www.example.com/pagepath) on a different server?
Hi, Our company is organized into three different segments and our development team recently needed to switch a portion of the business to subdomain because they wanted to move to a different server platform. We are now seeing the impact of moving this segment of the business to a subdomain on the main domain. SEO is hurting and our MOZ score has dropped significantly. One fix they are debating is moving everything back to one domain, but place segments of the business on different page paths and hosting specific paths on different servers. I.e. the main domain could be www.example.com hosted in one location and then www.example.com/segment1 would be hosted on a different server. They are hoping to accomplish this using some sort of proxy/caching redirection solution. The goal of this change would be to recapture our domain strength. Is this something that is a good option or no? If not, what are the challenges and issues you see arising from doing something like that as I don't know of any other site set up like this. Thanks in advance.
Web Design | | bradgreene0 -
Is it still necessary to have a "home" page button/link in the top nav?
Or is it not necessary to have a "home" tab/link because everybody by this time knows you can get to the home page by clicking on the logo?
Web Design | | FindLaw0 -
Link juice passing from a .org.uk link to a .org/uk websites
Hi all, A client I am working on had a CMS built in recently which has resulted in all their canonicals tags being taken off the website, and as such the same page with both a .org/uk and .org.uk/uk domain have appeared in the search results and I am wondering what your guys take is on the best cause of action. For further background: Historically they have always used .org.uk/uk (not sure why) for their UK website and used .org/xxx for other countries (they also have a .org splashpage FYI). Having seen the .org/uk pages, and knowing they have to choose one to avoid duplication, they would like to move their uk website to the .org/uk domain to fit in with the rest of the divisions. However due to the historical use of .org.uk/uk their backlink profile contains links to both the .org.uk and .org domains. My question then: would a canonical tag on all the .org.uk/uk pages pointing to the .org/uk pages be strong enough to pass on link juice to the .org/uk pages (from all links pointing to .org.uk) or would a 301 redirect be required in this instance, or indeed would it be best to stay with the .org.uk/uk domain? Thanks, Diana
Web Design | | Diana.varbanescu0 -
Comparing the site structure/design of my live site to my new design
Hi SEOmoz team, for the last few months I've been working on a new design for my website, the old, live design can be viewed at http://www.concerthotels.com - it is primarily focused on helping users find hotels close to concert venues throughout North America. The old structure was built in such a way that each concert venue had a number of different pages associated with it (all connected via tabs) - a page with information about the venue, a page with nearby hotels to the venue, a page of upcoming events, a page of venue reviews. An example of these pages can be seen at: http://www.concerthotels.com/venue/madison-square-garden/304484 http://www.concerthotels.com/venue-hotels/madison-square-garden-hotels/304484 http://www.concerthotels.com/venue-events/madison-square-garden-events/304484 http://www.concerthotels.com/venue-reviews/madison-square-garden-reviews/304484 The /venue-hotels/ pages are the most important pages on my website - and there is one of these pages for each concert venue - they are the landing pages for about 90% of the traffic on the website. I decided that having four pages for each venue was probably a poor design, since many of the pages ended up having little or no useful, unique content. So my new design attempts to bring a lot of the venue information together into fewer pages. My new website redesign is temporarily situated at: (not currently launched to the public) http://www.concerthotels.com/frontend The equivalent pages for Madison Square Garden are now: http://www.concerthotels.com/frontend/venue/madison-square-garden/304484 (the page above contains venue information, events and reviews) and http://www.concerthotels.com/frontend/venue-hotels/madison-square-garden-hotels/304484 I would really appreciate any feedback from you guys, based on what you think of the new site design compared to the old design from an SEO point of view. Of course, any feedback on site speed, easy of use etc compared to the old design would also be greatly appreciated. 🙂 My main fear is that when I launch the new design (the new URLs will be identical to the old ones), Google will take a dislike to it - I currently receive a large percentage of my traffic through Google organic search, so I don't want to launch a design that might damage that traffic. My gut instinct tells me that Google should prefer the new design - vastly reduced number of pages, each page now contains more unique content, and it's very much designed for users, so I'm hoping bounce rate, conversion etc will improve too. But my gut has been wrong in the past! 🙂 But I'd love to hear your thoughts, and thanks in advance for any feedback, Cheers Mike
Web Design | | mjk260 -
Redirects (301/302) versus errors (404)
I am not able to convincingly decide between using redirects versus using 404 errors. People are giving varied opinions. Here are my cases 1. Coding errors - we put out a bad link a. Some people are saying redirect to home page; the user at least has something to do PLUS more importantly it does NOT hurt your SEO ranking. b. Counter - the page ain't there. Return 404 2. Product removed - link1 to product 1 was out there. We removed product1; so link1 is also gone. It is either lying in people's bookmarks, OR because of coding errors we left it hanging out at some places on our site.
Web Design | | proptiger0 -
Wordpress/ Insert Tables/ SEO
I'm using Wordpress to create websites and blogs. I have limited (non-existent) HTML Coding knowledge. I'm looking to insert tables within my pages with information. Inside of these tables I want certain names to link to another page with more specific information about that name. I'm using a plugin called "WP Tables Reloaded" it simple helps you to create aesthetically pleasing tables without needing to know HTML Code or CSS. The issue is... when you create this table and insert it to the post, the only thing that shows on the sites back-end page is the table I.D. and the only thing that shows in the HTML is the tables I.D. It looks like this... [table id=2 /] I don't think search engines will be able to crawl this table, thus I won't be receiving any credit for the links being used within the table. Am I right about this?
Web Design | | AndySolo0 -
Landing Page/Home Page issues
Hi. I was speaking with my designer last night (we are setting up a new website) and we were discussing the design of our homepage, now the designer said he wanted the first page of the website to be a sort of landing page page were the visitor has to click and enter, im sure everyone has all come across these before. However, I am concerned as to the SEO implications of this? Any help guys?
Web Design | | CompleteOffice0