Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Disallow: /sr/ and Disallow: /si/ - robots.txt

Web Design

650

McTaggart last edited by

Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember.

If any of you know what these directives do, then please let me know.
1 Reply Last reply
Reply Quote 0
McTaggart last edited by

Thanks Tomas and Mike - good advice - I have done that and found legacy stuff they've since moved away from - there is indeed no current use for the directives.

I wonder whether there's any resource on the web that lists all robots.txt directives - and interprets them - if not then perhaps it would an idea for Moz?
1 Reply Last reply
Reply Quote 0
ThomasHarvey last edited by

Have a look at your site through http://web.archive.org/. You'll be able to see what the directories were used for.

However, if there's no use for them on the current site then what's the purpose of keeping these disallows in the robots.txt?
1 Reply Last reply
Reply Quote 2

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Google is a definition engine / Question answering engine

But what do you do to rank when you sell a product and people don't have questions and none of the companies that rank on your keywords answers questions... Let me give you a few example. Title tag for example : People want to know what is it and then have question about it What is the maximum number of characters ? How do you write a good one ? Etc... How to remove stains is another example : Please want to know how to remove all the different types of stains, (ketchup, grease etc..) But what about when you are a online business and want to sell usb keys , rent bicycles https://www.spinlister.com or even sell a software to do A/B testing on your website https://bit.ly/2a6cBuF Can someone explain me how those people mentioned rank without giving definition or answering questions... because according to me they don't do any of that on the pages that I mentioned. I look forward to your reply. Thank you,
Web Design | | seoanalytics

0
Directing a domain.com/BLOG to a different host

Hello,
i cant seem to figure out how to approach this matter. we have a website and a domain setup in one place and we wanted to setup a blog for that website on a different hosting, what do i have to do to link between the two to make it so when a user type domain com /blog he will get to our blog that is hosted on a different hosting plan? with the blog hosting plan we received a temporary domain that does not expire but i have been told that if i redirect to that temporary domain it wont work correctly also, if we do manage to go through that process will accessing the blog will the URL show domain com/blog or will still show the temporary given domain? is my only option is buying a new domain and just link between the two? Thanks!
Web Design | | SharonEKG

0
Bing Indexation and handling of X-ROBOTS tag or AngularJS

Hi MozCommunity, I have been tearing my hair out trying to figure out why BING wont index a test site we're running. We're in the midst of upgrading one of our sites from archaic technology and infrastructure to a fully responsive version.
This new site is a fully AngularJS driven site. There's currently over 2 million pages and as we're developing the new site in the backend, we would like to test out the tech with Google and Bing. We're looking at a pre-render option to be able to create static HTML snapshots of the pages that we care about the most and will be available on the sitemap.xml.gz However, with 3 completely static HTML control pages established, where we had a page with no robots metatag on the page, one with the robots NOINDEX metatag in the head section and one with a dynamic header (X-ROBOTS meta) on a third page with the NOINDEX directive as well. We expected the one without the meta tag to at least get indexed along with the homepage of the test site. In addition to those 3 control pages, we had 3 pages where we had an internal search results page with the dynamic NOINDEX header. A listing page with no such header and the homepage with no such header. With Google, the correct indexation occured with only 3 pages being indexed, being the homepage, the listing page and the control page without the metatag. However, with BING, there's nothing. No page indexed at all. Not even the flat static HTML page without any robots directive. I have a valid sitemap.xml file and a robots.txt directive open to all engines across all pages yet, nothing. I used the fetch as Bingbot tool, the SEO analyzer Tool and the Preview Page Tool within Bing Webmaster Tools, and they all show a preview of the requested pages. Including the ones with the dynamic header asking it not to index those pages. I'm stumped. I don't know what to do next to understand if BING can accurately process dynamic headers or AngularJS content. Upon checking BWT, there's definitely been crawl activity since it marked against the XML sitemap as successful and put a 4 next to the number of crawled pages. Still no result when running a site: command though. Google responded perfectly and understood exactly which pages to index and crawl. Anyone else used dynamic headers or AngularJS that might be able to chime in perhaps with running similar tests? Thanks in advance for your assistance....
Web Design | | AU-SEO

0
Is it still necessary to have a "home" page button/link in the top nav?

Or is it not necessary to have a "home" tab/link because everybody by this time knows you can get to the home page by clicking on the logo?
Web Design | | FindLaw

0
Using Multiple links/names for the same product?

I am being asked to change these product links on the home page: Home/Condo
Watercraft/Boat to Home
Condo
Watercraft
Boat (Along with several other product links) How does this affect the customer experience/usability, and SEO? Is it a good idea or is it confusing? Thank you.
Web Design | | RoxBrock

0
Google pagespeed / lazy image load

Hi, we are using the apache module of google pagespeed. It works really great, helps a lot. But today I've asked me one question: Does the "lazy load" feature for images harm the ranking? The module reworks the page to load the images only if the are visible at the screen. Is this behavior also triggered by the google bot? Or are the images invisible for google? Any expirience about that? Best wishes, Georg.
Web Design | | GeorgFranz

0
Is it cloaking/hiding text if textual content is no longer accessible for mobile visitors on responsive webpages?

My company is implementing a responsive design for our website to better serve our mobile customers. However, when I reviewed the wireframes of the work our development company is doing, it became clear to me that, for many of our pages, large parts of the textual content on the page, and most of our sidebar links, would no longer be accessible to a visitor using a mobile device. The content will still be indexable, but hidden from users using media queries. There would be no access point for a user to view much of the content on the page that's making it rank. This is not my understanding of best practices around responsive design. My interpretation of Google's guidelines on responsive design is that all of the content is served to both users and search engines, but displayed in a more accessible way to a user depending on their mobile device. For example, Wikipedia pages have introductory content, but hide most of the detailed info in tabs. All of the information is still there and accessible to a user...but you don't have to scroll through as much to get to what you want. To me, what our development company is proposing fits the definition of cloaking and/or hiding text and links - we'd be making available different content to search engines than users, and it seems to me that there's considerable risk to their interpretation of responsive design. I'm wondering what other people in the Moz community think about this - and whether anyone out there has any experience to share about inaccessable content on responsive webpages, and the SEO impact of this. Thank you!
Web Design | | mmewdell

0
/index.php/ What is its purpose and does it hurt SEO?

Hello Moz Forum, I am still in the process of cleaning up the lack of attention to detail and betrayal set by our soon to be ex-SEO company. You can see a previous question I ask regarding betrayal SEO. I am analyzing every page on our website and i am noticing this /index.php/ in most of our URLs. We want to leave our expression engine cms and convert to wordpress. I have been reading about index.php but most of it is over my head for now. What does concern me is the "layman's" findings i am seeing through analytics. Our main domain has two URLs. one that ends in .com and the other ends in .com/index.php/ The one that ends in .com has a higher page rank than the ladder. And there are other internal pages with the same two variations. Can someone please explain to me what is /index.php/ ? what are the benefits of it? what are the cons? What will happen to my site once we move to wordpress? As always, your comments and suggestions are greatly appreciated.
Web Design | | CamiloSC

0