Robots.txt question
-
What is this robots.txt telling the search engines?
User-agent: * Disallow: /stats/ -
Oh - and it's affect the domain negatively.. when cleaning up your site directories via robots.txt. Its actually better as I explained below
-
Hey Mark,
It's good practice to disallow access to any folder/content you don't want indexed as well as anything that has any security involved (login's, databases etc).
It will also keep the most important pages from the domain in front of the search spiders eyes, while keeping poor content out of the indes. This helps the domain on a site authority level provide valuable content and information to users.
Lower ranking pages, can cause the domain to be pulled down by serarch engines (Google and Bing have attested to this already) as they want businesses to focus on high value content - which leads to better user experience.
Cheers!
-
Thanks- wanted to make sure all was copacetic there. I'm assuming that it's good practice to disallow access to stats and won't impact the site negatively?
-
Assuming that this is the entire contents of this file: It says that no robot (search engine spider, other crawler, etc.) should visit or index anything in the /stats/ directory or any directories inside of it.
More info available here: http://www.robotstxt.org/robotstxt.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Adding your sitemap to robots.txt
Hi everyone, Best practice question: When adding your sitemap to your robots.txt file, do you add the whole sitemap at once or do you add different subcategories (products, posts, categories,..) separately? I'm very curious to hear your thoughts!
Technical SEO | | WeAreDigital_BE0 -
Website(s) Merge Questions
Hello Moz Community I have three separate websites that I have planned on merging into one. I recently acquired a strong domain ( stronger than my existing ones ) & after much deliberation, I have decided to move them together into one domain. Site 1) 1 Forum - Around since 2007 - Originally VBulletin, last year migrated to discourse. Site 2) E-commercece Magento Site - Around since 2010 Site 3) WordPress - Blog & Articles - Around since last year - Currently using the new ( Main Domain I want to use ) After a lot of research and White Board Fridays, I was thinking this would be my best bet: newdomain.com - Merged WordPress & Magento CMS Page. newdomain.com/ forum <-- Will move my forum content & Install to the new domain. newdomain.com/ store <-- Will move my Ecommerce content & Install. Between my developer and I, we have our heads around how to handle the technical aspect of the move, 301s to the new location etc. But one area we want to research before pulling the trigger ( we haven't found much data on this ) **My Main Question:**What is the possible penalty coming for shopping websites vs. content-driven websites, and possibility that forum-based content hosted on the same domain might cause another penalty to be applied to the other content on the domain. **My Developer Says:**In recent years, we noticed that forum content had been penalized in favor of editorially reviewed content when this is identified by search engines.
Technical SEO | | Shop-Sq
What big hiccups would / could we encounter from combining these 3 types of platforms into one site?Many thanks for any direction or insight.0 -
Google My Business Service Area Question
Hello Moz Friends I just wanted to make sure I'm doing things correctly. On google my business your given the option to list your service area. I serve the entire state of Colorado with my internet marketing services. So I listed Colorado as my service area. but Moz Friends, is this the wrong idea? Like should I list the major cities and call it good? So instead of service area Colorado, I should put Denver, Colorado Springs, Pueblo etc Thank you for your friendly help Chris
Technical SEO | | asbchris0 -
Hi - I have a question about IP addresses
- would it hurt link juice to host a blog on a different server to the rest of your website? I have a web host saying they can't run Wordpress as they won't support PHP for "security reasons" - one solution would be to set up Wordpress on a different server and redirect domain.com/blog there (I presume this is do-able?). But I don't know if that affects the SEO adversely?
Technical SEO | | abisti21 -
Htaccess redirect with question mark
Hi I have a problem setting up my htaccess for a specific page that has a question mark in the link, and one that has a space in the link and also a question mark. So I would like 2 redirects in my htaccess like that: www.olddomain.com/page.php?page=pagename1 to www.newdomain.com/newpage1.html www.olddomain.com/page.php?page=page name2 to www.newdomain.com/newpage2.html I have tried with something like this but doesn't work: RewriteEngine on RewriteRule ^page.php?page=pagename1 "http://www.newdomain.com/newpage1.html" [R=301,L] RewriteRule ^page.php?page=page name2 "http://www.newdomain.com/newpage2.html" [R=301,L] Could someone tell me what exactly I have to change? Thanks
Technical SEO | | darkanweb0 -
Question about content on ecommerce pages.
Long time ago we hired a seo company to do seo in our website and one of the things they did is that they wrote long text on the category pages of our products. Example here: http://www.theprinterdepo.com/refurbished-printers/wide-format-laser-refurbished-printers Now my marketing person is saying that if its possible to put the text below the items, technically I will find out how to do it, but from your seo experience, is it good or bad? What about if we short those texts to one paragraph only? Thanks
Technical SEO | | levalencia10 -
Question concerning a 302 Redirect
Hi! I've already done some research on redirects, but I still have a question concerning a 302 redirect implemented at the homepage of a website. The Website www.domainA.com has a 302 redirect to www.domainA.com/content/.... Also all subsequent pages have the /content/ directory in their URLs: e.g domainA.com/content/products First thing I was wondering about, was the use of a redirect to a new site using an additional directory /content/... Why would anyone do this? Would it be enough to replace the 302 with a 301 redirect, or would you recommend to change the entire structure and eliminate this /content/ directory? The most logical structure would be www.domainA.com/products/.., and not www.domainA.com/content/products, right? Second thing: Given that 302 means temporary redirect, what are the actual implications when redirecting from domainA.com to domainA.com/content? I've heard that 302 redirects don't pass linkjuice and are detrimental for the site's rankings... What are the actual implications concerning the example above (302 redirect from domainA.com to domainA.com/content ? Would be great to get some advice about the first problem and maybe some insights about the second one concerning 302s in general. Thanks in advance! Cheers, Chris
Technical SEO | | adwordize0
