The use of robots.txt
-
-
Thank you Martijn. It helps indeed.
-
Hi Daniela,
I can confirm that it won't be any problem if you don't have a robots.txt file if you don't want to block any pages. For myself I find it more useful to still have a robots.txt file in there which allows search engines to crawl the complete site. But that's just my personal opinion.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Optimal use of keywords in header tag
what does optimal use of keywords in header tag actually mean given you indicate this as hurting seo factor?
Technical SEO | | Serg1550 -
I want to use a domain that has previously been forwarded elsewhere. Any considerations?
There's a domain name (we will call it A) with no domain authority that is currently forwarded to a domain with 36 DA (we will call this domain B). B has been dormant for about two years. I am getting both domains, but domain A works better for what I will be using it for. So basically, I want to swap things around so B forwards to A, instead of A forwarding to B. Any dangers here or things to consider that I may be overlooking?
Technical SEO | | CWBFriedman0 -
Bloking pages in roborts.txt that are under a redirected subdomain
Hi Everyone, I have a lot of Marketo landing pages that I don't want to show in SERP. Adding the noindex meta tag for each page will be too much, I have thousands of pages. Blocking it in roborts.txt could have been an option, BUT, the subdomain homepage is redirected to my main domain (with a 302) so I may confuse search engines ( should they follow the redirect or should they block) marketo.mydomain.com is redirected to www.mydomain.com disallow: / (I think this will be confusing with the redirect) I don't have folders, all pages are under the subdomain, so I can't block folders in Robots.txt also Would anyone had this scenario or any suggestions? I appreciate your thoughts here. Thank you Rachel
Technical SEO | | RaquelSaiz0 -
Can I Block https URLs using Host directive in robots.txt?
Hello Moz Community, Recently, I have found that Google bots has started crawling HTTPs urls of my website which is increasing the number of duplicate pages at our website. Instead of creating a separate robots.txt file for https version of my website, can I use Host directive in the robots.txt to suggest Google bots which is the original version of the website. Host: http://www.example.com I was wondering if this method will work and suggest Google bots that HTTPs URLs are the mirror of this website. Thanks for all of the great responses! Regards,
Technical SEO | | TJC.co.uk
Ramendra0 -
Should I rename URLs to use hyphens instead of underscores?
Hello. We are about to launch a redesigned and significantly expanded site that has traditionally used underscores as separators between words in its URLs. Would you recommend replacing all the underscores with hyphens? That would then require many 301 redirects to maintain any links that might be out there. Thank you!
Technical SEO | | nyc-seo0 -
What is campaign based rank tracking tool? How to use it?
I'm having difficulties with SEOmoz Rank Tracker tool. During last month, it hasn't worked properly. I was suggested to use "campaign based rank tracking tool"- I would like to learn if anyone has already used it? Thanks, Sema
Technical SEO | | WTGEvents0 -
Robots.txt blocking site or not?
Here is the robots.txt from a client site. Am I reading this right --
Technical SEO | | 540SEO
that the robots.txt is saying to ignore the entire site, but the
#'s are saying to ignore the robots.txt command? See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file To ban all spiders from the entire site uncomment the next two lines: User-Agent: * Disallow: /0 -
Is use of javascript to simplify information architecture considered cloaking?
We are considering using javascript to format URLs to simplify the navigation of the googlebot through our site, whilst presenting a larger number of links for the user to ensure content is accessible and easy to navigate from all parts of the site. In other words, the user will see all internal links, but the search engine will see only those links that form our information hierarchy. We are therefore showing the search engine different content to the user only in so far as the search engine will have a more hierarchical information architecture by virture of the fact that there will be fewer links visible to the search engine to ensure that our content is well structured and discoverable. Would this be considered cloaking by google and would we be penalised?
Technical SEO | | JohnHillman0