The use of robots.txt
-
-
Thank you Martijn. It helps indeed.
-
Hi Daniela,
I can confirm that it won't be any problem if you don't have a robots.txt file if you don't want to block any pages. For myself I find it more useful to still have a robots.txt file in there which allows search engines to crawl the complete site. But that's just my personal opinion.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
X-robots tag causing no index issues
I have an interesting problem with a site which has an x-robot tag blocking the site from being indexed, the site is in Wordpress, there are no issues with the robots.txt or at the page level, I cant find the noindex anywhere. I removed the SEO plug-in which was there and installed Yoast but it made no difference. this is the url: https://www.cotswoldflatroofing.com/ Its coming up with a HTTP error: x-robots tag noindex, nofollow, noarchive
Technical SEO | | Donsimong0 -
Unique use of nofollow tag
Love the community here. I just had a quick question about the using noindex, nofollow. We are a car dealership group that uses a website provider (cobalt). Since they provide the website they are the only ones with access to remove pages etc. We can add pages but only they can remove them. There are some pages we need to have removed but according to them they are unable to remove them, (I think the manufacture might mandate having some pages), anyway some of these pages literally have nothing on them, and there isn't really any useful content we could add to them. So we are using noindex on them to ensure that they stay out of search indices, but I am wondering if we should also use nofollow on them. If I understand nofollow correctly it just means search engines won't follow the links on the page, well for most of these pages the only links on them are the navigation, and since we don't plan on adding any content to these pages and we can't remove them should we use noindex and nofollow as a way to "remove" them from the site as much as we can?
Technical SEO | | Murdock_Auto_Group0 -
Using 302 redirect for SEO
Hello, I'm in charge of SEO for an information website on which articles are only accessible if you have a login and password. Most of the natural links we get point to our subscribers' subomain : subscribers.mywebsite.com/article1 If they follow these natural links, visitors who are not logged get redirected (302) to www.mywebsite.com/article1 on which there is an extract of the article and they can request a free test subscription to read the end of the article. My goal is to optimize SEO for the www.mywebsite.com/article1 page. Does this page benefit from the links I get to the subscribers.mywebsite.com/article1 page or are theses links lost in terms of SEO? Thanks for your help, Sylvain
Technical SEO | | Syl200 -
How ro write a robots txt file to point to your site map
Good afternoon from still wet & humid wetherby UK... I want to write a robots text file that instruct the bots to index everything and give a specific location to the sitemap. The sitemap url is:http://business.leedscityregion.gov.uk/CMSPages/GoogleSiteMap.aspx Is this correct: User-agent: *
Technical SEO | | Nightwing
Disallow:
SITEMAP: http://business.leedscityregion.gov.uk/CMSPages/GoogleSiteMap.aspx Any insight welcome 🙂0 -
Using Sitemap Generator - Good/Bad?
Hi all I recently purchased the full licence of XML Sitemap Generator (http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html) but have yet used it. The idea behind this is that I can deploy the package on each large e-commerce website I build and the sitemap will be generated as often as I set it be and the search engines will also be pinged automatically to inform them of the update. No more manual XML sitemap creation for me! Now it sounds great but I do not know enough about pinging search engines with XML sitemap updates on a regular basis and if this is a good or bad thing? Can it have any detrimental effect when the sitemap is changing (potentially) every day with new URLs for products being added to the site? Any thoughts or optinions would be greatly appreciated. Kris
Technical SEO | | yousayjump0 -
Using Schema.org: Product or Event as the schema type?
Hello, Most of you heard from the launch of the new format for microdata: Schema.org and my question is about the different types of Schema they provide. Our websites provide an overview of courses, visitors can search/filter training courses and most important: read peer reviews. Until now we formatted (the source) of those courses with the schema type "Product" because it allows us to provide search engines with metadata about reviews via the "Aggregrated Rating". Recently we updated the information about courses, to also provide start dates and locations to users, just like the schema type for: "Events". Because we would like to provide search engines also with both types of data I would like to know your opinion. Schema.org looks like not to support the Aggregated Rating for Events and vice versa for Startdates/Locations for the Product type. And combining the two Schema types also does not looks like an option because we can't put them on the same level like it should be. So what would you recommend to use for kind of schema type(s), are we able to use the 'Product' type next to the 'Event' type and so to combine them? Thanks a lot!
Technical SEO | | Martijn_Scheijbeler0 -
SeoMoz robot is not able to crawl my website.
Hi, SeoMoz robot crawls only two web pages of my website. I contacts seomoz team and they told me that the problem is because of Javascript use. What is the solution to this? Should I contact my webdesign company and ask them to remove Javascript code?
Technical SEO | | ashish2110 -
When does it make sense to use no-follow on your own domain?
Hey guys, I'm not too sure if I'm over-thinking this, but I've seen no-follow being used with SEOmoz and I'm looking to implement this myself. Most of my links point to my root domain (yes I'm working on building links to deep pages) so would it make sense to 'limit' or 'no-follow' links on my root domain so that only the most important pages are being passed link juice? Thanks
Technical SEO | | reegs0