Good robots txt for magento
-
Dear Communtiy,
I am trying to improve the SEO ratings for my website www.rijwielcashencarry.nl (magento). My next step will be implementing robots txt to exclude some crawling pages.
Does anybody have a good magento robots txt for me? And what need i copy exactly?Thanks everybody!
Greetings,
Bob
-
This is fine, as long as you don't want to exclude robots from crawling any part of your site.
-
Me to have this problem, if someone can help with setting root.txt
my webcurrent configuration is
Sitemap: http://www.myweb/sitemap.xml
User-agent: *
Disallow:THIS IS GOOD ?
-
Hi Ruth,
Also thanks for your response!
Greetings,
Bob
-
Hi Peter,
Thanks for your response! I am going to follow up your advice and build a good Robots TXT.
Greetings,
Bob
-
Peter is correct - your search, admin and user pages are common pages to block for Magento. What you block is up to you, though. Don't forget that a page that is blocked by robots.txt can still be found by search engines, so if it's a page that will contain private information you should protect it with a password.
-
Hi there! Did Peter's response take care of this for you? If so, please mark it as a "Good Answer."
-
Hi,
Creating robots.txt file for the site is one of the most important thing, you need to understand your website or stores basic needs what to keep private and what to make public; I think you need to block some part in your magento site like your search pages (?*sid) and admin pages, user dashboard pages, here is an example links Robots.txt for Magento and Robots.txt File Examples
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New Magento store, is better to place it in a new url or it can work fine in a subdomain?
Hi friends, We are working on a new Magento store for one of our websites. Our strategy is for organic positioning of the products, so we need to understand if the Magento products will position better if the system is hosted in a subdomain of the main company domain or if it is better to host it under its own domain. Thanks,
Technical SEO | | FWC_SEO0 -
How to solve the meta : A description for this result is not available because this site's robots.txt. ?
Hi, I have many URL for commercialization that redirects 301 to an actual page of my companies' site. My URL provider say that the load for those request by bots are too much, they put robots text on the redirection server ! Strange or not? Now I have a this META description on all my URL captains that redirect 301 : A description for this result is not available because this site's robots.txt. If you have the perfect solutions could you share it with me ? Thank You.
Technical SEO | | Vale70 -
Can anyone show me a good example of Microdata for a site
Hi, i am just learning about microdata after being told that it would be a good idea to have this for my site www.in2town.co.uk as i understand it, if i put this in, then it will allow google to understand more about my page I have been reading about it here https://support.google.com/webmasters/bin/answer.py?hl=en&answer=99170 and would like to know if anyone is using it and if it is making a difference, also can anyone show me some good examples on a working website please i work in joomla so as i understand it, i would have to put this in each page to help google understand more about the page and its content any help would be great.
Technical SEO | | ClaireH-1848860 -
Robots.txt file
How do i get Google to stop indexing my old pages and start indexing my new pages even months down the line? Do i need to install a Robots.txt file on each page?
Technical SEO | | gimes0 -
Confirming Robots.txt code deep Directories
Just want to make sure I understand exactly what I am doing If I place this in my Robots.txt Disallow: /root/this/that By doing this I want to make sure that I am ONLY blocking the directory /that/ and anything in front of that. I want to make sure that /root/this/ still stays in the index, its just the that directory I want gone. Am I correct in understanding this?
Technical SEO | | cbielich0 -
How to put news on your site from other sources and is it a good idea
Hi i have seen this site and have been watching it for some time and really like it. I would like to put news on my site for one topic and would like it displayed like this site does but i am not sure how they do it. here is the site http://www.topix.com/uk/grimsby I use joomla so i am not sure you can do exactly what they do where they have so much of the story before being sent over to the main site. Also i would like to know if this is a good idea or a bad idea as i am thinking this could take away my juice from my site any advice would be great.
Technical SEO | | ClaireH-1848860 -
Robots.txt usage
Hey Guys, I am about make an important improvement to our site's robots.txt we have large number of properties on our site and we have different views for them. List, gallery and map view. By default list view shows up and user can navigate through gallery view. We donot want gallery pages to get indexed and want to save our crawl budget for more important pages. this is one example of our site: http://www.holiday-rentals.co.uk/France/r31.htm When you click on "gallery view" URL of this site will remain same in your address bar: but when you mouse over the "gallery view" tab it will show you URL with parameter "view=g". there are number of parameters: "view=g, view=l and view=m". http://www.holiday-rentals.co.uk/France/r31.htm?view=l http://www.holiday-rentals.co.uk/France/r31.htm?view=g http://www.holiday-rentals.co.uk/France/r31.htm?view=m Now my question is: I If restrict bots by adding "Disallow: ?view=" in our robots.txt will it effect the list view too? Will be very thankful if yo look into this for us. Many thanks Hassan I will test this on some other site within our network too before putting it to important one's. to measure the impact but will be waiting for your recommendations. Thanks
Technical SEO | | holidayseo0 -
My website pages (www.ommrudraksha.com) is getting good rank slowly. But no good sales ?
My website has been doing good slowly. I have been using seomoz recommendations. And it is a great help to see that my pages are slowly coming to the first page. I am also running PPC on google. I see there are many visitors to my website. But i do not get good conversion - or not getting customer buying products. My website : www.ommrudraksha.com My target keyword is : rudraksha
Technical SEO | | Ommrudraksha0