Question about robots.txt
-
Solved!
-
Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.
-
Consider deleting all of this:
Disallow: /&limit
Disallow: /?limit
Disallow: /&sort
Disallow: /?sort
Disallow: /?route=checkout/
Disallow: /?route=account/
Disallow: /?route=product/search
Disallow: /?route=affiliate/
Disallow: /?marca
Disallow: /&manufacturer
Disallow: /?manufacturer
Disallow: /?filter
Disallow: /&filter
Disallow: /?order
Disallow: /&order
Disallow: /?price
Disallow: /&price
Disallow: /?filter_tag
Disallow: /&filter_tag
Disallow: /?mode
Disallow: /&mode
Disallow: /?cat
Disallow: /&cat
Disallow: /?product_id
Disallow: /&product_id
Disallow: /?route=affiliate/
Disallow: /*?keywordThose rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 Redirects - Large .htaccess file question
We are moving about 5000 pages from root into different folders. We need to individually 301 each page because the are sitting at root level now: mysite.com/page.com We want to move them to: mysite.com/folder/page.html etc I dont think redirect match can works because of the different files names and folders they are being moved in to. Will 5000 entries in .htacess slow site loading? Any other suggestions how to handle?
On-Page Optimization | | leadforms0 -
Robots.txt Question for E-Commerce Sites
Hi All, I have a couple of e-commerce clients and have a question about URLs. When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout I'm not sure that I want these indexed. Could I be causing any harm/danger if I add this to the robots.txt file? /*? Any suggestions welcome! Gavin
On-Page Optimization | | IcanAgency0 -
Two Robots.txt files
Hi there Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one. Thanks in advance! Cheers,
On-Page Optimization | | TrulyTravel
Satla0 -
Silly question about noindex and canonical
Hi, This is probably going to sound a bit stupid, but I nevertheless want to check. We have a site that's going to have identical pages (really not my choice) for a sales reason. The two examples would be example.com/profile-name and example.com/location/profile-name Users using the onsite navigation will always end up in the latter example naturally as they have to select a location before viewing content (plus having the location in the url is nice as there are multiple profiles across different locations that have the same name). However, it's easier to sell our services when we can offer just example.com/profile-name to users for their own marketing reasons. I'd like to make the example.com/profile-name noindex follow, and have just the example.com/location/profile-name indexed, but not sure if it would be better to implement canonical tags instead? Can anyone see any potential pitfalls of using either method or does it not really make a difference (which is what I suspect, but I'd rather look stupid than get this wrong)? Thanks!
On-Page Optimization | | LeahHutcheon0 -
Question about Multi-national Websites
I am about to work on a multi-national site and need some more information about what I should consider regarding: content keyword research anything else My biggest question is regarding content. The company would like a UK version of the site with a different URL, but plan to keep the content essentially the same, with the exception of a few minor details. In this case, would duplicate content still be an issue? If so, any suggestions for working around this? Any strategy information on multi-national sites would be really helpful. Thank you! Erin
On-Page Optimization | | HiddenPeak0 -
Quick question about bold italics keywords in today's SEO world
Hello guyz do you think that , **or **tags still help you in ranking better for some keyword or this method has become obsolete?****
On-Page Optimization | | ksbnok0 -
Disallow a spammed sub-page from robots.txt
Hi, I have a sub-page on my website with a lot of spam links pointing on it. I was wondering if Google will ignore that spam links on my site if i go and hide this page using the robots.txt Does that will get me out of Google's randar on that page or its useless?
On-Page Optimization | | Lakiscy0 -
Duplicate Page Content Question
This article was published on fastcompany.com on March 19th. http://www.fastcompany.com/magazine/164/designing-facebook It did not receive much traffic, so it was re-posted on Co.Design today (March 27th) where it has received significantly more traffic. http://www.fastcodesign.com/1669366/facebook-agrees-the-secret-to-its-future-success-is-design My question is if google will dock us for reprinting/reusing content on another site (even if it is a sister site within the same company). If they do frown on that, is there a proper way to attribute the content to the source material/site (fastcompany.com)?
On-Page Optimization | | DanAsadorian0