Question about robots.txt
-
Solved!
-
Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.
-
Consider deleting all of this:
Disallow: /&limit
Disallow: /?limit
Disallow: /&sort
Disallow: /?sort
Disallow: /?route=checkout/
Disallow: /?route=account/
Disallow: /?route=product/search
Disallow: /?route=affiliate/
Disallow: /?marca
Disallow: /&manufacturer
Disallow: /?manufacturer
Disallow: /?filter
Disallow: /&filter
Disallow: /?order
Disallow: /&order
Disallow: /?price
Disallow: /&price
Disallow: /?filter_tag
Disallow: /&filter_tag
Disallow: /?mode
Disallow: /&mode
Disallow: /?cat
Disallow: /&cat
Disallow: /?product_id
Disallow: /&product_id
Disallow: /?route=affiliate/
Disallow: /*?keywordThose rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Canonical question - Would this create some sort of crawler redirection loop?
What happens if a canonical link, links to the url with / but the main url does not have the / For example: rel="canonical" href="https://www.exampleURL.co.uk/"> Main URL - https://www.exampleURL.co.uk (without the /) 301 Redirect https://www.exampleURL.co.uk/ to https://www.exampleURL.co.uk Would this create some sort of crawler redirection loop?
On-Page Optimization | | Evosite10 -
Ecommerce Canonical Question
Hi all, first question (eek) Could I pick the brains of fellow users around an issue we are having with canonical urls on a magento website. At the moment we do not have these enabled as it seems to break our indexing. Cut a long story short, we have thousands of products but haven't rewritten many of the descriptions from the manufacturers yet and so have noindexed all the product pages (freeing them as we go). The goal, for now, is to pull in traffic via the filtering options we have on the site The goal, for now, is to pull in traffic via the filtering options we have on the site. For example, if you go to Dresses, there then are several filtering options which would allow you to choose a colour, shape and material - if you wished to filter that precisely. These filtering options are all crawlable and so we would then have a page that google could index for, for example, Green Lace Maxi Dress. All good there, few people search for specific products and a lot search for types of products so we are covered. To get back to the issue at hand. If we enable the canonical option on our magento plugin it will stop us from being able to target these terms. Whereas the filtering option would create domain.com/dress/green/maxi/lace with the page title of Green Lace Maxi Dress, if we enable the canonical part of the seo plugin the canonical link which would be added to the page would be - instantly removing our ability to rank for longer tail dress related searches (we are not going to compete with the big players on the premium terms, yet!). There are alternative plugins we can buy for magento to add the correct tag, however, if every page's canonical just points back it itself like this, is there really much point spending nearly $1000 on the 4 licences we would need to cover our range of sites. Is it really necessary, in this case, that we have a canonical for the product filtering? Sorry for the long post, hope it made sense. Thanks for any assistance.
On-Page Optimization | | DSCarl0 -
When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following: www.mywebsite.com/blog/title-of-post www.mywebsite.com/blog/tag/tag1 www.mywebsite.com/blog/tag/tag2 www.mywebsite.com/blog/category/categoryA etc My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
On-Page Optimization | | williammarlow0 -
Summarize your question.Images being seen as duplicate content/pages
My images suddenly are appearing in my crawl reports as duplicate content, without meta tags, this happened over night and cant figure out why.
On-Page Optimization | | RBYoung0 -
Geographic targeting general question
Hi, I've been looking at some of the Google help centre pages for Geo-targeting and it says flat out if you use for example .co.uk and Geo-target in webmaster tools US then you will not rank as well as using .net, .com and a few others they list. So if you got a UK online business and want to expand into another country then staright away your in a negative situation if you want to rank in the SE for that location.
On-Page Optimization | | activitysuper0 -
The SEOmoz crawler is being blocked by robots.txt need help
SEO moz is showing me that the robot.txt is blocking content on my site
On-Page Optimization | | CGR-Creative0 -
URL question
Hi guys, the pro campaign thing you got going is wicked, love it. I'm recieving good results with my keywords and have noticed that categories that go beyond sub/sub/sub don't do to well. So I wanna move those that do one step up which makes it go from: http://spytunes.com/practice-guitar/advanced-routine/scales/aeolian to here http://spytunes.com/practice-guitar/advanced-routine/aeolian The existing menu system that follow all these categories across the site will soon go so it won't be a user friendly problem, I will have other type of menus. But, and here is the question: Would I greatly benefit from taking the non existent menu away and just go for: http://spytunes.com/practice-guitar/aeolian while i'm at it? Or do I stick with my current structure? I guess my real question is; how much is there to flat URLs? Cheers -dan lundholm spytunes.com
On-Page Optimization | | spytunes0
