Question about robots.txt
-
Solved!
-
Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.
-
Consider deleting all of this:
Disallow: /&limit
Disallow: /?limit
Disallow: /&sort
Disallow: /?sort
Disallow: /?route=checkout/
Disallow: /?route=account/
Disallow: /?route=product/search
Disallow: /?route=affiliate/
Disallow: /?marca
Disallow: /&manufacturer
Disallow: /?manufacturer
Disallow: /?filter
Disallow: /&filter
Disallow: /?order
Disallow: /&order
Disallow: /?price
Disallow: /&price
Disallow: /?filter_tag
Disallow: /&filter_tag
Disallow: /?mode
Disallow: /&mode
Disallow: /?cat
Disallow: /&cat
Disallow: /?product_id
Disallow: /&product_id
Disallow: /?route=affiliate/
Disallow: /*?keywordThose rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
One more question about rel=canonical
I'm still trying to wrap my head around rel=canonical and its importance. Thanks to the community, I've been able to understand most of it. Still, I have a couple of very specific questions: I share certain blog posts on the Huffington Post. Here's an example: http://www.huffingtonpost.ca/cedric-lizotte/munich-travel-guide_b_13438956.html - Of course I post these on my blog as well. Here: http://www.continentscondiments.com/things-munich-classics/ - Obviously the HuffPo has a huge DA, and I'll never match it. However the original post is mine, on my blog, and not on the HuffPo. They wont - obviously - add a rel=canonical just for me and for the sake of it, they have a million other things to do. QUESTION: Should I add a rel=canonical to my own site pointing to the post on the HuffPost? What would be the advantage? Should I just leave this alone? I share blog posts on Go4TravelBlog too. Example: http://www.go4travelblog.com/dallmayr-restaurant-munich/ - but, once again, the original post is on one of my blogs. In this case, it's on another blog of mine: http://www.thefinediningblog.com/dallmayr-restaurant-in-munich/ QUESTION: Well it's pretty much the same! Should I beg Go4TravelBlog to add a rel=canonical pointing to mine? If they refuse, what do I do? Would it be better to add a rel=canonical from my site to theirs, or do I fight it out and have a rel=canonical pointing to my own post? Why? Thanks a million for your help!
On-Page Optimization | | cedriklizotte0 -
Image Titles and Descriptions Question
Hello, I have a question about optimizing the SEO on my pages through image titles and descriptions. There are a few times on my website that I use the same image on multiple pages. I am under the impression that giving it a title such as "social-media-marketing-agency-graphic.jpg" will help the SEO for the phrase "social media marketing agency" on that page. My question was, if I want to use the same image on multiple pages, am I better off uploading an entirely new image with a new title to make it more relevant to the new page? Or will this not make large enough of a difference? Or is there an easier solution? Please let me know your thoughts on how to best optimize the pages
On-Page Optimization | | brightsocial0 -
On-Page Optimization Question
My company sells Blue widgets and we are located in Denver, CO. Keyword research indicates that the the highest volume phrase is "blue widgets for sale in denver co". Should my meta title tag be: Blue Widgets for sale in Denver CO , and my h1 tag be the same? or should they be semantic phrases? Thanks in advance!
On-Page Optimization | | FicklingCompany0 -
Can I robots.txt an entire site to get rid of Duplicate content?
I am in the process of implementing Zendesk and will have two separate Zendesk sites with the same content to serve two separate user groups (for the same product-- B2B and B2C). Zendesk does not allow me the option to changed canonicals (nor meta tags). If I robots.txt one of the Zendesk sites, will that cover me for duplicate content with Google? Is that a good option? Is there a better option. I will also have to change some of the canonicals on my site (mysite.com) to use the zendesk canonicals (zendesk.mysite.com) to avoid duplicate content. Will I lose ranking by changing the established page canonicals on my site go to the new subdomain (only option offered through Zendesk)? Thank you.
On-Page Optimization | | RoxBrock0 -
H1 question
is it recommended to have the page title labeled as H1? or better to have page title and separate H1 ? thank you
On-Page Optimization | | gergi0 -
Does Google Bias Against Homepages for Search Queries that are a Question?
If you are trying to rank for a keyword phrase that is a question (e.g. how to ___) does it matter if your page is the home page of your site or an article on your site? I suspect that Google would treat question search queries differently and would show preference for articles over a home page in most cases. An article would be more likely to satisfy the searcher by providing the answer to their question, whereas the home page usually doesn't provide specific answers. I looked at one keyword phrase that is a question and only 1 of the first 20 results was a home page. Any thoughts or experience with this?
On-Page Optimization | | Charlessipe0 -
Meta Description Question
Is it ok to put your domain name in the meta description if that is the name of your company? so if my company name is BlaBla.com and I want to have a meta description like Shop for your next Widget at BlaBla.com. We have Widget in 30 colors. Free shipping for orders over $50. and I am obviosuly promoting the widget page is this ok. I am concerned that Google will not like the domain name in the meta description. After Peguin I am paranoid about my shadow.....LOL
On-Page Optimization | | freestone0 -
How do you block development servers with robots.txt?
When we create client websites the urls are client.oursite.com. Google is indexing theses sites and attaching to our domain. How can we stop it with robots.txt? I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy. Thanks, TR
On-Page Optimization | | DisMedia0