Question about robots.txt
-
Solved!
-
Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.
-
Consider deleting all of this:
Disallow: /&limit
Disallow: /?limit
Disallow: /&sort
Disallow: /?sort
Disallow: /?route=checkout/
Disallow: /?route=account/
Disallow: /?route=product/search
Disallow: /?route=affiliate/
Disallow: /?marca
Disallow: /&manufacturer
Disallow: /?manufacturer
Disallow: /?filter
Disallow: /&filter
Disallow: /?order
Disallow: /&order
Disallow: /?price
Disallow: /&price
Disallow: /?filter_tag
Disallow: /&filter_tag
Disallow: /?mode
Disallow: /&mode
Disallow: /?cat
Disallow: /&cat
Disallow: /?product_id
Disallow: /&product_id
Disallow: /?route=affiliate/
Disallow: /*?keywordThose rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Image Titles and Descriptions Question
Hello, I have a question about optimizing the SEO on my pages through image titles and descriptions. There are a few times on my website that I use the same image on multiple pages. I am under the impression that giving it a title such as "social-media-marketing-agency-graphic.jpg" will help the SEO for the phrase "social media marketing agency" on that page. My question was, if I want to use the same image on multiple pages, am I better off uploading an entirely new image with a new title to make it more relevant to the new page? Or will this not make large enough of a difference? Or is there an easier solution? Please let me know your thoughts on how to best optimize the pages
On-Page Optimization | | brightsocial0 -
Question For SEO Companies From An On-line Retailer
When employing an SEO company what should I be looking for? When I go to a companies website I look for how many keywords they have ranked on Semrush. Most of the companies that I have looked at do not rank very many. Saying most I am talking about 10-12 companies. There was one company that had a lot more than anyone else but their reviews were awful. Then I look at their clients websites to see how many keywords they have ranked and this is where I am baffled. Quite a lot of them especially retail sites with lots of products have very few keywords ranked and my belief is that the more related keywords that you have ranked the better you will do. I also look at how the thier organic traffic is doing over a number of months. what surprises me most is that companies are advertising clients websites where it looks like traffic is climbing and then all of a sudden there is a big drop. Am I missing something in order to find a company that will do what we are paying them for? Thank you in advance.
On-Page Optimization | | Palmbourne0 -
2 sites made - almost same day question
So i made 2 sites within 24 hours of each other:
On-Page Optimization | | mweidner2782
babysitters-nyc.com
and
http://nanny-nyc.com/ nanny has 530 pages index and babysitters only 10. Any idea why that would happen and what the problem would be? Thanks and happy holidays!0 -
IFRAMES - SEO Question
If 30 websites IFRAM my website, is there any SEO advantage for me? Does google register time on site, metrics on how far in to my site the user goes? I have a real estate site with listings (homes for sale). Agents want to use my listing feed on their sites via an iframe. The clients would be scrolling through my listings through an iframe. SEO advantages for me? Disadvantages for the real estate agent using my listings?
On-Page Optimization | | JML11790 -
Best practice for Meta-Robots tag in categories and author pages?
For some of our site we use Wordpress, which we really like working with. The question I have is for the categories and authors pages (and similiar pages), i.e. the one looking: http://www.domain.com/authors/. Should you or should you not use follow, noindex for meta-robots? We have a lot of categories/tags/authors which generates a lot of pages. I'm a bit worried that google won't like this and leaning towards adding the follow, noindex. But the more I read about it, the more I see people disagree. What does the community of Seomoz think?
On-Page Optimization | | Lobtec0 -
Duplicate Title question
Thanks Mozzers in advance for any insight into what I'm sure is a basic SEO question. I'm working with a resort in the great state of Maine. Their home page title reads Maine Resorts, Resorts in Maine, (company name). The site has about 400 URL's and over half of the URL's utilize the first keyword phrase of the home page title, "Maine Resorts." Predominately, I find them used on the Accommodations pages (pages that describe each room with a picture) which I would label as deeper pages and non-conversion type pages. The page titles themselves are not exact duplicates of the Home Page Title but might read something like "Maine Resorts, Company Name, Accommodation Listing." My concern is that the heavy use of "Maine Resorts" as the first phrase in over 200 plus pages might be competing against the home page and pulling the home page ranking down. Thanks for any help given!
On-Page Optimization | | hawkvt10 -
Does Google respect User-agent rules in robots.txt?
We want to use an inline linking tool (LinkSmart) to cross link between a few key content types on our online news site. LinkSmart uses a bot to establish the linking. The issue: There are millions of pages on our site that we don't want LinkSmart to spider and process for cross linking. LinkSmart suggested setting a noindex tag on the pages we don't want them to process, and that we target the rule to their specific user agent. I have concerns. We don't want to inadvertently block search engine access to those millions of pages. I've seen googlebot ignore nofollow rules set at the page level. Does it ever arbitrarily obey rules that it's been directed to ignore? Can you quantify the level of risk in setting user-agent-specific nofollow tags on pages we want search engines to crawl, but that we want LinkSmart to ignore?
On-Page Optimization | | lzhao0 -
Title Keyword Question
I'm writing up keywords for new pages on a website. There are a number of variations on the way we can say what we're looking for, and I don't want to post the specific keywords but I'll give an example using fruits. Let's say I want to optimize for Granny Smith apples, McIntosh apples, Jonathan apples, etc. Could my title be Apples - Granny Smith, McIntosh, Jonathan and my page will come up when someone searchs "Granny Smith apples" or "McIntosh apples" etc. or do the words have to be repeated in order. Obviously I will also be repeating these in the description and on the page I'm optimizing. Thanks!
On-Page Optimization | | crlana0