Question about robots.txt
-
Solved!
-
Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.
-
Consider deleting all of this:
Disallow: /&limit
Disallow: /?limit
Disallow: /&sort
Disallow: /?sort
Disallow: /?route=checkout/
Disallow: /?route=account/
Disallow: /?route=product/search
Disallow: /?route=affiliate/
Disallow: /?marca
Disallow: /&manufacturer
Disallow: /?manufacturer
Disallow: /?filter
Disallow: /&filter
Disallow: /?order
Disallow: /&order
Disallow: /?price
Disallow: /&price
Disallow: /?filter_tag
Disallow: /&filter_tag
Disallow: /?mode
Disallow: /&mode
Disallow: /?cat
Disallow: /&cat
Disallow: /?product_id
Disallow: /&product_id
Disallow: /?route=affiliate/
Disallow: /*?keywordThose rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question on site structure
My client is a nationwide company. They provide building maintenance services in 7 different cities. In each city they provide a different range of services. They currently have a single service page for each service and no mention on that page of the cities they offer the service. The service pages are getting no SERP visibility. We are running Paid Search and recommending SEO. I'm wondering whether it would be beneficial to build out specific service pages for each city so the content is more relevant to both users and search engines. What is best practice in this situation? Client wants to dominate SERPs in each market for the services they offer.
On-Page Optimization | | SEOinSunnyNelson0 -
Question about multiple versions of home page
Hi guys, I'm having a question, answer to which i'm unable to find anywhere. I browesed the whole internet lol. So the question is about multipple versions of the home page. In particular, i want to know how should i deal with home page URLs with this extention: /index.html All the rest possible versions of home page i know how to deal with but this one "/index.html" i don't. I did add a cononical tag to it but i'm wondering whether or not i should add 301 redirect to chosen version og the home page (let's say it is www.mydomain.com). Please advise the best practices on how to deal with this. Thanks in advance!
On-Page Optimization | | odmsoft0 -
Does anyone use Genesis Framework? If so can a newbie use it and a few other questions
Hi, So as I search the wonderful land of the internet, I see this Genesis framework brought up quite a bit. I have researched it for a few weeks, but it seems like it uses hooks instead of shortcodes. So I am curious if anyone has used it? And if so what your thoughts are about it? I am a COMPLETE newbie here, so hooks look scary. I am sure with time they will seem like second nature. They claim it has airtight security. So if you have used this framework, how is this any different than an updated stock wordpress site? I understand that vulnerabilities may be in plugins and such, but if it is really airtight, that seems great. Any thoughts are appreciated as I just want the best user experience. So many people use this framework, yet my site gets if I'm lucky 1000 views each month. It is a basic site to let people know we exist. So its not like I have a popular blog with 50,000 pageviews each month. But... going into the future, I want a pleasant and consistent user experience. Maybe a wordpress theme is all you need. Maybe a framework is more for developers. Any thoughts are greatly appreciated. Chris
On-Page Optimization | | asbchris0 -
URL Question
This url looks bad: http://www.patrickmunoz.com/#!classes/c1vw1 And when you click around the page change doesn't actually occur, it's a fade into the next page. I think this is a major problem for rankings. Although pages are crawled: https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.patrickmunoz.com%2F&oq=site%3A&aqs=chrome.2.69i57j69i58j69i59l3j69i61.3548j0j7&sourceid=chrome&espv=210&es_sm=122&ie=UTF-8 When I search for a simple page - "patrick munoz FAQs" nothing comes up:
On-Page Optimization | | tylerfraser
https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.patrickmunoz.com%2F&oq=site%3A&aqs=chrome.2.69i57j69i58j69i59l3j69i61.3548j0j7&sourceid=chrome&espv=210&es_sm=122&ie=UTF-8#q=patrick+munoz+|+FAQs Do you think this is a bad url configuration? Thanks! Tyler0 -
When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following: www.mywebsite.com/blog/title-of-post www.mywebsite.com/blog/tag/tag1 www.mywebsite.com/blog/tag/tag2 www.mywebsite.com/blog/category/categoryA etc My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
On-Page Optimization | | williammarlow0 -
Robots file include sitemap
Hello, I see that google, facebook and moz... have robots.txt include sitemap at the footer.
On-Page Optimization | | JohnHuynh
Eg: http://www.google.com.vn/robots.txt Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this? Thanks,0 -
Boatload of 301 Redirects Question
We have a client that came to us and they recently did a site makeover. Previously they had all their pages in root directory including 75+ spammy article pages. On their makeover, they moved all the article pages into a directory and added 301 redirects. In going over their site we noticed they have redundant articles, like an article on blue-marble-article.htm and blue-marbles-article.htm Playing on singular and plural with dulpicate content for most part with exception to making it plural. If they have 75 articles, Id say 1/3 are actually somewhat original content. I would like to 301 redirect 2/3's of the articles to better re-written article pages but that would add a whole lot more 301 redirects. We would then have a 301 redirect from root directory to article directory, then another 301 redirect from spam article to new re-written article. My question is, would this be too many redirects for googlebot to sort through and would it be too confusing or send bad signals? Or should I create a new directory with all good articles and just redirect the entire old articles directory to the new one? Or just delete the redirects and old spammy directory and let those fall on a 404 error page. Id hate to lose 50-75 pages but I think its in fact those spammy pages that could be why the site fell from top of first page google to third page and now 10th page in a years time. I know, Im confused just typing this out. Hope it makes sense for some good feedback and advise. Thanks.
On-Page Optimization | | anthonytjm0 -
Url question for seo
Would it be beneficial to use url that is a match for my keyword to help with seo, then have my currently url forward to that one so I don't have to change any marketing materials? I was one of the feedback that I got when doing the on page keyword optimization tool on seo moz. Thanks J
On-Page Optimization | | fertilityhealth0
