Blocked URL parameters can still be crawled and indexed by google?
-
Hy guys,
I have two questions and one might be a dumb question but there it goes. I just want to be sure that I understand:
IF I tell webmaster tools to ignore an URL Parameter, will google still index and rank my url?
IS it ok if I don't append in the url structure the brand filter?, will I still rank for that brand?
Thanks,
PS: ok 3 questions :)...
-
If you want to permanently remove URLs from the index, this is the basic process:
Have your developer implement NoIndex, Follow to all pages that have the URL parameter you want removed. For example, if the URL contains categoryFilter= (like above), then add the NoIndex, Follow tag to the of the page. Do this for all URL paramters you want removed from the index.
Make sure Google is allowed to crawl those pages. If they are blocked by robots.txt or told not to crawl them via Google Webmaster Tools, Google will not be able to see the newly implement NoIndex, Follow tag.
Then, give it some time and wait. It may take Google a long time to crawl all of these paramtered URLs again. Fallout of the index might be slow.
Once the URLs are gone, consider blocking the crawling of them via robots.txt or in GWT parameter handling.
-
Hi Anthony,
What if we are trying to permanently remove e-commerce website URL's that have multiple parameters from (Google) index. How would we apply noindex to all these URL's with parameters??
The aim is to recrawl and rebuild the index of the whole website using appropriate robots, canonical's & meta-tags, rather than using GWT.
Many thanks
-
Parameter handling in Google Webmaster Tools won't get a URL out of the index if it is already indexed.
You need to use the NoIndex robots meta tag in the of your page. Once you add this tag, be sure you are allowing Google to crawl the page. Make sure it is Not blocked via robots.txt or with Parameter handling.
Once the pages have left the index, you can block them from being crawled.
-
If you want a page or url not crawled then you should use the robots.txt file and robots meta tags. Then, in WMT, make sure those same pages are actually not being crawled
Hope that answers your question
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Weird, long URLS returning crawl error
Hi everyone, I'm getting a crawl error "URL too long" for some really strange urls that I'm not sure where they are being generated from or how to resolve it. It's all with one page, our request info. Here are some examples: http://studyabroad.bridge.edu/request-info/?program=request info > ?program=request info > ?program=request info > ?program=request info > ?program=programs > ?country=country?type=internships&term=short%25 http://studyabroad.bridge.edu/request-info/?program=request info > ?program=blog > notes from the field tefl student elaina h in chile > ?utm_source=newsletter&utm_medium=article&utm_campaign=notes%2Bfrom%2Bthe%2Bf Has anyone seen anything like this before or have an idea of what may be causing it? Thanks so much!
Technical SEO | | Bridge_Education_Group0 -
Will Google Recrawl an Indexed URL Which is No Longer Internally Linked?
We accidentally introduced Google to our incomplete site. The end result: thousands of pages indexed which return nothing but a "Sorry, no results" page. I know there are many ways to go about this, but the sheer number of pages makes it frustrating. Ideally, in the interim, I'd love to 404 the offending pages and allow Google to recrawl them, realize they're dead, and begin removing them from the index. Unfortunately, we've removed the initial internal links that lead to this premature indexation from our site. So my question is, will Google revisit these pages based on their own records (as in, this page is indexed, let's go check it out again!), or will they only revisit them by following along a current site structure? We are signed up with WMT if that helps.
Technical SEO | | kirmeliux0 -
Affiliate Link is Trumping Homepage - URL parameter handling?
An odd and slightly scary thing happened today: we saw an affiliate string version of our homepage ranking number one for our brand, along with the normal full set of site-links. We have done the following: 1. Added this to our robots.txt : User-agent: *
Technical SEO | | LawrenceNeal
Disallow: /*? 2. Reinserted a canonical on the homepage (we had removed this when we implemented hreflang as had read the two interfered with each other. We haven't had canonical for a long time now without issue. Is this anything to do with the algo update perhaps?! The third thing we're reviewing I'm slightly confused about: URL Parameter Handling in GWT. As advised - with regard to affiliate strings - to the question: "Does this parameter change page content seen by the user?" We have NO selected, which means they should be crawling one representative URL. But isn't it the case that we don't want them crawling or indexing ANY affiliate URLs? You can specify Googlebot to not crawl any of particular string, but only if you select: "Yes. The parameter changes the page content." Should they know an affiliate URL from the original and not index them? I read a quote from Matt Cutts which suggested this (along with putting a "nofollow" tag in affiliate links just in case) Any advice in this area would be appreciated. Thanks.0 -
What to do about Google Crawl Error due to Faceted Navigation?
We are getting many crawl errors listed in Google webmaster tools. We use some faceted navigation with several variables. Google sees these as "500" response code. It looks like Google is truncating the url. Can we tell Google not to crawl these search results using part of the url ("sort=" for example)? Is there a better way to solve this?
Technical SEO | | EugeneF0 -
Mobile Google Not Indexing Mobile Website
Google currently does not index our mobile website. It has the WWW website in it's index. When a user from a mobile phone clicks on a mobile search result for WWW we redirect them to our mobile website. This is posing problems for us as our mobile website is a fraction of the # of pages/sections as our WWW. So for example, mobile search results show that we have a "careers" section; but that's not the case for the mobile website. As a result a user gets a 404. How do we force mobile Google to index our mobile website instead of our WWW?
Technical SEO | | RBA0 -
How can I see if my website was penalize by Google?
Hello, I have a website http://digitaldiscovery.eu that I have been working for 7 months. Everything is alright in the index of the search engines like Google, Bing e Yahoo. I also have like 1000 visits a month wich is not bad for the topic Im pointing at in my country. However my pagerank insist to be on 0, and I really dont understand why. Some of the my competitors that started at the same time, already have a pagerank of 3 and they do not have the same visitors that I do. In the rank system of Alexa im climbing very fast and the visits of my website are growing. So why does the pagerank dont climb aswell?! Tks in advance, Pedro M Pereira
Technical SEO | | PedroM0 -
Google Indexed URLs for Terms Have Changed Causing Huge SERP Drop
We haven't made any significant changes to our website, however the pages that google has indexed for our critical keywords have changed to pages that have caused our SERP to drop dramatically for those pages. In some cases, the changes make no sense at all. For example, one of our terms that used to be indexed to our homepage is now indexed to a dead category page that has nothing on it. One of our biggest terms, where we were 9th, changed and is now indexed to our FAQ. As a result, we now rank 44th. This is having a MAJOR impact on our business so any help on why this sudden change happened and what we can do to combat it is greatly appreciated.
Technical SEO | | EvergladesDirect0 -
Are Google now indexing iFrames?
A client is pulling content through an iFrame, and when searching for a snippet of that exact content the page that is pulling the data is being indexed and not the iFrame page. Seen this before?
Technical SEO | | White.net0