Google indexing thousands crazy search results with %25253
-
In GWT I started seeing very strange pages indexed a few weeks, and Google is no reporting over 21,000 of pages (blocked by robots.txt) with weird URLs like this:
The current robots.txt looks like this:
User-agent: *
Disallow: /wp-contentDisallow: /wp-admin
Disallow: /wp-includes
Disallow: /data
Disallow: /slideshows
Disallow: /page/*/?s=
Disallow: /?s=
Disallow: /searchThis website is running an up to date WP install with Yoast's Google Analytics and SEO plug-in. I can't point to anything specific that happened with the site when these URLs started appearing even after I modified the robots.txt.
What can be done to try and stop Google from creating and indexing these goofy URLs?
I see lots of sites having this issue when I search in Google, but no one seems to have a solution.
-
As it turns out the problem is with Yoast's Google Analytics plug-in per Yoast. However, he has not yet released a fix nor given a date for the fix as of yet. So one either needs to deal with it until fixed or switch plug-ins.
-
Hi Sha,
Well, that is a new possible lead, but unfortunately Pictage is basically worthless when it comes to any technological issues.
Hmm, is there some way I could add "noindex" tags to anything link that appears on the Proof page as they are dynamic in appearance?
Thanks,
Joe
-
Hi again Joe,
After a more detailed look at your site (which has no obvious search box available to users) I was curious as to why all of the things that you are doing on the site seem to have no effect upon the issues you are trying to resolve...and why your site is generating thousands of search queries without a search box!
This says to me "do you have control of all of the content?" ... and it appears that you are using an external service called Pictage to upload and display client portfolios.
So, are you pulling content into your site from Pictage? Is it some kind of white label add-on to your site?
If the pages from Pictage are being generated externally, then the yoast plugin cannot add the "noindex" tag to those pages...if this is the case then I would say you need to contact the Pictage help people and advise them that there is a problem they need to attend to.
Hope that helps,
Sha
-
Hi Egol,
Hmm, I have never heard of that possibility.
How can I change the resultant search URL with a Wordpress install?
Thanks.
-
Hi Sha,
I made the changes weeks ago, but more pages keep appearing which tells me Google is still trying to index them?
There is already an "s" parameter set in GWT, but I don't really see many options in this screen - are there some settings I'm missing?
There are also page URLs like this one, can they be blocked as well?
-
In addition to the suggestions already given... if this was my site I would change the URL of the search results page. Someone might have a robot that is tossing crap queries into your search box.
-
Hi Joe,
A couple of things:
- If you have made the change to noindex search results recently, it may take some time for the errors to disappear from GWT. If the number of pages continues to grow, then clearly the noindex is not implemented as you expect.
- You could try using the parameter handling feature in GWT to tell googlebot to ignore all pages with the parameter in question. In your search string, the ? says "here come some parameters" and the "s" is the parameter that you want to ignore.
Incidentally, there is definitely something funky happening with the generation of those search strings which should be investigated and resolved as well.
Hope that helps,
Sha
-
Yoast's WordPress SEO plug-in automatically does the following:
- RSS feeds are now always noindex, followed. No search engine should ever list an RSS feed as a result in the resultpages.
- Admin, login and registration pages are always noindexed now for the same reason.
- Search result pages are now always noindex, follow.
-
This is in your own website's search, right?
I've always heard that you should do on page robots that make it:
no-index, follow
So that all of the links on the page can be followed, but Google will not index it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Stop Google indexing entire website based on search location
OK - bear with me... We have a .co.uk website. However, we only want it indexing in the US Google and NOT the UK Google. Is there a way of configuring this in Search Console /Webmaster tools?
Technical SEO | | AbsoluteDesign0 -
My sites "pages indexed by Google" have gone up more than qten-fold.
Prior to doing a little work cleaning up broken links and keyword stuffing Google only indexed 23/333 pages. I realize it may not be because of the work but now we have around 300/333. My question is is this a big deal? cheers,
Technical SEO | | Billboard20120 -
Should I be concerned about Google indexing an old domain if the listings redirect to the new domain?
I noticed this about Moz's old domain SEOMoz.org. If the URLs from the old domain are redirecting, is there any reason to be concerned about an old domain still appearing to be indexed by Google? See here: https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=site%3Aseomoz.org Links to seomoz.org are listed, but if you click them they redirect to moz.com. Is this anything to be concerned about or is everything operating as expected?
Technical SEO | | 352inc0 -
Google index graph duration in Google Webmaster Tools
Hello guys, I wonder, my sites are currently being indexed every 7 days, exactly. At Index Status page in GWT. However, this new site gets updated almost everyday, how can I ask google to index faster and more frequently/almost daily? Is it about SItemap.xml frequency ? I changed it today to Daily. Thanks!
Technical SEO | | mdmoz0 -
Why my site is not indexing in google
In google webmaster i have updated my sitemap in Mar 6th..There is around 22000 links..But google fetched only 5300 links for long time...
Technical SEO | | Rajesh.Chandran
I waited for 1 month till no improvement in google index..So apr6th we have uploaded new sitemap (1200 links totally)..,But only 4 links indexed in google ..
why google not indexing my urls? Is this affect our ranking in SERP? How many links are advisable to submit in sitemap for a website?0 -
Site being indexed by Google before it has launched
We are currently coming towards the end of migrating one of our retail sites over to magento. To our horror, we find out today that some pages are already being indexed by Google, and we have started receiving orders through new site. Do you have any suggestions for what may have caused this? Or similarly, what the best solution would be to de-index ourselves? We most recently excluded anything with a certain parameter from robots.txt - could this being implemented incorrectly have caused this issue? Thanks
Technical SEO | | Sayers0 -
Why is Google stripping/replacing my TITLE tag for the site with the BRAND Name only when looking at BRAND level search
When doing a search in Google (US Proxy) - Google is stripping and replacing my functional TITLE with the brand name only (say 'Nike'), but if you do a specific search term like ('buy nike shoes') and see a top 10 listing for my site's homepage, now the title works and shows correctly. I saw this a few years ago with another one of my company domains, but didn't ask the question as it worked out. Thanks for any insight.. NOTE: It's not damaging any results, or rankings for the site.. but: when searching for BRAND name of the company, like I explained, it's replacing a optimized title for the BRAND name, and then re-placing it naturally when deep search brings up the homepage and the TITLE looks fine.. Very weird at best! Thanks, Rob
Technical SEO | | RobMay0 -
Sitemap.xml showing up in Google Search
Hello when I do a Google search my sitemap.xml shows up for lots of queries. Does anyone have any advise on this? Should I remove url in Google Webmaster? Thanks,
Technical SEO | | Socialdude0