Website blog is hacked. Whats the best practice to remove bad urls
-
Hello
So our site was hacked which created a few thousand spam URLs on our domain. We fixed the issue and changed all the spam urls now return 404. Google index shows a couple of thousand bad URLs.
My question is-
What's the fastest way to remove the URLs from google index. I created a site map with sof the bad urls and submitted to Google. I am hoping google will index them as they are in the sitemap and remove from the index, as they return 404.
Any tools to get a full list of google index? ( search console downloads are limited to 1000 urls). A Moz site crawl gives larger list which includes URLs not in Google index too. Looking for a tool that can download results from a site: search.
Any way to remove the URLs from the index in bulk? Removing them one by one will take forever.
Any help or insight would be very appreciated.
-
Technically 404 means "temporarily unavailable but coming back later" so you might want to consider Status 410 instead of 404. You could also supplement it with Meta no-index, if you can't use the HTML implementation then fire the no-index directive through the HTTP header using X-robots:
https://developers.google.com/search/reference/robots_meta_tag (scroll down a little to find the relevant part)
E.g:
"HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)"... something like that.
You can't use Search Console to remove URLs from Google at all. The remove URL tool, only removes URLs one at a time and it only does so 'temporarily', the URLs pop back again after a bit. The best thing you can do is give Google some harsher directives and hope they listen, in a month or two most of those should be gone
Don't use robots.txt on the URLs as, if Google can't crawl them it won't find the 410s or the no-index directives
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best way to redirect friendly URL in direct mail ?
Hi, When we do direct mail to our customers talking about a specific product we sell we usually put a link in the letter so the customer can go directly to the product just by typing a short link, something like:
Technical SEO | | BigJoe
www.example.com/blue-widget This link will then re-direct to:
www.example.com/shop/product/brand-name-big-blue-widget-with-green-ends-200m-50diameter.php Which we are happy with at the moment but I want to check we are doing it correctly in terms of redirects, we currently re-direct it using .htaccess like:
Redirect /blue-widget http://www.example.com/shop/product/brand-name-big-blue-widget-with-green-ends-200m-50diameter.php This re-directs it as a 302 but should it be done as a 301 ? I am not sure why we did 302's to start with but I am thinking they should be 301's, I think it might have been because the URL we were redirecting from was imaginary ? Also should we use the Redirect line in the .htaccess or should we do each one with a RewriteRule ? Thanks BigJoe0 -
Should I Remove Thousands of Bad Links over a Short Time or Long Time?
Hey Moz Community! I've got a website that has hundreds of thousands of old links that don't really offer any great content. They need to be removed. Would it be a better idea to remove them in batches of 5000,10000, or more over a long time... or remove them all at the same time because it doesn't matter? Cheers, Alex
Technical SEO | | Anti-Alex0 -
Company blog. What are the best solutions?
Hello Moz Community! Our company has its own blog (www.awarablogs.com) - the blog was created some time ago by means of a simple blog-engine. Now we see that the structure of the blog is bad for SEO (it has long URLs, many useless folders, subdomains and so on), so we'd like to simplify it. But the engine doesn't allow to change its structure in the way we 'd like to. Our webmaster suggested that we use "Alias". Will this method really help us make our blog SEO-friendly? Or is it better to choose another blog software like Wordpress? Thank you very much!
Technical SEO | | Awaraman0 -
Best Google Practice for Hacked SIte: Shift Servers/IP or Disavow?
Hi - Over the past few months, I've identified multiple sites which are linking into my site and creating fake pages (below is an example and there's over 500K+ of similar links from various sites}. I've attempted to contact the hosting companies, etc. with little success. Was wondering if my best course of action might be at this point: A) which servers (or IP address). B) Use the Google Disavow tool? C) both. example: { http://aryafar.com/crossings/200-krsn-team-part19.html } Thanks!!
Technical SEO | | hhdentist0 -
Removing Redirected URLs from XML Sitemap
If I'm updating a URL and 301 redirecting the old URL to the new URL, Google recommends I remove the old URL from our XML sitemap and add the new URL. That makes sense. However, can anyone speak to how Google transfers the ranking value (link value) from the old URL to the new URL? My suspicion is this happens outside the sitemap. If Google already has the old URL indexed, the next time it crawls that URL, Googlebot discovers the 301 redirect and that starts the process of URL value transfer. I guess my question revolves around whether removing the old URL (or the timing of the removal) from the sitemap can impact Googlebot's transfer of the old URL value to the new URL.
Technical SEO | | RyanOD0 -
Best Practices to Choosing a Domain Name
I have the following list of domains to choose from: http://www.xxx.net/ http://www.xxx.uk/ www.es-xxx.com Which of these domain structures seem the best, or are all 3 questionable?
Technical SEO | | theLotter0 -
Shorter URLs
Hi Is there a real value in having the keywords in the URL structure? we could use the URL: Mybrand.com/software/tablets/ipad/supertrader.html Or instead have the CMS create the shorter version mybrand.com/supertrader.html and just optimize this page for the keyword 'supertrader ipad software'
Technical SEO | | FXDD1 -
Whats the best tools for site architecture
Look for tools that can visualise a sites architecture (idealy automated). Also looking for tools that can visualise internal linking sturures
Technical SEO | | Motionlab0