Huge number of crawl anomalies and 404s - non- existent urls
-
Hi there,
Our site was redesigned at the end of January 2020. Since the new site was launched we have seen a big drop in impressions (50-60%) and also a big drop in total and organic traffic (again 50-60%) when compared to the old site.
I know in the current climate some businesses will see a drop in traffic, however we are a tech business and some of our core search terms have increased in search volume as a result of remote-working.
According to search console there are 82k urls excluded from coverage - the majority of these are classed as 'crawl anomaly' and there are 250+ 404's - almost all of the urls are non-existent, they have our root domain with a string of random characters on the end. Here are a couple of examples:
root.domain.com/96jumblestorebb42a1c2320800306682
root.domain.com/01sportsplazac9a3c52miz-63jth601
root.domain.com/39autoparts-agency26be7ff420582220
root.domain.com/05open-kitchenaf69a7a29510363
Is this a cause for concern? I'm thinking that all of these random fake urls could be preventing genuine pages from being indexed / or they could be having an impact on our search visibility. Can somebody advise please?
Thanks!
-
Unlikely, as long as they're returning 404 errors you should be OK. Maybe update your disavow file and you should be good to go!
-
Thanks for your reply.
I’m new to the business and I’ve found that that the old website had a spam attack, all of these fake urls are from the old pages (as they have 301s).
There are 82,000 crawl anomalies from these fake/spam URLs and around 200 404s. None of the fake /spam urls have been indexed. Could this be having a negative effect of search visibility/DA or rankings?
Thanks!
-
It's tough to say without seeing the site. Overall it's unlikely if you don't use that string anywhere. We usually see it more for broken relative URLs. Maybe a third party site is using that string.
-
Thanks for your reply, would broken urls from the internal linking structure explain the random characters? e.g. root.domain.com/96jumblestorebb42a1c2320800306682
We've never had any page content/urls relating to 'jumblestore'.
Thanks!
-
From what I can tell, this probably isn't the reasons for the drops. I'd go back and ensure that any URLs that changed are 301 redirecting to the correct destination URL. I'd also ensure that no pages that were associated with high volume keywords no longer exist.
For your issue, Google is likely finding some broken URLs, possibly from your internal linking structure. Perform a crawl of the site and see if you can find "Inlinks" to those broken pages. If so, you can work with dev to eliminate the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
URL not indexed but shows in results?
We are working on a site that has a whole section that is not indexed (well a few pages are). There is also a problem where there are 2 directories that are the same content and it is the incorrect directory with the indexed URLs. The problem is if I do a search in Google to find a URL - typically location + term then I get the URL (from the wrong directory) up there in the top 5. However, do a site: for that URL and it is not indexed! What could be going on here? There is nothing in robots or the source, and GWT fetch works fine.
Technical SEO | | MickEdwards0 -
Changing URL - Ranking Disappeared?
Hi All, I named a page URL /plectrums/ within the back end framework. But then decided to change it to /personalised-plectrums/ I resubmitted a GWT sitemap and 301 redirected plectrums -> personalised-plectrums My ranking for personalised plectrums has disappeared and has not come back does anyone know why this is? Or is there something I have missed? Lewis
Technical SEO | | SO_UK0 -
My website pages are not crawled, what to do?
Hi all. I have made some changes on the website so i like to crawled them by the search engines Google especially. I have made these changes around 2 weeks ago. I have submitted my website on good bookmarking websites. Also i used a tool available in Google webmasters "Fetch as Google", Resubmitted a sitemap.xml. Still my pages are not crawled your opinion please. Thanks
Technical SEO | | lucidsoftech0 -
Help! www and non-www urls are driving me mad!
Sorry folks, I'm a very recently joined member, and after a five year gap in creating websites, I've decided to get back into the saddle and start again. Boy how things have changed! I'm soaking up all sorts of information from everywhere I can to get up to date with these changes, but I've come across this www v non-www problem in a big way. I realise there are already posts in here about this, but each time I read them, my mind seems to slip into some sort of loop that does not get anywhere. Basically, I think Google has indexed most of my pages as non-www, and only a hadful as www's. I have opened two accounts in Google Webmaster Tools for both www and non-www, and declared my preference for both accordingly. That was two days ago. As unprofessional as it may sound, I use Serif Web Plus X6, simply beacause it did the job six years ago, and it's all I know until I find and teach myself something better. My question is this - I can only create one page on X6, and yet there are two versions indexed in Google (although not all of them). I can only amend the one page that exists in X6, so how do I canonicalize two pages when there's the only version I have access to amending? Or am I miissing the point??? I hope that made sense?! I wouldn't mind, but I specified that I didn't want the site to be indexed yet with 'no follow', as it's nowhere near finished, but for some reason (probably due to placing Adsense ads on there) Google went ahead and indexed it anyway! The site is either http://www.cushioncutengagementringsstore.com or http://cushioncutengagementringsstore.com, depending on how you look at it! Any light you can shed on this would be gratefuly received! Thanks. Cem.
Technical SEO | | ConwyWebDesign0 -
How to keep a URL social equity during a URL structure/name change?
We are in the process of making significant URL name/structure change to one of our property and we want to keep the social equity (likes, share, +1, tweets) from the old to the new URL. We have been trying many different option without success. We are running our social "button" in an iframe. Thanks
Technical SEO | | OlivierChateau0 -
Can anyone help me understand why google is "Not Selecting" a large number of my webpages to include when crawling my site.
When looking through my google webmaster tools, I clicked into the advanced settings under index status and was surprised to see that google has marked around 90% of my pages on my site as "Not Selected" when crawling. Please take a look and offer any suggestions. www.luxuryhomehunt.com
Technical SEO | | Jdubin0 -
Automatic redirect to external urls
Hi all, I'm developing a dynamic qr code service.. The service works in the following way: You create an account with an associated QR CODE pointing to a url like:
Technical SEO | | raulo79
- http://domain.me/username The user can change the target of this url.. he can:
- point to an external url ( his website for example)
- point to a vCard download page
- a mobile ready webpage ( no redirection in this case)... Visiting http://domain.me/username my company logo is displayed and we redirect the visitor with a: header("Refresh: 5;URL=http://userdomain.tld"); Google is indexing many user's URLs, this is good for those users pointing to the mobile ready webpage, in this case there is no redirection, but Google is indexing many urls that redirect to an external url and I don't know how to avoid this.. I can't do an header('Location: http://www.example.com/'); because I need to display our logo after redirection.. how can I do google friendly? Sorry for my english, I hope you can undestand the problem. Best regards.
Mauro.0 -
Domain Crawl Question
We have our domain hosted by two providers - web.com for the root and godaddy for the subdomain. Why SEOMOZ is not picking up the total pages of the entire domain?
Technical SEO | | AppleCapitalGroup0