404s in Google Search Console and javascript
-
The end of April, we made the switch from http to https and I was prepared for a surge in crawl errors while Google sorted out our site. However, I wasn't prepared for the surge in impossibly incorrect URLs and partial URLs that I've seen since then.
I have learned that as Googlebot grows up, he'she's now attempting to read more javascript and will occasionally try to parse out and "read" a URL in a string of javascript code where no URL is actually present. So, I've "marked as fixed" hundreds of bits like
/TRo39,
category/cig
etc., etc....But they are also returning hundreds of otherwise correct URLs with a .html extension when our CMS system generates URLs with a .uts extension like this:
https://www.thompsoncigar.com/thumbnail/CIGARS/90-RATED-CIGARS/FULL-CIGARS/9012/c/9007/pc/8335.html
when it should be:
https://www.thompsoncigar.com/thumbnail/CIGARS/90-RATED-CIGARS/FULL-CIGARS/9012/c/9007/pc/8335.utsWorst of all, when I look at them in GSC and check the "linked from" tab it shows they are linked from themselves, so I can't backtrack and find a common source of the error.
Is anyone else experiencing this? Got any suggestions on how to stop it from happening in the future? Last month it was 50 URLs, this month 150, so I can't keep creating redirects and hoping it goes away.
Thanks for any and all suggestions!
Liz Micik -
Hi Liz,
What I would do as well is go with a solution around your robots.txt to make sure the crawlers will respect it and don't go on a hunch trying to find new URLs that are embedded somewhere else. Usually it's something you shouldn't worry about too much, it's just the crawler doing a good job trying to find more content/URLs on your site.
Martijn.
-
Google Search Console 404s can be very irritating.
What does your robots.txt file currently say?
You can remove all instances of .html in your htaccess file to help solve for this issue. More on that here: https://alexcican.com/post/how-to-remove-php-html-htm-extensions-with-htaccess/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google like pricing information?
Over the last year I have noticed a trend in a couple of industries. Google seems to prioritise landing pages with pricing information in the content. This seems more important than it used to. One industry is high end industrial machines. Traditionally there isn't a price list as everything is bespoke for the customer. Low end machines that display an off the shelf price are now ranking higher than they used to. This is frustrating because the different machines meet different customer requirements. However, both sorts of customers are likely to use the same search terms. Has anyone else noticed this trend?
Algorithm Updates | | Brighton-Soundsystem0 -
Google creating it own content
I am based in Australia but a US founded search on 'sciatica' shows an awesome answer on the RHS of the SERP https://www.google.com/search?q=sciatica&oq=sciatica&aqs=chrome.0.69i59.3631j0j7&sourceid=chrome&ie=UTF-8 The download on sciatica is a pdf created by google. Firstly is this common in the US? secondly any inputs on where this is heading for rollout would be appreciated. Is google now creating its own content to publish?
Algorithm Updates | | ClaytonJ0 -
Traffic drop only affecting google country domains
Hello, I have noticed that our our traffic is down by 15% (last 30 days to the 30 days before it) and I dug deeper to figure out whats going on and I am not sure I understand what is happening. Traffic from google country domains( for example google.com.sa) dropped by 90% on the 18th of September, same applies to other country specific domains. Now my other stats (visits organic keywords, search queries in WMT) seem to be normal and have seem some decrease (~5%) but nothing as drastic as the traffic drop from the google country domains. Is this an https thing that is masking the source of the traffic that came into effect on that date? Is the traffic that is now missing from google country domains being reported from other sources? Can anyone shed some light on what is going on? qk0CS7X
Algorithm Updates | | omarfk0 -
Google Maps marker inconsistency
We just discovered that depending on the address format you enter into Google, you may come across incorrectly placed marker locations on Google Maps. Is this because our Google Places address format is not consistent with Google Maps' format? If so, when I go into Google Places to update the address format, am I going to have to go through the citation process all over again?
Algorithm Updates | | SSFCU0 -
Importance of Links for Local Search
**According to an article about the "no no's for local SEO" links are not very important. Here is an excerpt: "**Local SEO is very different when compared to traditional SEO. The importance of backlinks in local SEO isn’t as important. In other words, links simply don’t matter as much when it comes to local SEO. Googles’ local search algorithm treats links completely differently than its standard algorithm." How accurate is this statement? Wouldn't more links help your local pages rank better in non-local organic results such as the results outside of the new carousel?
Algorithm Updates | | pbhatt0 -
Why Am I Ranking in Bing but Not Google
My website is ranking is ranking in Bing, but it's nowhere to be found on Google? What can be some causes for this?
Algorithm Updates | | locallyrank0 -
Did Google just give away how Penguin works?
At SMX during the You&A with Matt Cutts, Danny asked why the algo update was called Penguin. Matt said: "We thought the codename actually might give too much info about how it works so the lead engineer got to choose." Last night Google released their 39 updates for the month of May. Among them was this: "Improvements to Penguin. [launch codename "twref2", project codename "Page Quality"] This month we rolled out a couple minor tweaks to improve signals and refresh the data used by the penguin algorithm." Whoa, codename twref2 for Penguin improvement? Is this giving us an insight about how it works? I would guess the ref2 means second refresh perhaps. But tw I am not sure about. What do you think? Is there a hidden insight here?
Algorithm Updates | | DanDeceuster1 -
Are you getting any action from Google +1 ?
If you have added google plus one to your website you can check on the impact by visiting your webmaster tools account. In your GWT account you will see a left menu item for "+1 Metrics". If you click on "Search Impact" you can see the CTR change attributed to +1. Anybody seeing anything there yet?
Algorithm Updates | | EGOL0