404s in Google Search Console and javascript
-
The end of April, we made the switch from http to https and I was prepared for a surge in crawl errors while Google sorted out our site. However, I wasn't prepared for the surge in impossibly incorrect URLs and partial URLs that I've seen since then.
I have learned that as Googlebot grows up, he'she's now attempting to read more javascript and will occasionally try to parse out and "read" a URL in a string of javascript code where no URL is actually present. So, I've "marked as fixed" hundreds of bits like
/TRo39,
category/cig
etc., etc....But they are also returning hundreds of otherwise correct URLs with a .html extension when our CMS system generates URLs with a .uts extension like this:
https://www.thompsoncigar.com/thumbnail/CIGARS/90-RATED-CIGARS/FULL-CIGARS/9012/c/9007/pc/8335.html
when it should be:
https://www.thompsoncigar.com/thumbnail/CIGARS/90-RATED-CIGARS/FULL-CIGARS/9012/c/9007/pc/8335.utsWorst of all, when I look at them in GSC and check the "linked from" tab it shows they are linked from themselves, so I can't backtrack and find a common source of the error.
Is anyone else experiencing this? Got any suggestions on how to stop it from happening in the future? Last month it was 50 URLs, this month 150, so I can't keep creating redirects and hoping it goes away.
Thanks for any and all suggestions!
Liz Micik -
Hi Liz,
What I would do as well is go with a solution around your robots.txt to make sure the crawlers will respect it and don't go on a hunch trying to find new URLs that are embedded somewhere else. Usually it's something you shouldn't worry about too much, it's just the crawler doing a good job trying to find more content/URLs on your site.
Martijn.
-
Google Search Console 404s can be very irritating.
What does your robots.txt file currently say?
You can remove all instances of .html in your htaccess file to help solve for this issue. More on that here: https://alexcican.com/post/how-to-remove-php-html-htm-extensions-with-htaccess/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site appearing and disappearing from google serps.
Hi, My website is normally on page 2-3 on google consistently. Over the past month it has been appearing and then completely disappearing from the serps. One day it will be on page 2, then the next day completely missing from the serps. When i check the index it seems to be indexed correctly when doing site:mysite.com. I don't understand why this keeps happening, any experience with this issue? It doesn't seem to be a google dance as far as I can tell. When my other sites dance they typically just go up or down a few ranks for a couple weeks until they stabilize. Not completely fall off the search engine.
Algorithm Updates | | Chris_www0 -
Are we confusing Google with our internal linking?
Hi all, We decided to give importance to one of our top pages as it has "keyword" in it's slug like website.com/keyword. So we internally linked even from different sub-domain pages more than homepage to rank for that "keyword". But this page didn't show up in Google results for that "keyword"; neither homepage, but our login page is ranking. We wonder why login page is ranking. Has our internal linking plan confused Google to ignore homepage to rank for that primary keyword? And generally do we need to internally link homepage more than anyother page? Thanks
Algorithm Updates | | vtmoz0 -
Your search - site:domain.com - did not match any documents.
I've recently started work on a new clients website and done some preliminary work with on-page optimisation, and there is still plenty of work to be done and issues to resolve. They are ranking ok on Bing, but they are not getting any ranking on Google at all (except paid) - I tried the site:domain.com search and comes up with no results... so this confirms that something is going on with the google search rank! Can anyone shed light on what can cause this or why this would happen? My next step is to look at their webmaster tools (haven't had access yet), but if anyone has any tips to resolve this or where to look, that would be great! Thanks!
Algorithm Updates | | ElevateCreativeAU0 -
Google Reconsideration - To do or not to do?
We haven't been manually penalized by Google yet but we have had our fair share of things needing to be fixed; malware, bad links, lack/if no content, lack-luster UX, and issues with sitemaps & redirects. Should we still submit a reconsideration even though we haven't had a direct penalty? Does hurt us to send it?
Algorithm Updates | | GoAbroadKP0 -
Google is really NOT SAYING IN "HOW SEARCH WORKS” ?
Hi All SEOmoz members and team, As I was reading this, is it true that Google does this . Simply, I don't think so, I haven't experienced any of such what is being talked [http://www.fairsearch.org/search-manipulation/what-google-isnt-saying-in-how-search-works/ C](http://www.fairsearch.org/search-manipulation/what-google-isnt-saying-in-how-search-works/ "http://www.fairsearch.org/search-manipulation/what-google-isnt-saying-in-how-search-works/")ome on, let us discuss the real thing about Google. Teginder Ravi
Algorithm Updates | | Futura0 -
Content, for the sake of the search engines
So we all know the importance of quality content for SEO; providing content for the user as opposed to the search engines. It used to be that copyrighting for SEO was treading the line between readability and keyword density, which is obviously no longer the case. So, my question is this, for a website which doesn't require a great deal of content to be successful and to fullfil the needs of the user, should we still be creating relavent content for the sake of SEO? For example, should I be creating content which is crawlable but may not actually be needed / accessed by the user, to help improve rankings? Food for thought 🙂
Algorithm Updates | | underscorelive0 -
Search Engines Traffic for New Site?
Hi, Can anyone tell me please when a new website starts receiving traffic from the search engines? Regards
Algorithm Updates | | kywebsol0 -
Why is there no compiled list of the different types of search results on Google, and what the content qualifications are to generate those results?
Seems to me that this list should exist out there somewhere, but I can't seem to find it. Am I just not as good of a Googler as I thought I was?
Algorithm Updates | | Draftfcb0