404s in Google Search Console and javascript
-
The end of April, we made the switch from http to https and I was prepared for a surge in crawl errors while Google sorted out our site. However, I wasn't prepared for the surge in impossibly incorrect URLs and partial URLs that I've seen since then.
I have learned that as Googlebot grows up, he'she's now attempting to read more javascript and will occasionally try to parse out and "read" a URL in a string of javascript code where no URL is actually present. So, I've "marked as fixed" hundreds of bits like
/TRo39,
category/cig
etc., etc....But they are also returning hundreds of otherwise correct URLs with a .html extension when our CMS system generates URLs with a .uts extension like this:
https://www.thompsoncigar.com/thumbnail/CIGARS/90-RATED-CIGARS/FULL-CIGARS/9012/c/9007/pc/8335.html
when it should be:
https://www.thompsoncigar.com/thumbnail/CIGARS/90-RATED-CIGARS/FULL-CIGARS/9012/c/9007/pc/8335.utsWorst of all, when I look at them in GSC and check the "linked from" tab it shows they are linked from themselves, so I can't backtrack and find a common source of the error.
Is anyone else experiencing this? Got any suggestions on how to stop it from happening in the future? Last month it was 50 URLs, this month 150, so I can't keep creating redirects and hoping it goes away.
Thanks for any and all suggestions!
Liz Micik -
Hi Liz,
What I would do as well is go with a solution around your robots.txt to make sure the crawlers will respect it and don't go on a hunch trying to find new URLs that are embedded somewhere else. Usually it's something you shouldn't worry about too much, it's just the crawler doing a good job trying to find more content/URLs on your site.
Martijn.
-
Google Search Console 404s can be very irritating.
What does your robots.txt file currently say?
You can remove all instances of .html in your htaccess file to help solve for this issue. More on that here: https://alexcican.com/post/how-to-remove-php-html-htm-extensions-with-htaccess/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New Featured Links in Organic Search Results?
Hi guys, I just performed a search and came across something that looks like "featured links" under a regular organic search result (see screenshot). This is the first time I'm seeing this. It looks like a combination of callout and sitelink ad extensions for Google ads. Basically, linked callouts. I went to the landing page to check out the source code and it seems like they are calling it "featured link" in their code. I tried to find more info online but wasn't able to find anything. (I might not be using the correct search terms.) Does anyone know how to take advantage of this? Thanks a lot for your feedback. dJ9dmTr
Algorithm Updates | | HinterP0 -
Sudden Drop in Organic Traffic through Image Search
We've have been facing a strange issue with our Organic Image Search Traffic since July month, 10th July 2016 to be precise. There is a significant decline in our Organic Image Search Traffic that we can see in our Google Search Console Account, We have noticed a sudden drop (Almost 80%-90%) in our daily clicks through image search in Google Search Console, Does anyone here have any idea why this has happened suddenly though everything is same as it was before and we haven't done any changes in image names and images path. IWPbQ
Algorithm Updates | | tigersohelll0 -
How long for google to de-index old pages on my site?
I launched my redesigned website 4 days ago. I submitted a new site map, as well as submitted it to index in search console (google webmasters). I see that when I google my site, My new open graph settings are coming up correct. Still, a lot of my old site pages are definitely still indexed within google. How long will it take for google to drop off or "de-index" my old pages? Due to the way I restructured my website, a lot of the items are no longer available on my site. This is on purpose. I'm a graphic designer, and with the new change, I removed many old portfolio items, as well as any references to web design since I will no longer offering that service. My site is the following:
Algorithm Updates | | rubennunez
http://studio35design.com0 -
Googlebot soon to be executing javascript - Should I change my robots.txt?
This question came to mind as I was pursuing an unrelated issue and reviewing a site's robots/txt file. Currently this is a line item in the file: Disallow: https://* According to a recent post in the Google Webmasters Central Blog: [http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") Googlebot is getting much closer to being able to properly render javascript. Pardon some ignorance on my part because I am not a developer, but wouldn't this require Googlebot be able to execute javascript? If so, I am concerned that disallowing Googlebot from the https:// versions of our pages could interfere with crawling and indexation because as soon as an end-user clicks the "checkout" button on our view cart page, everything on the site flips to https:// - If this were disallowed then would Googlebot stop crawling at that point and simply leave because all pages were now https:// ??? Or am I just waaayyyy over thinking it?...wouldn't be the first time! Thanks all! [](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better")
Algorithm Updates | | danatanseo0 -
Our root domain is no longer appearing in search results
Hi all The root domain for our site, roadtrippers.com, has been disappearing from Google's search results. Subfolders and subdomains still appear, but our root domain isn't found at all. I believe I've verified this by searching "-inurl:trips -inurl:byways -inurl:support -inurl:blog -inurl:places -inurl:guides -inurl:destinations site:https://roadtrippers.com/" in Google and our root domain is nowhere to be found. This may or may not be related to another issue we've had, where the root domain is appearing with a seemingly rotating set of parameters. Sometimes it'll be ?mod=, sometimes it'll be ?tag=translation. Originally they appeared to simply displace our ranking root domain, but now they and our root domain are completely disappearing. Our dev team believes they fixed the problem with recent 301 tags to any unapproved parameter being added to the root domain, but this hasn't fixed the original problem. Any insight into this is greatly appreciated! Brandon
Algorithm Updates | | brandonRT0 -
How Do I Optimize with Google's Video Search?
Hi everyone, I am looking here https://developers.google.com/webmasters/videosearch/schema and I don't fully understand. Could someone please explain, step by step, what I have to do to optimize for Google video search? I.e. Step 1 do this Step 2 do this. I don't fully understand Thank you!
Algorithm Updates | | jhinchcliffe0 -
Site not in Google top 50 for key terms
Dear Moz Community, Our site - http://www.sportsdirectnews.com publishes a high volume of daily sport stories and aims to follow Google's Webmaster Guidelines, yet our pages don't appear anywhere in Google's SERP's. We've looked in details at the issue and think it could be something to do with: a) Unusual links or b) High page loading time or c) Too many on-page links If you could have a look at the site - http://www.sportsdirectnews.com - and give your professional opinion as to why our website is not appearing in SERP's, we would be most appreciative. SDN
Algorithm Updates | | BoomDialogue690 -
Google showing different pages for same search term in uk and usa
Hi Guys, I have an interesting question and think Google is being a bit strange.. Can anyone tell me why when I input the term design agency in Google.co.uk it shows one page, but when i tyupe in the same search term in Google.com (worldwide search) it shows another page.. Any ideas guys? Is this not bit strange?? Any help here be much appreciated.. Thanks Gareth
Algorithm Updates | | GAZ090
