404 Errors for Form Generated Pages - No index, no follow or 301 redirect
-
Hi there
I wonder if someone can help me out and provide the best solution for a problem with form generated pages.
I have blocked the search results pages from being indexed by using the 'no index' tag, and I wondered if I should take this approach for the following pages. I have seen a huge increase in 404 errors since the new site structure and forms being filled in. This is because every time a form is filled in, this generates a new page, which only Google Search Console is reporting as a 404.
Whilst some 404's can be explained and resolved, I wondered what is best to prevent Google from crawling these pages, like this: mydomain.com/webapp/wcs/stores/servlet/TopCategoriesDisplay?langId=-1&storeId=90&catalogId=1008&homePage=Y
Implement 301 redirect using rules, which will mean that all these pages will redirect to the homepage. Whilst in theory this will protect any linked to pages, it does not resolve this issue of why GSC is recording as 404's in the first place. Also could come across to Google as 100,000+ redirected links, which might look spammy.
Place No index tag on these pages too, so they will not get picked up, in the same way the search result pages are not being indexed.
Block in robots - this will prevent any 'result' pages being crawled, which will improve the crawl time currently being taken up. However, I'm not entirely sure if the block will be possible? I would need to block anything after the domain/webapp/wcs/stores/servlet/TopCategoriesDisplay?. Hopefully this is possible?
The no index tag will take time to set up, as needs to be scheduled in with development team, but the robots.txt will be an quicker fix as this can be done in GSC.
I really appreciate any feedback on this one.
Many thanks
-
Hi there
I wonder if you would be able to still help. The number of 404's is increasing significantly and the majority only appear in GSC. The reason I think this could be search URL related is these are increasing significantly every day.
The robots.txt has blocked some, but as the number continues to increase I am thinking there could be a few reasons, which I need to look into more.
A siteliner report cannot crawl the site due to 'too many redirections for this URL'. This is one reason why I suspect there is a wider issue to investigate with the https http.
Moz and Screaming Frog are recording some errors (which we expected and need to resolve) but in the 100's, compared to the 1000's recorded in GSC.
Any other ideas / suggestions would be appreciated.
Many thanks
-
Hi Ric,
That makes sense, so do these pages result in a non-404 from a search, but direct traffic would result in a 404? Or are these 404's only appearing in GSC?
Did the robots.txt blocking work out? Are any of these URL's mentioned in the sitemap.xml? Have you tried crawling the site with a crawler like screaming frog to see if they surface in that? If they do you might need to approach your search results a different way.
-
Hi - thank you for your response. Apologies, I mean't test in GSC.
To answer your question, these are not soft 404's
Many thanks
-
Hi RIc,
I believe your first step would be blocking via robots.txt something along the lines of:
Disallow: domain/webapp/wcs/stores/servlet/TopCategoriesDisplay?*
But I think you are mistaken that you can make this change within GSC, you can test in GSC, but this doesn't change anything on your site. You will still have to reach out to a dev to get this change complete.
Out of curiosity are these 404's being marked as soft 404's?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pages not indexed
Hey everyone Despite doing the necessary checks, we have this problem that only a part of the sitemap is indexed.
Technical SEO | | conversal
We don't understand why this indexation doesn't want to take place. The major problem is that only a part of the sitemap is indexed. For a client we have several projects on the website with several subpages, but only a few of these subpages are indexed. Each project has 5 to 6 subpages. They all should be indexed. Project: https://www.brody.be/nl/nieuwbouwprojecten/nieuwbouw-eeklo/te-koop-eeklo/ Mainly subelements of the page are indexed: https://www.google.be/search?source=hp&ei=gZT1Wv2ANouX6ASC5K-4Bw&q=site%3Abrody.be%2Fnl%2Fnieuwbouwprojecten%2Fnieuwbouw-eeklo%2F&oq=site%3Abrody.be%2Fnl%2Fnieuwbouwprojecten%2Fnieuwbouw-eeklo%2F&gs_l=psy-ab.3...30.11088.0.11726.16.13.1.0.0.0.170.1112.8j3.11.0....0...1c.1.64.psy-ab..4.6.693.0..0j0i131k1.0.p6DjqM3iJY0 Do you have any idea what is going wrong here?
Thanks for your advice! Frederik
Digital marketeer at Conversal0 -
I need help with redirecting chain to another page and 301, I don't understand on how to fix
Redirect Chain <label>What it is:</label> Your page is redirecting to a page that is redirecting to a page that is redirecting to a page... and so on. Learn more about redirection best practices. <label>Why it's an issue:</label> Every redirect hop loses link equity and offers a poor user experience, which will negatively impact your rankings. <label>How to fix it:</label> Chiaryn says: “Redirect chains are often caused when multiple redirect rules pile up, such as redirecting a 'www' to non-www URL or a non-secure page to a secure/https: page. Look for any recurring chains that could be rewritten as a single rule. Be particularly careful with 301/302 chains in any combination, as the 302 in the mix could disrupt the ability of the 301 to pass link equity.” This is not helping me I don't understand about the 301 do I use the www.jasperartisanjewelry.com or the /jasperartisanjewelry.com I'm confused
Technical SEO | | geanmitch0 -
Soft 404 errors
Google webmaster tools is telling me I have 8 "soft 404's". They are all like this page...
Technical SEO | | sdwellers
http://www.seadwellers.com/search/page/8/ All 8 pages are the same except the number at the end...... I just can't figure this....any insight at all is appreciated and do i need to correct somehow?0 -
CNAME vs 301 redirect
Hi all, Recently I created a website for a new client and my next job is trying to get them higher in Google. I added them in OSE and noticed some strange backlinks. To my surprise the client has about 20 domain names. All automatically poiting to (showing) the same new mainsite now. www.maindomain.nl www.maindomain.be
Technical SEO | | Houdoe
www.maindomain.eu
www.maindomain.com
www.otherdomain.nl
www.otherdomain.com
... Some of these domains have backlinks too (but not so much). I suggested to 301 redirect them all to the main site. Just to avoid duplicate content. But now the webhoster comes into play: "It's a problem, client has only 1 hosting account, blablabla...". They told me they could CNAME the 20 domains to the main domain. Or A-record them to an IP address. This is too technical stuff for me. So my concrete questions are: Is it smart to do anything at all or am I just harming my client? The main site is ranking pretty well now. And some backlinks are from their copy sites (probably because everywhere the logo links to the full mainsite url). Does the CNAME or A-record solution has the same effect as a 301 redirect, from SEO perspective? Many thanks,
Hans0 -
Redirect them to some other related pages in your website
dear sir, i found 404 errors in my google adsense account as well google webmaster tool. how can i escape from these errors. and also tell me why can i increase google click rate. my google rate is so low.
Technical SEO | | learningall0 -
Redirecting website page to another
Hi there one of my old pages on my site is currently ranking for a phrases that I want to rank for on a new page I created. My old page from 1 year ago is ranking for 'Property Management Training' (it's a blog post dating 2011) I have cretaed a new main Page on my site and would like to rank for 'Property Management' as it's more relevant. What is the best suggestion to keep my ranking but have people go to my new page? 301 redirect old page to new page? Thanks,
Technical SEO | | daracreative1 -
301 Redirects Change?
Hi, Mozzers- I've noticed pages that are getting 301 redirected are staying out there longer. It used to be that you would implement a 301 redirect and then after a couple of months the old page would disappear out of Google's index. Over the last couple of months I've noticed pages lingering, popping up.... What gives? Thanks in advance! LHC
Technical SEO | | lhc670 -
IIS Server Load for 500 Page Level 301 Redirects
We are migrating content from 10 sub domains to our www site. On an IIS sever, what is potential server load impact, if any, for setting up 500 plus page level redirects?
Technical SEO | | DigitalMkt0