Do you add 404 page into robot file or just add no index tag?
-
Hi,
got different opinion on this so i wanted to double check with your comment is.
We've got /404.html page and I was wondering if you would add this page to robot text so it wouldn't be indexed or would you just add no index tag? What would be the best approach?
Thanks!
-
Hello Rubix,
Saijo gave you some great advice, but I'm concerned about the fact that you have that page in the first place, and that it produces those URL parameters. It suggests to me that instead of showing a 404 error on the contact-office.aspx page (assuming that pages doesn't exist on that URL) you are redirecting the user who tries to access that URL to the /404.html page (e.g. /404.html?aspxerrorpath=/contact-office.aspx).
Typically you want the 404 http status code to show on the URL the user is trying to unsuccessfully access. In this case instead of redirecting them to your "404 page URL" you would want to show your customized 404 message (and ensure it returns a 404 status code, use this tool) on www.yourdomain.com/contact-office.aspx.
I hope this makes sense to you. If not, feel free to ask for clarification.
-
404 are OK on your site just make sure you send the proper 404 header response for the 404 page ... Google does NOT index 404 pages ( as long as it sends the 404 header response ) , so you don't need to block them via robots.txt or meta robots.
Infact GWT warns you about these if they are able to crawl the so called 404 pages that doesn't send a 404 header response , so I think its a good idea NOT to noindex them you will get the warning if something is wrong.
Google will only index your 404 if you don't do that..they call it soft 404 : https://support.google.com/webmasters/answer/181708?hl=en
worth reading : http://moz.com/learn/seo/http-status-codes
-
Thanks Martijn,
I actually want to know what would you do for the 404 page itself. It is something like:
www.mainurl.com/404.html and for some reason this started to create some other links such as
www.mainrul.com/404.html?aspxerrorpath=/contact-office.aspx
Do you think I should add 404 page and subpages to Robot.txt ?
Thanks!
-
Hi Sida,
I would add a noindex to the page and as you also will return the 404 status code this is enough data for Google to tell not to index the page itself.
Hope this answers your question.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
This one is complicated... canonicals, href lang tags and no index
Bear with me, this is complicated (I REALLY hope one of you comes along and says, no it isn't!) Scenario A client has multiple english pages, as they have a unique product offering in AUS, US, UK, NZ and also have a global site in english. Obviously there is a lot of duplicate content and they have the relevant href lang tags set-up to help Google untangle what should be ranked where. They also have rel-canonical on each page. I've set-up search console for each of the folder structures, i.e. en-us, en-gb, en-au and so on. They have an optimised page for one of their primary keywords, which ranks nowhere for this exact keyword, but this page DOES rank for 40 similar keywords. For the exact keyword, they rank 52nd, and frustratingly, it's the homepage that ranks. We know the correct page is ranking and is indexed because search console tells us so and we see the exact page appear in SERPs for the other 40 keywords. When I look at the en-us site in Search Console, it tells me that the home page is not being indexed, because a rel canonical tag is prioritising an alternative page (probably the global site) - however, the en-us homepage is showing up in rankings for a lot of their important keywords. The site has been live for 6 months and the optimised page for about 3 months. Questions 1. If search console is saying the homepage is not ranking, how is it showing up in SERPs?
Intermediate & Advanced SEO | | Algorhythm_jT
2. Why is the homepage ranking for this important keyword, when there is virtually no mention of the keyword versus the page that is almost perfect according to Moz's on-page grader?
3. Do you need href lang tags AND rel canonical on a page?
4. How long before a new page that is optimised for a keyword take to replace (and hopefully surpass) the homepage?
5. If the US is the most important market, should we guide Google to that fact using rel-canonical? Really appreciate your feedback, hivemind. Thanks0 -
My product category pages are not being indexed on google can someone help?
My website has been indexed on google and all of its pages can be found on google except for the product category pages - which are where we want our traffic heading to, so this is a big problem for us. Our website is www.skirtinguk.com And an example of a page that isn't being indexed is https://www.skirtinguk.com/product-category/mdf-skirting-board/
Intermediate & Advanced SEO | | chelseaskirtinguk0 -
Should I add no-follow tags to my widget links?
Matt Cutts recommended in a video in 2013 to add rel="nofollow" on widget links that link back to your website. Some background of my company: We're a software company for website chat. There's a 'powered by' link in our widgets that links back from our users' websites to our website. Currently these are all follow links. I checked out the links of our competitors, and it seems none of them have no follow on their widget backlinks. This, together with the fact that the video is quite old and information on this issue rather scarce, makes me doubt whether we should change our widget backlinks to no follow. Does anyone have thoughts on this?
Intermediate & Advanced SEO | | Maximuxxx0 -
Pages that 301 redirect to a 404
We are going through a website redesign that involves changing URL's for the pages on our site. Currently all our pages are in the format domain.com/example.html and we are moving to stip off the .html file extension so it would just be domain.com/example We have thousands of pages as the site deals with news so building a redirect for each individual page isn't really feasible. My plan is to have a generic rewrite rule that redirects any page that ends .html to the stripped off version of this. A problem I can see with this is that it will also redirect pages that don't exist. So for example, domain.com/non-existant-page.html would 301 to domain.com/non-existant-page which would then return a 404 status. What would the SEO repercussions be for this? Obviously if a page doesn't exist already then it shouldn't show up in the search engine indexes and shouldn't be a problem but I'm a bit worried about how old pages that currently legitimately 404 will be treated when they start to 301 redirect to a 404 instead. Not sure if there any other potential issues from this that I've missed either? Thanks!
Intermediate & Advanced SEO | | sbb0240 -
New Web Page Not Indexed
Quick question with probably a straightforward answer... We created a new page on our site 4 days ago, it was in fact a mini-site page though I don't think that makes a difference... To date, the page is not indexed and when I use 'Fetch as Google' in WT I get a 'Not Found' fetch status... I have also used the'Submit URL' in WT which seemed to work ok... We have even resorted to 'pinging' using Pinglar and Ping-O-Matic though we have done this cautiously! I know social media is probably the answer but we have been trying to hold back on that tactic as the page relates to a product that hasn't quite launched yet and we do not want to cause any issues with the vendor! That said, I think we might have to look at sharing the page socially unless anyone has any other ideas? Many thanks Andy
Intermediate & Advanced SEO | | TomKing0 -
Are 17000+ Not Found (404) Pages OK?
Very soon, our website will go a rapid change which would result in us removing 95% or more old pages (Right now, our site has around 18000 pages indexed). It's changing into something different (B2B from B2C) and hence our site design, content etc would change. Even our blog section would have more than 90% of the content removed. What would be the ideal scenario be? Remove all pages and let those links be 404 pages Remove all pages and 301 redirect them to the home page Remove all unwanted pages and 301 redirect them to a separate page explaining the change (Although it wouldn't be that relevant since our audience has completely changed)- I doubt it would be ideal since at some point, we'd need ot remove this page as well and again do another redirection
Intermediate & Advanced SEO | | jombay0 -
Duplicate Content From Indexing of non- File Extension Page
Google somehow has indexed a page of mine without the .html extension. so they indexed www.samplepage.com/page, so I am showing duplicate content because Google also see's www.samplepage.com/page.html How can I force google or bing or whoever to only index and see the page including the .html extension? I know people are saying not to use the file extension on pages, but I want to, so please anybody...HELP!!!
Intermediate & Advanced SEO | | WebbyNabler0