Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Generating 404 Errors but the Pages Exist
-
Hey
I have recently come across an issue with several of a sites urls being seen as a 404 by bots such as Xenu, SEOMoz, Google Web Tools etc. The funny thing is, the pages exist and display fine.
This happens on many of the pages which use the Modx CMS, but the index is fine. The wordpress blog in /blog/ all works fine.
The only thing I can think of is that I have a conflict in the htaccess, but troubleshooting this is difficult, any tool I have found online seem useless.
Have tried to rollback to previous versions but still does not work.
Anyone had any experience of similar issues?
Many thanks
K.
-
FYI, we finally found our error. The short URL turned out to be the same name as the folder (photo-gallery) so once this was changed, wordpress was able to access the correct path. A bit of custom javascript had to be amended as well, but that was limited to our custom code. Using your web-sniffer.net link we were able to test immediately and fix it fairly quickly. Thank you for your help!
-
That's true Ryan I guess it is coding related really.
Issues like this are a real pain in the ass. And most people don't even check WMT to realise the issues exist. TBH, I don't check as often as I should.
-
I agree with you Paul.
As you pointed out one possible cause is a CMS-related issue which I would refer to as "coding" meaning something in the code which was used to present the website. Perhaps there is a better way to phrase it but nothing comes to mind at the moment.
Another possibility you mentioned is Litespeed which would be a server-side issue directly. Either way, it is a legitimate issue which should be addressed.
-
FWIW, I don't think it's a coding issue. If it were coding, it would either show a 200OK or it would show a 404. It wouldn't sometimes serve a 404.
If you're using Litespeed, I'd guarantee that is the issue and if you're using Joomla, it's another prime culprit.
-
Please keep in mind, that 404 error does not mean the page doesn't exist. It means your server, is sending a response code to indicate that it doesn't exist.
When I installed Litespeed on my server, this issue happened over and over again.
I believe Joomla for example, has some kind of security module that serves a 404 if a single IP requests a page too many times. I remember running SEOFrog on a friends Joomla site and tons of 404's were showing up.
-
Dev team are looking into it, must be quite a complex htaccess issue. Will get to the bottom of it this week and post any findings.
-
Thanks Ryan! I will get it looked at...Sue
-
@DentalID, the same reply I offered to Guy applies for you as well. This is an SEO issue which does need to be fixed. Something on your end is causing the page to show with a 403 response code. You really need a programmer to get in there and determine the root cause of the issue. You could try asking your web host if you have managed hosting, but this level of assistance would normally be outside the support of managed hosting.
-
Guy,
In looking at the page this appears to be a legitimate problem. Your server settings allow you to present a page with any header code you wish. You can 301 a page but still present the page with a 200 code if you want. Presently it appears the page is being presented fine but your server is offering a 404 header code.
I can't tell the actual source of the problem other then to say it appears to be on your end and should be fixed. I originally looked at the code with the MOZbar but then checked independently with another tool as well. http://web-sniffer.net/
All tools show a 404 header code for the page. This response code is generated by your web server.
-
We are having a similar problem with this URL: http://dentalimplantsportland.com/photo-gallery/ and also the following locations:
http://cosmeticdentistportland.net/photo-gallery/
http://dentalveneersportland.com/photo-gallery/
SEO Moz and Google webmaster tools show it as a 403 error but the pages display fine. I am not able to tell if this is really a problem for SEO or if we should reconstruct this gallery system and would really love your input.
This is Wordpress with a Spry gallery...
Thanks so much!
-
It is just a small affiliate site I am looking at - this page creates a 404.
http://www.insure-uk.com/post-office-car-insurance.html
Currently testing on some beta servers. Hopefully should fix soon as otherwise it will lose indexation.
-
I also see this now and again, but next crawl they fix themselfs. i assume robots can not always reach page for a number of reasons
-
Can you offer an example of a URL which is causing this problem?
-
I have had the same issues, I think it is often the bot's problem
Just to be certain check your links are correct and manually test them. Also ensure your sitemap is up to date and that you are not blocking the crawlers with metarobots, robots.txt, or some weird stuff in htaccess.
I have found that renaming pages or moving them will often cause 404 issues with crawlers
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Few pages without SSL
Hi, A website is not fully secured with a SSL certificate.
Intermediate & Advanced SEO | | AdenaSEO
Approx 97% of the pages on the website are secured. A few pages are unfortunately not secured with a SSL certificate, because otherwise some functions on those pages do not work. It's a website where you can play online games. These games do not work with an SSL connection. Is there anything we have to consider or optimize?
Because, for example when we click on the secure lock icon in the browser, the following notice.
Your connection to this site is not fully secured Can this harm the Google ranking? Regards,
Tom1 -
Category Page as Shopping Aggregator Page
Hi, I have been reviewing the info from Google on structured data for products and started to ponder.
Intermediate & Advanced SEO | | Alexcox6
https://developers.google.com/search/docs/data-types/products Here is the scenario.
You have a Category Page and it lists 8 products, each products shows an image, price and review rating. As the individual products pages are already marked up they display Rich Snippets in the serps.
I wonder how do we get the rich snippets for the category page. Now Google suggest a markup for shopping aggregator pages that lists a single product, along with information about different sellers offering that product but nothing for categories. My ponder is this, Can we use the shopping aggregator markup for category pages to achieve the coveted rich results (from and to price, average reviews)? Keen to hear from anyone who has had any thoughts on the matter or had already tried this.0 -
What if page exists for desktop but not mobile?
I have a domain (no subdomains) that serves up different dynamic content for mobile/desktop pages--each having the exact same page url, kind of a semi responsive design, and will be using "Vary: User-Agent" to give Google a heads up on this setup. However, some of the pages are only valid for mobile or only valid for desktop. In the case of when a page is valid only for mobile (call it mysite.com/mobile-page-only ), Google Webmaster Tools is giving me a soft 404 error under Desktop, saying that the page does not exist, Apparently it is doing that because my program is actually redirecting the user/crawler to the home page. It appears from the info about soft 404 errors that Google is saying since it "doesn't exist" I should give the user a 404 page--which I can make it customized and give the user an option to go to the home page, or choose links from a menu, etc.. My concern is that if I tell the desktop bot that mysite.com/mobile-page-only basically is a 404 error (ie doesn't exist), that it could mess up the mobile bot indexing for that page--since it definitely DOES exist for mobile users.. Does anyone here know for sure that Google will index a page for mobile that is a 404 not found for desktop and vice versa? Obviously it is important to not remove something from an index in which it belongs, so whether Google is careful to differential the two is a very important issue. Has anybody here dealt with this or seen anything from Google that addresses it? Might one be better off leaving it as a soft 404 error? EDIT: also, what about Bing and Yahoo? Can we assume they will handle it the same way? EDIT: closely related question--in a case like mine does Google need a separate sitemap for the valid mobile pages and valid desktop pages even though most links will be in both? I can't tell from reading several q&a on this. Thanks, Ted
Intermediate & Advanced SEO | | friendoffood0 -
Does an H1 have to be at the top of a page?
Because H1 "may" carry some weight with Google does it have to be placed at the top of the page? Can I place it towards the bottom of the page instead in normal body size? My goal is to keep the main keywords in the H1 but create a much friendlier title for the customer to read at the top of the page.
Intermediate & Advanced SEO | | PottyScotty0 -
Stuck on Page 2 - What Would You Do?!?
My site is : http://goo.gl/JgK1e My main keyword is : Plastic Bins i have been going back and forth between page 1 and 2 for this keyword and i was wondering if any of you could provide any guidance as to why i can't get on the top of page 1, and stay there... My site has been around for a while, we believe we have a great user experience, all unique, fresh content, and the lowest prices... I must be missing out on something major if I cannot get a steady page 1 ranking... Any thoughts? Thanks in advance...
Intermediate & Advanced SEO | | Prime850 -
Are there any negative effects to using a 301 redirect from a page to another internal page?
For example, from http://www.dog.com/toys to http://www.dog.com/chew-toys. In my situation, the main purpose of the 301 redirect is to replace the page with a new internal page that has a better optimized URL. This will be executed across multiple pages (about 20). None of these pages hold any search rankings but do carry a decent amount of page authority.
Intermediate & Advanced SEO | | Visually0 -
Multiple URLs for the same page
I am working with a client and recently discovered that they have several URLs that go to the same page. http://www.maps.com/FunFacts.aspx
Intermediate & Advanced SEO | | WebMarketingandDesign
http://www.maps.com/funfacts.aspx
http://www.maps.com/FunFacts.aspx?nav=FF
http://www.maps.com/FunFacts.aspx?nav=FS
http://www.maps.com/funfacts.aspx?nav=FF
http://www.maps.com/funfacts.aspx?nav=ffhttp://www.maps.com/FunFacts.aspx?nav=MShttp://www.maps.com/funfacts.aspx?nav=
http://www.maps.com/FunFacts.aspx?nav=FF#
http://www.maps.com/FunFacts
http://www.maps.com/funfacts.aspx?.nav=FF I am afraid this is happening all over the site. So, my question is: Is this hurting the SEO and how? If so what is the best way to go about fixing this problem? Thanks for your help!0 -
Does Google crawl the pages which are generated via the site's search box queries?
For example, if I search for an 'x' item in a site's search box and if the site displays a list of results based on the query, would that page be crawled? I am asking this question because this would be a URL that is non existent on the site and hence am confused as to whether Google bots would be able to find it.
Intermediate & Advanced SEO | | pulseseo0