Www vs non www - Crawl Error 902
-
I have just taken over admin of my company website and I have been confronted with crawl error 902 on the existing campaign that has been running for years in Moz. This seems like an intermittent problem. I have searched and tried to go over many of the other solutions and non of them seem to help.
The campaign is currently set-up with the url http://companywebsite.co.uk when I tried to do a Moz manual crawl using this URL I got an error message. I changed the link to crawl to http://www.companywebsite.co.uk and the crawl went off without a hitch and im currently waiting on the results. From testing I now know that if i go to the non-www version of my companies website then nothing happens it never loads. But if I go to the www version then it loads right away.
I know for SEO you only want 1 of these URLS so you dont have duplicate content. But i thought the non-www should redirect to the www version. Not just be completely missing.
I tried to set-up a new campaign with the defaults URL being the www version but Moz automatically changed it to the non-www version. It seems a cannot set up a new campaign with it automatically crawling the www version.
Does it sound like im out the right path to finding this cause? Or can somebody else offer up a solution?
Many thanks,
Ben.
-
Glad to hear thats all fixed! Though i will say thats a very slow response time for any development / hosting company typically i would expect a maximum response time of 8 hours.. We try to keep it under 2 heh.
But yes, glad thats working for you now
-
The problem was as you anticipated, after spending a few days chasing the party who actually look after it im pleased to say i checked this morning and all is working as expected.
Thanks very much for your help Toby!
-
Thanks Toby, Ive emailed off, I expect a reply to be a couple of days away (what it normally take them) Thanks for the help thus far and ill message back when they do!
-
In that case that would probably be the best place to start. If you want any evidence for a missing A record, heres a DNS checking tool (it currently throws an error because it can't find an A record).
Let me know what they say
-
I dont have access to the DNS, my access is limited to the Magento CMS, and ftp access to the root folder.
I guess its a case of popping an email off to the guys who built the site and control the web space and getting them to address the issue?
-
Ok, so the file is working, thats a good start!
Looks like we need to go back a step in the request process then. Do you have access to the DNS settings for the http://atp-instrumentation.co.uk domain? If so, please could you check if there is an A record set for it? (looks to be registered through Civica UK Ltd - Whois Report)
What i suspect might be the case is that you're missing an A record for atp-instrumentation.co.uk but that there is one set for the www.atp-instrumentation.co.uk.
I've run a couple of tests against the domain DNS and i get nothing back for the non-www address, which is what suggests that we're not even making it as far as your servers.
To set the A record, you'll be looking for something in your control panel for 'DNS settings' or maybe 'Host Records', you should see in there ether an option to select A record settings, or perhaps a dropdown with things like A, AAA, CNAME etc. You need to:
- select 'A'
- In the domain box type: atp-instrumentation.co.uk
- In the IP box, type: 82.118.110.42
Hopefully that makes sense. If your at all unsure, let me know and i'll do what i can to help more specifically. Domain control panels are so different for each provider its difficult to provide direct instructions without knowing what your panel looks like
-
I added the line and it did indeed break the website, got internal service error etc
So it seems the file is working
-
Just a note here, the Redirects do seem to be working, so it looks like the .htaccess file IS being loaded.
Please check anyway, its possible that the redirects are comming from another location if they have been set elseware as well. From a (very quick) look, the file seems to be formatted correctly so no obvious reason for the www redirects to not be working...
-
Hmm ok,
before i do anything else, we need to make sure that the .htaccess file is actually being loaded. To do that, we need to break it for a second.
at the top of the htaccess file, put something like (specifics dont matter here)
THISisInValid666
What we're trying to do is put some invalid text into the htaccess file so that the site breaks when it loads. The idea being that we can confirm that the .htaccess file is actually being used. So if you put that in and the site throws a 500 error (when navigating to it with or without www) we can confirm that changes we make should work.
If the site continues to load without issue then we know that the .htaccess file isnt in use so we need to look at server configureation directly (specifically the AllowOverrides settings)
Once you have confirmed if it does / doesnt break, remove the line again
-
Thanks Toby, here is the entire HTACCESS file with your fix implemented. Doesn't seem to have worked if i go to
http://atp-instrumentation.co.uk with no www it still fails to load
Edited out to shorten convo
-
You're correct, you can make it a little more generic though, without seeing all of your .htaccess file, try this:
Replace:
RewriteCond %{HTTP_HOST} ^companyname.co.uk [NC]
RewriteRule ^(.*)$ http://www.companyname.co.uk/$1 [L,R=301]With:
RewriteCond %{HTTP_HOST} !^www. [NC]
RewriteRule ^ http%{ENV:protossl}://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]This is what could be called a wildcard redirect in that a direct copy paste should work for you with no need to edit. (you dont have to manually add in the correct domain name)
What it does:
- First it checks to see if the requested url has a www in it
- if it does -not- it then runs the rule, otherwise it ignores it.
- The rule first checks for http or https. Then adds in the www. followed by the domain and tld, finally adding the URI (/somepage/page for example).
- the L in square brakets means do not process anything else in the htaccess file
- the R=301 means that it will be a 301 (perminant) redirect.
If that still doesnt work for you, paste up your full .htaccess file, or you can send it to me directly if you'd rather and i'll take another look
-
Thanks Highland,
How do I go about changing this? I believe its to do with the .htacess file.
The website was developed in Magento via an external company who monitor it. Looking in the root folder I can see the htaccess file but it contains a lot lines of code and rewrites that I dont fully understand.
These lines are the one I think could be relevant so far.
############################################
enable rewrites
Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^companyname.co.uk [NC]
RewriteRule ^(.*)$ http://www.companyname.co.uk/$1 [L,R=301]Then there is lots of category pages etc that are 301 redirected
Follow by
RewriteRule ^home http://www.companyname.co.uk/ [R=301,L]Then some more redirects for pages
I know this is specific but is this editable a different way in Magento? Thanks for any help offered i know this is getting more technical
-
You're 100% right. You should have one 301 redirect to the other. While there are some SEO reasons for this (mainly with duplicate content), the best reason is that it's just less confusing to end users to only have one URL to use. If your non-www has trouble loading I would say you need a 301 to the www version.
Moz restricts Top Level Domain (i.e. domain.com) and crawls accordingly. I have some set up with www.domain.com and some with just domain.com. The 301 to the www forces the issue but Moz is smart enough to crawl the proper pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No index and Crawl Budget
Hello, If we noindex pages, will it improve crawl budget ? For example pages like these - https://x-z.com/2012/10/
Technical SEO | | Johnroger
https://x-y.com/2012/06/
https://x-y.com/2013/03/
https://x-y.com/2019/10/
https://x-y.com/2019/08/ Should we delete/redirect such pages ? Thanks0 -
404 errors
Hi I am getting these show up in WMT crawl error any help would be very much appreciated | ?escaped_fragment=Meditation-find-peace-within/csso/55991bd90cf2efdf74ec3f60 | 404 | 12/5/15 |
Technical SEO | | ReSEOlve
| | 2 | mobile/?escaped_fragment= | 404 | 10/26/15 |
| | 3 | ?escaped_fragment=Tips-for-a-balanced-lifestyle/csso/1 | 404 | 12/1/15 |
| | 4 | ?escaped_fragment=My-favorite-yoga-spot/csso/5598e2130cf2585ebcde3b9a | 404 | 12/1/15 |
| | 5 | ?escaped_fragment=blog/c19s6 | 404 | 11/29/15 |
| | 6 | ?escaped_fragment=blog/c19s6/Tag/yoga | 404 | 11/30/15 |
| | 7 | ?escaped_fragment=Inhale-exhale-and-once-again/csso/2 | 404 | 11/27/15 |
| | 8 | ?escaped_fragment=classes/covl | 404 | 10/29/15 |
| | 9 | m/?escaped_fragment= | 404 | 10/26/15 |
| | 10 | ?escaped_fragment=blog/c19s6/Page/1 | 404 | 11/30/15 | | |0 -
301 redirect: canonical or non canonical?
Hi, Newbie alert! I need to set up 301 redirects for changed URLs on a database driven site that is to be redeveloped shortly. The current site uses canonical header tags. The new site will also use canonical tags. Should the 301 redirects map the canonical URL on the old site to the corresponding canonical for the new design . . . or should they map the non canonical database URLs old and new? Given that the purpose of canonicals is to indicate our preferred URL, then my guess is that's what I should use. However, how can I be sure that Google (for example) has indexed the canonical in every case? Thx in anticipation.
Technical SEO | | ztalk1120 -
404s effecting crawl rate?
We made a change to our site where we all of a sudden we are creating a large number of 404 pages. Is this effecting the crawl/indexing rate? Currently we've submitted 3.4 million pages, have over 834K indexed but have over and 330K pages not found. Since the large increase in 404s we've noticed a decrease in pages crawled per day. I found this Q & A in Webmasters (http://googlewebmastercentral.blogspot.com/2011/05/do-404s-hurt-my-site.html) but it seems like the 404s should not have an effect. Is this article out of date? What do you think fellow Moz-ers? Is this a problem?
Technical SEO | | JoshKimber0 -
Sitemap issue - Tons of 404 errors
We've recreated a client site in a subdirectory (mysite.com/newsite) of his domain and when it was ready to go live, added code to the htaccess file in order to display the revamped website on the main url. These are the directions that were followed to do this: http://codex.wordpress.org/Giving_WordPress_Its_Own_Directory and http://codex.wordpress.org/Moving_WordPress#When_Your_Domain_Name_or_URLs_Change. This has worked perfectly except that we are now receiving a lot of 404 errors am I'm wondering if this isn't the root of our evil. This is a WordPress self-hosted website and we are actively using the WordPress SEO plugin that creates multiple folders with only 50 links in each. The sitemap_index.xml file tests well in Google Analytics but is pulling a number of links from the subdirectory folder. I'm wondering if it really is the manner in which we made the site live that is our issue or if there is another problem that I cannot see yet. What is the best way to attack this issue? Any clues? The site in question is www.atozqualityfencing.com https://wordpress.org/plugins/wordpress-seo/
Technical SEO | | JanetJ0 -
Redirects & 404 Errors
Hi everyone, I'm probably missing some GLARING error here, but I'm hoping you can help me! We recently built a new website on Wordpress and attempted to use a redirect plugin to take care of some old pages. The issue we are having though, is that when you click an old link you are not automatically redirected and instead are given a 404 error page. Then, when you try to view another page (by clicking a navigation item), every pages shows a 404 error. I implemented a redirect plugin, however it seems to start to work then still throws the 404 page. I believe this has something to do with the htaccess file which has the standard WP rewrite info in there... The way the old site was setup was kind of janky, so wondering if it's on that side or if I'm just going crazy. An old URL example would be http://orchards inn.com/index.php/specials and the new page is http://orchardsinn.com/special-offers. Sometimes the redirect seems to work, and others it actually throws a 404 page, then every other page in the navigation is 404'd as well. Your help is GREATLY appreciated!!
Technical SEO | | marisolmarketing0 -
Seo and ssl error (Error code: sec_error_revoked_certificate)
Hi. An error occurred during a connection to esta-register.org. Peer's Certificate has been revoked. (Error code: sec_error_revoked_certificate) ** i want to know this error can be effected on seo or not?** esta
Technical SEO | | vahidafshari450 -
Does this page crawl well?
I just put up a page that uses an image map to illustrate a national currency note. http://www.antiquebanknotes.com/NationalCurrency/National-Bank-Note-Information.aspx My goal with this page is get results for National Bank Note. But I know image maps are wierd creatures and not good for linking. My question is, will Google index my tooltips and find this page useful and therefore worthy? I think the content is useful for my users but I just don't know if the implementation will work well. This screen will eventually have 5 or 6 notes on it and I don't want to do it the concensus is negative... Thanks for any advice.
Technical SEO | | Banknotes0