Cannot crawl website with redirect intalled on subdomain url
-
Hi!
I want to crawl this website : http://www.car-moderne.ch.
I tried a got back the crawl just for that one url (not for all the pages of the website). This single line cvs says that the status of the http://www.car-moderne.ch is 200, but in fact it is a redirect 301 to http://www.car-moderne.ch/fr where the live home page is (actually the Moz bar sees the 301, not the 200 as the single-lined crawl does).
How can I proceed in this case (a 301 redirect being installed on the subdomain url) to still be able to have a full-fledged juicy cvs with all the broken links, duplicate content, etc.
Thank you for your help!
Pascal Hämmerli
-
So glad to help, Pascal!
-
Dear Chiaryn,
Thank you for your very helpful reply.
This website is hosted on a partner agency who create the website and I only act as a SEO consultant for them. What you say is very helpful because it means their home-made CMS should be corrected to provided better 301 redirection.
I wish you a good day,
Pascal
-
Hey Pascal,
Sorry for the confusion here! It looks like the subdomain, www.car-moderne.ch, returns a 200 HTTP status to our crawler and to other crawlers, such as the hurl.it tool. In the body of the screenshot I attached from the hurl.it tool, the only code there is the number 404, so basically the site is serving a page with no crawlable data. The page isn't redirecting and it doesn't return any real source code, so there is no data for us to include in the crawl. I would recommend working with your webmaster to resolve this issue and to get the page to correctly serve a 301 redirect to the /fr version of the site to all crawlers.
I can see that the site is correctly responding with a 301 redirect for some crawlers, such as this test I ran as googlebot, but the response doesn't seem to be consistent. One thing you will want to be sure to have your webmaster check is how the site responds to user-agents that are hosted on Amazon Web Services, as some of our crawlers and the hurl.it crawl are both hosted through AWS.
Once the issue of the HTTP response is resolved, you should be able to get much better data from the crawl test tool.
I hope this helps! Please let me know if I can help you with anything else.
Chiaryn
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have too many tittle tag issue for my site on moz site crawl error
I have too many tittle tag issue in site crawl error but when I checked manually for the error there is no title in source code. Please Help me to understand
Moz Bar | | Nileshaggarwal0 -
How do I cancel a crawl request?
I was farting around and exploring the Crawl Test tool, and accidentally sent out a crawl for a competitor's site (I wanted to see if the tool would decline to crawl without verification). I do NOT want to actually crawl that site, nor do I want the competitor to see that we requested it (for obvious reasons) - how do I cancel it?
Moz Bar | | mkbeesto0 -
On-Page Grader Url is inaccessible
Hi everybody. I'm trying to use on -page grader for https://www.upscaledinnerclub.com and get "Sorry, but that URL is inaccessible." Robots.txt are empty, another thread on MOZ was talking about DNS check - it's all good. So, I can't figure out why this is happening. Also I am trying the same for another website https://www.regexseo.com - the same story. Common thing is that they both are on Google App Engine. And at first i thought that was the problem. Bu then i checked this one : https://www.logitinc.com/ and it's working, even though this website is on GAE as well. None of these website have robots.txt or any differences in setup or settings. Any thoughts?
Moz Bar | | DmitriiK0 -
My crawl report only shows 1 link
Hello, I've tried a crawl for the site www.doctify.co.uk and it's only returned 1 link in the report which is the homepage. Do you know what the issue could be? Thanks, Nina
Moz Bar | | Global_Blue0 -
Moz Crawl Showing Duplicate Content But It's Not?!
Unfortunately I can't give out the URL, but here's the deal... I have two URL's which have completely different content on them but are being crawled as duplicate content. Any Idea how that would happen? I'm not seeing any errors in WMT's. Has anyone seen this before? Is the duplicate content reporting based on a % of the page content matching as the same?
Moz Bar | | Swarm-SEO0 -
Where to find one off crawl report
Hello, I don't know if I am being a bit daft but I don't seem to be able to find the area where I can request a one off crawl report anymore (rather than setting up a campaign). Can someone let me know where this is now? Thanks!
Moz Bar | | RikkiD220 -
Best way to submit multiple, simultaneous urls to SEOMOZ
I have a pro membership, and am looking to get inbound link data on multiple urls. Is there a way to submit multiple urls at once?
Moz Bar | | cmcilwain0