We recently switched from HTTP to HTTPS and we are having crawling issues!
-
We switched our website from HTTP to HTTPS and we started to get an email from Moz about the robots.txt being unable to crawl our website. The website is hosted through wordpress but we haven't had any issues until we switched. We have no idea what to do or even what the problem is! If you have had a similar problem and fixed it, we need your help! Thank you.
-
I know this is an old thread, but we are still having the same problem. I finally got around to sending a note to flywheel about this problem and it came back that everything is fine. I am not sure what to do here? It's on a shared hosted, so I don't have console\audit log access, however Flywheel is one of the best wordpress hosting companies out there (only thing they do).
As far as accessing the robots.txt file, I can go directly to it without any problems?
https://southernil.com/robots.txt -
Hi there!
Thanks so much for reaching out! I'm sorry you're having trouble!
I took a look at your crawl data and your site to see if I could figure out the issue. When I first tried to access your robots.txt file from a browser, it returned an error saying there were too many redirects in place. I checked to see what our crawler was receiving from your server and looks like it keeps being served a 301 redirect which points back to itself. However, when I tried to access the file from a browser a bit later, it loaded without a problem. I'm wondering if you can check your server logs to see what your server is sending back to our crawler Rogerbot?
If you could send any further info over to help@moz.com, that would be great! That way we can do some more digging and see what's going on.
Looking forward to hearing from you!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz DA Issue
Hey Community, This is my site which is the Best Drifitng in Australia. I am getting a problem and my site's DA is unable to show using. I want to see it as soon as possible, although I am using a premium tool. Thanks
Getting Started | | jndjue40 -
Why does Moz only seem to be crawling a snap shot of the site I am working with?
I was wondering if anyone can help? I am working using Moz to help improve the SEO on a website I am working with, the website contains thousands of pages, yet for some reason Moz only seems to be crawling a small snap shot of the website. I know there are particular pages that I had added a couple of weeks ago - about 300 in total - and none of these were showing on the first crawl, so I did another on-demand crawl and some of these showed up then. Despite this, it says it crawled 700ish pages, but there are getting close to 20-30ish thousand live pages on the site. Any thoughts and guidance as to why they crawling may be stopping?
Getting Started | | dsmith8020200 -
Do I have to manually mark as 'fixed' or 'Ignore' manually on Metadata Issues when completed?
When I have amended the missing description Metadata issue on an individual page do I have to manually mark as 'fixed' or 'Ignore'? I have attended to several pages regards Missing Description Metadata Issues and not manually marked them as fixed. However they still appear in metadata missing description issues after a second on demand crawl? Will they continue to appear until I manually mark them as fixed?
Getting Started | | LM_Marketing_Solutions_Ltd0 -
Scheduled update - Re-Crawl - recrawl
Can I not perform a manual update? I setup a campaign without GA as I did not have access, I got access, added the GA account to the campaign but no data is showing as I think I require an update, but have to wait 7 days? Is that right? Thanks
Getting Started | | SJMDT0 -
Error Code 804: HTTPS (SSL) Error Encountered
Recently we bought SSL and moved our site to https. (https://www.kitchenstuff.com) where all link are https even after I see an error "Error Code 804: HTTPS (SSL) Error Encountered". Can anyone help me out? What steps can I take to troubleshoot this error?
Getting Started | | torbett0 -
Why wont rogerbot crawl my page?
How can I find out why rogerbot won't crawl an individual page I give it to crawl for page-grader? Google, bing, yahoo all crawl pages just fine, but I put in one of the internal pages fo page-grader to check for keywords and it gave me an F -- it isn't crawling the page because the keyword IS in the title and it says it isn't. How do I diagnose the problem?
Getting Started | | friendoffood0 -
After fixing Crawl Errors, how long does it take to for Moz or Google to re-crawl a website?
Last night I found out through Moz that my robots.txt file was blocking any crawling of my website. I fixed the issue. Now do I just sit and wait?
Getting Started | | cmc-interactive0 -
Crawling Website
How often does Moz crawl the website? I have a lot of high priority crawl issues but my web development team says that it fixed them. However, Moz still shows me that none of the errors are fixed.
Getting Started | | Leoni0