I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl the site anymore
-
Hi Moz
I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl anymore.
When I look at the crawl report there is just one fault displayed:
"403 : Received 403 (Forbidden) error response for page."
How can i solve this issue asap?
I was not aware that there would be any issues with moz when activating a SSL certificate on the site.
The campaign with the problem is: ramxpert.dk.
Regards
Jesper Nielsen -
Hi Jesper,
Jo here from the Moz support team.
As WebBoost has noted our crawler isn't compatible with SNI, but the good news is that our team has been working hard on a new crawler which is in Beta right now and can handle SNI. If you're interested in being a part of the beta testing, the link to the signup page is here: http://goo.gl/forms/LCvL9Ix8JDHfbAvr1. Once you fill out that form, you will automatically be added into the Beta during the next round.
We will be rolling the new crawler out to all of our users once it's out of it's Beta phase, but I don't have an exact timeframe of when that will be at this time as it's still being worked on.
You're welcome to reach out to us at help@moz.com if you get stuck :]
Cheers!
Jo -
Thanks BeadedSEO
It seems through that the issue is that ramxpert.dk uses SNI, which isn't supported by the moz crawler yet
I've decided to sign up for the beta version of the new crawler, which I found on the link below. Hopefully this will work while they work on the final version.
https://moz.com/community/q/moz-pro-our-web-crawler-and-sites-that-use-sni
-
Have you tried updating your robots.txt file to use this;
User-agent: rogerbot
Disallow:
User-agent: dotbot
Disallow:
It's best practise to Disalow: none, rather than Allow: /
I've tried crawling your site with SEO Spider and ran in to no issues.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Bar not providing any data. Tried logging out/back in and un/re-installing, but no dice.
Used Mozbar for a long time, and normally works fine. Suddenly finding that it is not providing any data. All of the fields are there, but it does not provide me with PA/DA, etc, and all social metrics are at 0. This is across all sites, not just on in particular. Have tried logging out and in, deactivating and activating, and reinstalling. Nothing has worked.
Moz Bar | | SearchPros2 -
Cannot Crawl ... 612 : Page banned by error response for robots.txt.
I tried to crawl www.cartronix.com and I get this error: 612 : Page banned by error response for robots.txt. I have a robots.txt file and it does not appear to be blocking anything www.cartronix.com/robots.txt Also, Search Console is showing "allowed" in the robots.txt test... I've crawled many of our other sites that are similarly set up without issue. What could the problem be?
Moz Bar | | 1sixty80 -
I got a 404 in the Crawl Test Tool Report
I, yesterday i ran an crawl on http://www.everlastinggarden.nl and i get an 404. Does anybody know why this happens? <colgroup><col width="1535"></colgroup>
Moz Bar | | IMforYou
| # ---------------------------------------- |
| Crawl Test Tool Report | Moz,http://pro.seomoz.org/tools/crawl-test |
| www.everlastinggarden.nl |
| Report created: 15 Jul 18:34 |
| # ---------------------------------------- |
| URL,Time Crawled,Title Tag,Meta Description,HTTP Status Code,Referrer,Link Count,Content-Type Header,4XX (Client Error),5XX (Server Error),Title Missing or Empty,Duplicate Page Content,URLs with Duplicate Page Content (up to 5),Duplicate Page Title,URLs with Duplicate Title Tags (up to 5),Long URL,Overly-Dynamic URL,301 (Permanent Redirect),302 (Temporary Redirect),301/302 Target,Meta Refresh,Meta Refresh Target,Title Element Too Short,Title Element Too Long,Too Many On-Page Links,Missing Meta Description Tag,Search Engine blocked by robots.txt,Meta-robots Nofollow,Blocked by X-robots,X-Robots-Tag Header,Blocked by meta-robots,Meta Robots Tag,Rel Canonical,Rel-Canonical Target,Blocking All User Agents,Blocking Google,Blocking Yahoo,Blocking Bing,Internal Links,Linking Root Domains,External Links,Page Authority |
| http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page | Best regards, Jos0 -
Have there been recent changes to the On-Page Grader tool?
I use this page all the time to grade my primary and secondary keywords and for four years now have consistently managed to achieve A-C grades for all. Today however, I noticed I am flagging F grades for 50% of the keywords. I do make a note of the initial grades for all pages (as the majority of the keywords I grade using the tool are NOT added/tracked in MOZ 'campaigns'). To make sure I wasn't going mad I went back and graded some keywords I had achieved A-C grades for just 6 days ago. The changes are as follows: keyword 1 initially graded an A - 6 days later is still an A keyword 2 initially graded C and is now graded as an F keyword 3 initially graded C and now an F keyword 4 initially a C grade and now an F keyword 5 initially a B grade and has remained a B. Out of interest I then went and added one of the keywords that is now graded an F (but last week was a C) into the 'Add and manage Page Grades' section and graded once again - it's now grading this as a C. I then removed this keyword from my tracked keywords and graded again using the 'On-Page Grader' tool and ... it's back down to an 'F'. Please could you shed some light on what's going on? I don't want to track each keyword I grade over time but I do want the parameters for grading to remain the same if I use the 'On-Page Grader' tool - what's changed? I haven't received any alerts about any changes to the tool so if anyone could shed some light on any changes I would appreciate it.
Moz Bar | | em_h_uk0 -
Why'd Moz stop showing the list of users?
Curious to know if anyone else noticed that Moz stopped showing most of the active community users http://moz.com/community/users. It was nice to see who's who from visiting profiles and try to connect with them via email or see their websites, etc. There used to be pagination at the bottom. Why did they stop?
Moz Bar | | WhiteboardCreations0 -
Moz Toolbar Make My Browser Slower Drastically!
Hi, I really like Moz toolbar but it's totally make my browser (Firefox 26) slower (open new tab, new site, close / open the browser, etc.) What can I do to make it works faster With Moz toolbar? Thanks
Moz Bar | | JonsonSwartz0 -
Moz / more changes on the way?
I love Moz and the community and all the tools here. I admit I haven't rolled around in all the new things rolled out a few months ago. I thought there were more changes on the way but I wasn't sure if those already happened and I missed them or if I need to be patient? Affiliate program, client reporting? Thanks for any response. Have a great weekend! Matthew
Moz Bar | | Mrupp441 -
Open site explorer linking root domains
My company has been trying to increase the number of linking root domains for a specific page on our website using our PR company and press releases that are sent out and linked back to this page. This is working nicely, but the number of linking root domains is still not increasing under the "linking root domains" tab. I am noticing the correct links to this domain under "fresh web mentions" though. I know this tool can take a bit to update, but it has been quite some time and I still only see the one link.
Moz Bar | | isret_efront0