Rogerbot will not crawl my site! Site URL is https but keep getting and error that homepage (http) can not be accessed. I set up a second campaign to alter the target url to the newer https version but still getting the same error! What can I do?
-
Site URL is https but keep getting and error that homepage (http://www.flogas.co.uk/) can not be accessed. I set up a second campaign to alter the target url to the newer https://www.flogas.co.uk/ version but still getting the same error! What can I do? I want to use Moz for everything rather than continuing to use a separate auditing tool!
-
It will probably help if you delete the following line from your robots.txt file:
User-Agent: rogerbot
Disallow: /Dirk
PS You also block /js/ for all bots - Google doesn't really recommend that
-
I just check and see that http is 301 to https. But https works.
Can you mail this issue to help@moz.com?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Campaign in subdirectory
Hi. I have a main site under the TLD and Moz sucessfully scans all URL's under that domain including for a subfolder on the domain. However, I have also set up a campaign for the subdirectory as it is effectively a seperate site (let's say it's like an ecommerce site with a blog is a subfolder). Moz crawls the main campaign on the TLD including the subfolder correctly (I was previously told you cannot exclude a subfolder), but it does not scan any URL's beyond the top one in the subfolder campaign (I was told this setup was OK) Any ideas why the subfolder does not get crawled by Moz?
Moz Bar | | jsdw290 -
Why do I get different results with On-Page Grader and Page Optimization tool? What is the difference between these two tools?
I use the Page Optimization tool to evaluate and make updates to a page until I get a grade of "A." But when I add the page and the keyword to the Page Optimization section of my campaign, it scores a 76% or 85% (which looks like a C or B to me). What would account for the difference in the evaluation results? Thanks for your help!
Moz Bar | | GLabs0 -
On-Page Grader "Sorry, but that URL is inaccessible."
We have a new client with a squarespace page. http://www.mountainhouseestate.com The Moz On-Page grader returns the error "Sorry, but that URL is inaccessible." for all pages. Possibly related, Google seems to hate their site. Even a search for "mountain house estate" returns lousy results. Bing/Yahoo has no problem with it.
Moz Bar | | Duke_Ferris1 -
Ww.domain.com coming up with error
our domain is showing in moz with the following error in crawl reports Crawl Error We were unable to access your homepage, which prevented us from crawling the rest of your site. It is likely that other browsers as well as search engines may encounter this problem and abort their sessions. This could be a temporary outage, but we recommend making sure your network and server are working correctly. note that the url being displayed is ww.domain.com and not www.domain.com . we do not have a 301 in place, we have switched off wildcard forwarding from the server.. its acting as the url is a subdomain that is not working.. should i just ignore it?
Moz Bar | | Direct_Ram0 -
Open Site Explorer classic view is gone??! where is it?
I used to use open site explorer classic view a lot for pitching clients - it was great for comparing a few competitors at once. Now it seems to have disappeared - any idea where i can find it, or find similar functionality?
Moz Bar | | miguelitomana0 -
Given that I am currently using a bot sniffer...How can I identify the MOZ bot in order to whitelist it?
MOZ is currently blocked from crawling my sites because I use a bot sniffer. Does anyone know how I can properly identify the MOZ bot in order to whitelist it? MOZ is using Amazon web services and thus employs thousands of dynamic IPs to crawl.
Moz Bar | | Felix_LLC0 -
Can I delete a SEO campaign in Moz, and start a new one for a different website?
Would be nice to know, as I'm limited to 5 campaigns, and the most important work is done for 2 site's, so I would like to switch that to 2 other website's. Regards,
Moz Bar | | mrblue910 -
Since the revised website was launched, I can't find the "Crawl Test" function showing Titles and Descriptions of other websites. Anyone know where that link is located?
MOZ can "crawl" any website and show information like Title, Description, etc.....Can't find that link.
Moz Bar | | bpedrazas0