Crawling password protected sites such as dev or staging areas to look at sites b4 going live ?
-
Hi
Ive instructed clients to password protect dev areas so dont get crawled and indexed but how do we set up Moz crawl software so we can crawl theses sites for final check of any issues before going live ?
Is there an option i havnt seen to add logins/passwords for crawl software to access ?
cheers
dan
-
ok thanks Chiaryn
is that the actual name of the moz crawler (to allow in Robots) simply rogerbot ? or any other characters etc ?
Also is it not the case that even when blocked by robots.txt G can still crawl/index it once password removed, think i read few comments somewhere on Moz that can still happen somehow ?
Please advise asap ?
Many Thanks
Dan
-
Hey Dan,
Unfortunately, our crawler is not able to access password protected content on your site. If you create a staging subdomain that is not password protected, you could use the robots.txt file to allow rogerbot and block other crawlers, but I'm afraid our crawler will not crawl anything that a normal search engine crawl would not be able to crawl so we cannot crawl password protected pages.
I hope this helps.
Chiaryn
-
i dont suppose either of you are able to help at all with this related question:
http://moz.com/community/q/site-crawl-errors-download-list-of-all-urls
-
i dont suppose either of you are able to help at all with this related question:
http://moz.com/community/q/site-crawl-errors-download-list-of-all-urls
-
Hi Andy
Screaming Frog does have password access feature for your info i have just tried it
All Best
Dan
-
Thanks Matt
I have got screaming frog and can confirm that it has password access feature, but i really want Moz to be able to access too, i would have thought they should have this option somewhere. Are you saying Moz crawls have more info than SF (re 'moz level' analysis) ?
Dev site better password prtected than robots arnt they i think ?
Cheers
Dan
-
Hi Dan
I was about to ask the exact same question, so will keep an eye out for an answer.
I hope it is possible, but I couldn't work it out.
-
I don't know if there's a way to do this in Moz but you could always get Screaming Frog & tell it to ignore robots.txt - that will definitely crawl it. You can check titles, descriptions, canonicals, H1s, etc. that way. It doesn't give the Moz level analysis but it's a start that def works. You can also see if you have parameter issues that way.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
In Moz Campaigns, how are competitor domains tracked if they redirect their site?
Hello! One of our competitors (Company A) that we've tracked in Moz for a long time recently merged with another company (Company B) and redirected their whole site to Company B's site. Will our competitor tracking still work as-is? Or do we need to make an adjustment? I'm reluctant to delete Company A from our competitor tracking, because we will lose all of that data. But if all of the keywords are slowly going to drop off as Google starts showing Company B results only, it may be the only option. Any help is appreciated! Thanks!
Moz Bar | | PrimeFoodTeam0 -
Moz Crawl Report Increase in Errors?
Has anyone else noticed a huge increase over the past couple weeks in crawl issues in their dashboards? Without being able to see historical data week over week, I can't tell what's been added. Is this some update with the tool? I'm not seeing any health issues with this feature on the Moz Health page, it just seems strange that I'm seeing this across all our accounts.
Moz Bar | | WWWSEO0 -
Different Errors Running 2 Crawls on Effectively the Same Setup
Our developers are moving away from utilising robots.txt files due to security risks, so e have been in the process of removing them from sites. However we, and our clients still want to run Moz crawl reports as they can highlight useful information. The two sites in question sit on the same server with the same settings (in fact running on the same Magento install). We do not have a robots.txt files present (they 404), and as per Chiaryn's response here https://moz.com/community/q/without-robots-txt-no-crawling this should work fine? However for www.iconiclights.co.uk we got: 902 : Network errors prevented crawler from contacting server for page. While for www.valuelights.co.uk we got: 612 : Page banned by error response for robots.txt. These crawls were both run recently, and there was no robots.txt present. Not to mention, they are on the same setup/server etc as mentioned. Now, we have just tested this, by uploading a blank robots.txt file to see if it changed anything - but we get exactly the same errors. I have had a look, but can't find anything that really matches this on here - help would really be appreciated! Thanks!
Moz Bar | | I-COM0 -
Possible bug in Crawl Issues report?
Hi all - My crawl issues report shows 3 pages with missing titles. These are just google verification files and the robot.txt file - shouldn't these be excluded? Pages with Title Missing or Emptyas of May 11
Moz Bar | | A-Drive
URL Page Authority Linking Root Domains
https://www.mysite.com/googlea87e28121c071983.html
1 0
https://www.mysite.com/robots.txt
1 0
https://www.mysite.com/google9b9dc57478f61677.html0 -
Does anyone have a good article or video on how to read the SEO MOZ crawl report column by column?
I am trying to find a good how-to on how to read and analyze each column of the SEO MOZ crawl report, specifically, the excel sheet it allows you to export. What I'm really trying to get to the bottom of is what the "Yes" indiciates under rel-cononical. If it says "yes," does this mean that the link in question has been canonoicalized?
Moz Bar | | armcwill0 -
Moz crawl suddenly shows much less pages from what I really have
Hi! Moz crawl suddenly shows much less pages from what I really have and from what they used to show after completing the crawl. Should I be worried? What could that be? Regards, Yossey
Moz Bar | | Joseph-Green-SEO1 -
Is there a problem with New Campaign crawls being slow?
Hi I've created 3 new campaigns but results not showing as usual. I know the full results take up to 7 days but I usually see some basic info in a few minutes of creating the campaign. This problem happened a while ago and the campaign was never quite right in SEOMoz. Thanks for your help Steve
Moz Bar | | stevecounsell0