Can I crawl a password protected domain with SEOmoz?
-
Hi everyone,
Just wondered if anybody has been able to use the SEOmoz site crawler for password protected domains? On Screaming Frog you are prompted for the username and password when you set the crawler running, however SEOmoz doesn't. It seems you can only crawl sites that are live and publicly available - can anyone confirm if this is the case?
Cheers,
M
-
Hi,
I don't think it is possible to crawl any password protectec pages. Password protected pages are usually blocked and bots are unable access those pages.
And yes it is true that only sites that are publicly and live can be crawled.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What should I consider before setting up a sub domain?
Morning all! We've just been approached by IT. They've been asked to develop an online 'portal' where clients can upload and download materials. IT will be developing a portal that sits on the company network perimeter (hosted on our internal servers). The concept is that 3<sup>rd</sup> parties can get and update information in regards to progressing cases, the first use will be for agencies who will retrieve records via the portal and then post reports after a consultation. however I would like to have an automatic link to forward to the portal from the web address: oursite.com/dave We will look to create robot.txt and anything else to prevent from listings/indexes. Does any of the above mess with your SEO? The Directors have asked if they can have this on a sub-domain of our site. Is this wise? And, are there any major SEO considerations for my team to worry about? Better still, have any of you had to deal with this before? If so, what happened? All the best, John
Technical SEO | | Muhammad-Isap0 -
Crawl issues
Hello there, I have found that when crawling my site I have errors regarding the meta description and it says it is missing from few pages. I checked these pages but there is a meta description. I also ran the same report with other tools and it comes up the same issues. What should I do?
Technical SEO | | PremioOscar0 -
How do crawl errors from SEOmoz tool set effect rankings?
Hello - The other day I presented the crawl diagnostic report to a client. We identified duplicate page title errors, missing meta description errors, and duplicate content errors. After reviewing the report we presented it to the clients web company who operates a closed source CMS. Their response was that these errors are not worthy of fixing and in fact they are not hurting the site. We are having issues getting the errors fixed and I would like your opinion on this matter. My question is, how bad are these errors? Should we not fix them? Should they be fixed? Will fixing the errors have an impact on our site's rankings? Personally, I think the question is silly. I mean, the errors were found using the SEOmoz tool kit, these errors have to be effecting SEO.....right? The attached image is the result of the Crawl Diagnostics that crawled 1,400 pages. NOTE: Most of the errors are coming from Pages like blog/archive/2011-07/page-2 /blog/category/xxxxx-xxxxxx-xxxxxxx/page-2 testimonials/147/xxxxx--xxxxx (xxxx represents information unique to the client) Thanks for your insight! c9Q33.png
Technical SEO | | Gabe0 -
Domain redirection and seo implications
We have an existing site that is a subdomain but we recently acquired an exact match domain. Will building links to the exact match domain and having the domain point at our existing subdomain work or should we convert the entire site and redirect our existing subdomain to the new domain? What I'm trying to figure out is how to maximize the benefit here and how the existing mass of links pointing to our existing subdomain (shop.domain.com) can be used. New domain: keywordshop.com Existing URL: shop.domain.com
Technical SEO | | CHarkins0 -
Domain authority not showing on root domain?
I was going through our site earlier w/ the mozBar (still learning the tools, new here) and saw the attached image. There were far more links to the subdomain (#s on the left) than the root domain (#s on right). This is strange to me, because we are not using any subdomains. All links point to either our root domain or subfolders off our root domain. Is this hurting our ranking for the root domain? Not sure what's up with this. Zz9j0.jpg
Technical SEO | | askotzko0 -
Linklicious and Crawl rates
Can somebody please explain me what is 'crawl rate' and how does 'linklicious' help us with it? I mean I can always visit the website and know more about it, but I want to understand the concept. Please help.
Technical SEO | | KS__0 -
Domain with more Languages
Hey folks! I was wondering what you would do. I do have a Website. The website is provided in 8 other languages. Right now every language has it's own Domain name. The domain name is always the country in the language. I'm thinking about combine everything to one domain and hope to get some great linkjuice from the other 7 domains. So it would be www.example.com/en/ www.example.com/fr/ and so on. How do you handle that. Would this have a big positive impact on that one domain I'm forwarding to?
Technical SEO | | leitpix
I really think so!0 -
Blocking Google from Crawling Parameters
Hi guys: What is the best way to keep Google from crawling certain urls with parameters? I used the setting in Webmaster Tools, but that doesn't seem to be helping at all. Can I use robots.txt or some other method? Thanks! Some examples are: <colgroup><col width="797"></colgroup> www.mayer-johnson.com/category/assistive-technology?manufacturer=179 www.mayer-johnson.com/category/assistive-technology?manufacturer=226 www.mayer-johnson.com/category/assistive-technology?manufacturer=227 <colgroup><col width="797"></colgroup> www.mayer-johnson.com/category/english-language-learners?condition=212 www.mayer-johnson.com/category/english-language-learners?condition=213 www.mayer-johnson.com/category/english-language-learners?condition=214 <colgroup><col width="797"></colgroup>
Technical SEO | | DanaDV
| www.mayer-johnson.com/category/english-language-learners?roles=164 |
| www.mayer-johnson.com/category/english-language-learners?roles=165 |
| www.mayer-johnson.com/category/english-language-learners?roles=197 | | |0