Crawl test tool from SEOmoz - which URLs does it actually crawl?
-
I am using for the first time the crawl test tool from SEOmoz and I do not really understand which URLs the tool is going to crawl.
First, it says "enter any subdomain" --> why can´t I do the crawl for the root domain?
Second it says "we'll crawl up to 3,000 linked-to pages" --> does that mean that the tool crawls all internal links that it can find on the given domain?
Thanks for your help!
-
Hi Elke,
An initial Pro campaign will take up to 7 days to complete and will crawl up to a maximum 10,000 URLs for the root domain or sub domain that you choose to set it up for.
The Crawl Test tool is intended to provide a "quick start " option, which will report against the same diagnostic information, but for a limited number of URLs (3,000 & max of 2 domains per day). This allows you to get started on tidying up site issues while waiting for the full crawl to complete.
Since the Crawl Test tool allows you to run the quick crawl for any subdomain, it can also be useful for taking a look at the health of established sites you may be quoting on etc.
Hope that helps,
Sha
-
Alan, thank you for the answer. I am a Pro Member (trial version). But the only crawl diagnostics that I can find inside my campaign is a short Crawl Diagnostics summary which doesn´t include all the information that the crawl test tool gave me... I have just set up a campaign for the root domain, let´s see if the crawl diagnostics give me more information for that one.
-
When you become a pro meneber you can set up compaigns, then you can set up by root domain and crawl all sub domains.
the free version will crawl only one sudomain, such as
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Url-delimiter vs. SEO
Hi all, Our customer is building a new homepage. Therefore, they use pages, which are generated out of a special module. Like a blog-page out of the blog-module (not only for blogs, also for lightboxes). For that, the programmer is using an url-delimiter for his url-parsing. The url-delimiter is for example a /b/ or /s/. The url would look like this: www.test.ch/de/blog/b/an-article www.test.ch/de/s/management-coaching Does the url-delimiter (/b/ or /s/ in the url) have a negative influence on SEO? Should we remove the /b/ or /s/ for a better seo-performance Thank you in advance for your feedback. Greetings. Samuel
Moz Pro | | brunoe10 -
Has the Crawl Test gone?
Just checked the new Moz, am I right in thinking the super useful crawl test functionality has gone? I use it for existing sites to download all the title tags and meta name descriptions, is there more to come??
Moz Pro | | Karen_Dauncey0 -
Why does SEOMoz think I have duplicate content?
The SEOmoz crawl report shows me a large amount of duplicate content sites. Our site is built on a CMS that creates the link we want it to be but also automatically creates it's own longer version of the link (e.g. http://www.federalnational.com/About/tabid/82/Default.aspx and http://www.federalnational.com/about.aspx). We set the site up so that there are automatic redirects for our site. Google Webmaster does not see these pages as duplicate pages. Why does SEOmoz consider them duplicate content? Is there a way to weed this out so that the crawl report becomes more meaningful? Thanks!
Moz Pro | | jsillay0 -
SEOMoz Q&A having some issues?
Apologies if this has been asked. I was browsing Q&A and got a Ruby error page, reloaded and everything was fine. However a bunch of the topics are kind of messed up, and it looks like a chunk of them are missing now. For example my 'questions I've answered' section of My Q&A only has the newest question and the oldest question, and when I sort the SEOMoz Tools category by newest I get one recent post and then the next one is from November of last year. You guys having database problems?
Moz Pro | | icecarats0 -
Competitive Link finder tool
I tried to use competitive link finder tool. it says that it will identify 10 most important links that my competitor get that I do not. The results that I get are inconsistent with linkscape tool. For example my competitor has a link on www lawyer-links dot info Google rank of that site is 1. Is that really one of their most important links?
Moz Pro | | SirMax0 -
If there a basic overview of the SEOMoz Pro tools?
I'm surprised there is no simple overview of all the features anywhere, or a guide on how to use the tools? There's a nice intro video, and a good beginners guide, then BAM! - straight into the Pro Tools (it's a bit daunting for someone learning!) Or at least that I can find... and help or pointers would be great 🙂
Moz Pro | | seanuk0 -
Handling long URLs and overly-dynamic URLs on eCommerce site
Hello Forum, I've been optimizing an eCommerce site and our SEOmoz crawls are favorable for the most part, except for long URLs and overly-dynamic URLs. These issues stem from two URL types: Layered navigation (faceted search) and non-Google internal search results. I outline the issues for each below. We use an SEO-friendly URL structure for our product category pages, but once bots start "clicking" our layered navigation options, all the parameters are appended to our SEO-friendly urls, causing the SEOmoz crawl warnings. Layered Navigation :
Moz Pro | | pano
SEO-Friendly Category Page: oursite.com/shop/meditation-cushions.html Effects of layered navigation: oursite.com/shop/meditation-cushions.html?bolster_material_quality=414&bolsters_appearance=206&color=12&dir=asc&height=291&order=name As you can see the parameters include product attributes and page sorts. I should note that all pages generated by these parameters use the element to point back to the SEO-friendly URL We have also set up Google's Webmaster Tools to handle these parameters. Internal Search Function:
Our URLs start off simple: oursite.com/catalogsearch/result/?q=brown. Then the bot clicks all the layered navigation options, yielding oursite.com/catalogsearch/result/index/?appearance=54&cat=67&clothing_material=83&color=12&product_color=559&q=brown. Also, all search results are set to noindex,follow. My question is: Should we worry about these overly-dynamic and long ULR warnings? We have set up canonical elements, "noindex,follow" solutions, and configured Webmaster Tools to handle our parameters. If these are a concern, how would you resolve these issues?0 -
Is The Crawl Diagnostic tool working correctly?
The Crawl Diagnostic tool shows issues and displays a graph but they don't display the page specific results/suggestion like it used to. I get the "Congratulations, there are no pages affected by this issue!" message.
Moz Pro | | -PAUL-0