Tools that crawl 2 million page sites
-
Our site is about 2million pages deep, 50% of which is stale content. Yes, I know - OMG #unhygienic. Even if we get approval to get rid of half of it. SEOMoz Pro Elite only crawls 20k deep - what can i do to crawl and diagnose the whole site. Are there any tools anyone can suggest. SEOMoz??
-
That's good to know. It sounds like that's probably the best way. I also use Screaming Frog (http://www.screamingfrog.co.uk/seo-spider/) to try and crawl sites and with dedicated 2Gigs of ram, it's able to crawl around 50k pages. If your site is structured in sub-folders, you might be able to break it into parts and then crawl. But then if not, the SEOMOZ Enterprise looks like the way to go.
-
There is an enterprise version of SEOmoz which will do 1 million pages a month and up to 30k keywords which is well worth looking into if you have a enormous web property.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Site Crawl Stalled and Can't Restart
In my GreenSeed campaign, the site crawl continues to say "in progress." I can't figure out how to stop it or how to restart the site crawl. Can you please help?
Moz Pro | | Winger1 -
Web Stie STUCK on page 2 for 2years
Our website is a dating site for adults (not porn) (http://www.swingersocial.com) It’s been STUCK in the middle of page 2 for 1 ½ years. Moving up or down 2 ranking, that’s it. The way we determine our location is by entering the most common phrases used by people who search for our type of site. We surveyed 50+ people and the phrases they most frequently entered when using Google were “swingers websites” “swingers sites” and “free swingers sites”. In addition to our competitors ranking above us, they show “movement” in their rankings up & or down. Some even dropped to page 2 but they recovered – we are not seeing the same movement or recovery. Pages crawled on have remained virtually unchanged for months. The last 1 ½ years we have hired SEO companies, and as a result a great deal of effort has been put into the site. We are not naïve and realize some were better than others. We’ve invested a great deal of effort into content, key words, modifying tags, and throughout all of this we’ve seen no change. We had a detailed audit suggesting improvements (no surprise), the report overall was positive stating “There are a few things to do to optimize the site but looking at the keywords, SwingerSocial.com should be able to rank high.” We also implemented a blog area with over 70 blog posts. Our competitors on the other hand have only two blog posts and are continually ranking higher than us. Additionally, our competitors do not have sites on Pinterest, G+, FB, or Twitter – we have a presence on some of these. I do know that the number and quality of links are terrible. I’m looking for a partner to help us make strategic changes to the site that will allow us to rank higher.
Moz Pro | | sailor35700 -
OSE stats for 2 site: searoundus.org and www.seaaroundus.org.
Why are the numbers so different for the two site, one with and one without the www.? Which one is most accurate for external linking domains, for instance?
Moz Pro | | GaryDC0 -
Page authority questions?
I've been analyzing some IT communities ...in order to check how relevant is the page authority vs PageRank. I found one main site which is organized by "communities'..and every community is a sub-domain. The root domain has an authority of 90/100 which it should be great......so the sub-domains "inherit" part of this authority.... Until here everything seems to be perfect. However, I went deeper and I picked one of these communities. Analyzing the "Linking Root Domain" I discovered it only has only 5 root domains pointing to its home page. Those 5 Root Domains have generated more than 134k links. That doesn't seem to be "natural". Checking those 5 Root Domains I discovered that they have been registered by the same Root Domain site. Ex: Main domain: Domain.com Community1.domain.com Community2.domain.com.... Linking Root Domains: DomainXY.com DomainABC.com DomainRST.com DomainFGH.com DomainOPQ.com It seems to me that it is easy to cheat the authority domain score. Just creating others sites developing the same topic and generating back links to your main domain
Moz Pro | | SherWeb0 -
Drop in number of Pages crawled by Moz crawler
What would cause a sudden drop in the number of pages crawled/accessed by the Moz crawler? The site has about 600 pages of content. We have multiple campaigns set up in our Pro account to track different keyword campaigns- but all for the same domain. Some show 600+ pages accessed, while others only access 7 pages for the same domain. What could be causing these issues?
Moz Pro | | AllaO0 -
Why won't scheduled crawl of my site begin?
I currently have a campaign running on SEOMoz for over a month. It has been showing that a crawl was scheduled to start on 12/21. Now it's 12/23 and there has not been a new crawl, and it still says scheduled for 12/21.. Anyone know why this is happening or how to fix it? Thanks
Moz Pro | | Prime850 -
Is there a tool to upload multiple URLs and gather statistics and page rank?
I was wondering if there is a tool out there where you can compile a list of URL resources, upload them in a CSV and run a report to gather and index each individual page. Does anyone know of a tool that can do this or do we need to create one?
Moz Pro | | Brother220 -
Ultimate Ranking Tool integrating Analytics / Adwords / Google WM Tools
I currently use SEOMOZ Campaigns and Advanced Web Ranking for monitoring our KW rankings and those of competition. AWR is a brilliant tool with so many different reports, methods of viewing etc. SEOMOZ campaigns are good but don't come close to the monitoring power of AWR (EG I monitor over 50 competitors on over 1000 KW's on a Daily basis with AWR and recieve a variety of set emailed reports on the data). However, one thing that SEOMOZ campaigns have that is useful is the traffic data - but this is still a bit basic and I think could be improved. The problem with AWR is that it doesn't integrate with your Analytics / Adwords / Google WM Tools - so it is only showing you half the picture. Knowing how your site ranks for each keyword is helpful, but it would be nice to understand the value of each keyword. For example, being able to see your rank position and how much traffic that keyword has sent you over time would be helpful. It would also be nice to see the number of searches that are performed for that keyword each month . For example, lets say I saw that I was ranking at number 11 for “hover mower” and getting 500 hits per month. Two months from now, if I was ranking at position 7, it would be nice to be able to immediately see how that changed the amount of traffic I was receiving for the term. Is a position of 11 (first item on page two) better than position 10 (last item on page one)? If you can link it to your analytics, you could then link it to your goals, and goal values to get a complete picture of where your keywords rank the value of the rank, and the improvment on that value when rank changes. If browsed around for such software but can't find anything like this - does anyone know of any software that can do this - or something close to this? Many thanks
Moz Pro | | James770