Tools that crawl 2 million page sites
-
Our site is about 2million pages deep, 50% of which is stale content. Yes, I know - OMG #unhygienic. Even if we get approval to get rid of half of it. SEOMoz Pro Elite only crawls 20k deep - what can i do to crawl and diagnose the whole site. Are there any tools anyone can suggest. SEOMoz??
-
That's good to know. It sounds like that's probably the best way. I also use Screaming Frog (http://www.screamingfrog.co.uk/seo-spider/) to try and crawl sites and with dedicated 2Gigs of ram, it's able to crawl around 50k pages. If your site is structured in sub-folders, you might be able to break it into parts and then crawl. But then if not, the SEOMOZ Enterprise looks like the way to go.
-
There is an enterprise version of SEOmoz which will do 1 million pages a month and up to 30k keywords which is well worth looking into if you have a enormous web property.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Web Stie STUCK on page 2 for 2years
Our website is a dating site for adults (not porn) (http://www.swingersocial.com) It’s been STUCK in the middle of page 2 for 1 ½ years. Moving up or down 2 ranking, that’s it. The way we determine our location is by entering the most common phrases used by people who search for our type of site. We surveyed 50+ people and the phrases they most frequently entered when using Google were “swingers websites” “swingers sites” and “free swingers sites”. In addition to our competitors ranking above us, they show “movement” in their rankings up & or down. Some even dropped to page 2 but they recovered – we are not seeing the same movement or recovery. Pages crawled on have remained virtually unchanged for months. The last 1 ½ years we have hired SEO companies, and as a result a great deal of effort has been put into the site. We are not naïve and realize some were better than others. We’ve invested a great deal of effort into content, key words, modifying tags, and throughout all of this we’ve seen no change. We had a detailed audit suggesting improvements (no surprise), the report overall was positive stating “There are a few things to do to optimize the site but looking at the keywords, SwingerSocial.com should be able to rank high.” We also implemented a blog area with over 70 blog posts. Our competitors on the other hand have only two blog posts and are continually ranking higher than us. Additionally, our competitors do not have sites on Pinterest, G+, FB, or Twitter – we have a presence on some of these. I do know that the number and quality of links are terrible. I’m looking for a partner to help us make strategic changes to the site that will allow us to rank higher.
Moz Pro | | sailor35700 -
Campaigns - crawled
The new Pages Crawled: 2. I have many 404 and other errors, I wanted to start working on it tomorrow but the new crawl only crawled to pages and doesn't show any errors. Whats the problem and what can I do? Yoseph
Moz Pro | | Joseph-Green-SEO0 -
Domain and Page authority dropped on home page
My two strongest links did not show up in the latest Open Site Explorer run. Business.com and BOTW.com...Linking domains went from 21 to 15. Domain dropped 3 pts and Page authority dropped 3 pts I believe. Is there a function in SEOmoz.org that lets you track your domain scores over time? Thanks, Boo
Moz Pro | | Boodreaux0 -
Tool Request - What keywords does a site rank for?
Hi folks, Something I've never had to do before so I'm not sure which tool to use, but is there a way to determine the keywords that a website currently ranks for? Hope someone can assist 🙂
Moz Pro | | ChristopherM1 -
Rank Checker Tool
How many keywords/domains can I track using this tool on the Pro membership? Looking at the tool itself, it looks like 100 but in the 'help' section it states up to 300?
Moz Pro | | dentaldesign0 -
Crawl Diagnostic | Starter Crawl taken 14hrs.. so far
We started a starter crawl 14hrs ago and it's still going, can anyone help on why this is taking so long, when it says '2 hrs' on the interface.. Thanks, Rory
Moz Pro | | RoryMacDonald0 -
Is there a tool to show me exactly how a search engine sees my pages?
Our site is very graphics heavy and I know the search engines aren't seeing it as humans do. Is there a tool (almost like Wirify) that can show me exactly how Google sees my page(s)? Thanks in advance. 🙂
Moz Pro | | askotzko0 -
SEomoz slow to crawl?
Hello - I am just trying out the trial and it said the next crawl was nov 1st but I see no change in any of the errors since the initial crawl... so just waiting to find out if what I changed was fixed or not. Is this normal ?
Moz Pro | | Bethany_BabyBrowns0