Xenu Alternative for Large Sites
-
We're launching a new site and we're trying to crawl it to check for any problems. It's millions of pages and Xenu seems to start encountering errors as the numbers mount past 500,000. Does anyone know of an alternative, free or paid, that could handle the size better?
-
There's a post in the last 24 hours about Xenu and Screaming Frog, it's worth a read to see if a) screaming frog my work for you and b) if the 64 bit version of Xenu mentioned in the comments can solve any problems.
http://www.seomoz.org/blog/crawler-faceoff-xenu-vs-screaming-frog
-
WEB CEO and seopowersuite try them if it doesnt work: google webmaster tools but you will have to wait until it crawls your site, and not last thing seomoz crawling it has 10 000 limit minimum but it will be enougth to detect major problems
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Backlinks from an Association Site
My company is joining an Industrial Association. Part of the membership is a link to our site from theirs. I've found that going to their site triggers a "threat alert" through our company malware detection system and shows a link that may be infected with malware. With all of that said I have (2) questions... Since this is a paid membership, will Google penalize us for having a link to our company from this association's website? Since a link on their site has potential malware issues, should we add our link to their site or could it be harmful to us? Any helpful advice is appreciated.
Technical SEO | | SteveZero121 -
How to Switch My Site to HTTPS in GWT?
I recently bought an SSL certificate and moved my site over to HTTPS. Now how do I make the change in Google Webmaster Tools?
Technical SEO | | sbrault740 -
Bringing a large news site back on line - anything to look out for?
Hi, I'm advising an online news site site that has been completely offline for almost 6 months, and is now looking to start back up again. The site seems to be completely gone from google's cache. This might mean moving to new hosting, but with the same URL. The archive has about 7000 original articles. Most of these are date specific news, although there are some longer investigative pieces that are more timeless. Is there any difference (from an SEO/digital marketing perspective) between putting the whole archive online at once, or gradually republishing the old articles? Is there anything I should be aware of, when restarting a website of this size? Thanks - Chris
Technical SEO | | AISFM0 -
Site architecture & breadcrumbs
Hi A client hasn't structured site architecture in a silo type format so breadcrumbs are not predicating in a topical hierarchy as one would desire (or at least i think one would prefer) For example: say the site is called www.fruit.com and it has a category called 'types of fruit' and then sub/content pages called things like 'apples' and 'pears'. So in terms of architecture that should be: www.fruit.com/types-of-fruit/apples and www.fruit.com/types-of-fruit/pears etc etc The client has kept it all flat so instead architecture is: www.fruit.com/types-of-fruit and www.fruit.com/apples and www.fruit.com/pears As a result breadcrumbs follow suit and hence since also not employing logical predication dont reflect the topical & sub-topical hierarchy I have seen that some seo's at least used to think this was better for seo since kept the page/s nearer the root but surely its better to structure site architecture in a logical topical hierarchy so long as dont go beyond say 3 or 4 directories/forward slashes in the url's? Also is it theoretically possible to keep url structure as is (flat) and just edit/customise the breadcrumbs to reflect a topical hierarchy in a silo structure rather than change the entire site architecture & required 301'ing etc in order to do this (or is that misleading or just not possible?) Cheers Dan
Technical SEO | | Dan-Lawrence0 -
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Please advise.
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Are there any other precautions I should be taking? Please advise.
Technical SEO | | BVREID0 -
Redirect from old wordpress site to new php site? Best approach
Hi I have two websites one legacy site done in wordpress the other in php. However I would like to merge the two together and remove the wordpress site. However it has a good link profile and the pages rank well. What is the best approach to do a 301 redirect from the old site with all its pages pointing to the homepage of the new site? If so what's the best way to do this in wordpress? Many thanks
Technical SEO | | ocelot0 -
Does 301 redirecting a site multiple times keep the value of the original site?
Hi, All! If I 301 redirect site www.abc.com to www.def.com, it should pass (almost) all linkjuice, rank, trust, etc. What happens if I then redirect site www.def.com to www.ghi.com? Does the value of the original site pass indefinitely as long as you do the redirects correctly? Or does it start to be devalued at some point? If anyone's had experience redirecting a site more than once and they've seen reportable good/bad/neutral results, that would be very helpful. Thanks in advance! -Aviva B
Technical SEO | | debi_zyx0 -
Problem with my site
the site is casino.pt we created the site 7-8 month ago, we started to push it by good and natural links (http://www.opensiteexplorer.org/www.casino.pt/a!links!!filter!all!!source!external!!target!page), links in sites with content rich and most of them related to gambling and sport topics. During the first 3-5 months, the rankings were better and better, after the 6 months, the site lose all its rankings. Aditional details http://www.casino.pt/robots.txt http://www.google.pt/#hl=pt-PT&source=hp&biw=1280&bih=805&q=site:http%3A%2F%2Fwww.casino.pt&aq=f&aqi=&aql=&oq=&fp=2651649a33cd228 no critical errors in google webmaster tools any idea how can I fix it? thanks
Technical SEO | | Yaron530