Measuring the size of a competitors website?
-
I think our website is too big, far too many indexed pages. I'd like to do some research on how big our competitors' websites are (how many indexed pages). Is there a way to do this?
Cheers,
Rhys
-
Thanks, both!
-
Xenu's Link Sleuth is free so you may want to check that out (not sure just now whether there are any limits with regard to website size) but I also recommend Screaming Frog - it's money well-spent, such a feature rich tool!
-
I highly recommend buying the license for Screaming Frog, at $100/year, you won't find a more valuable SEO tool for the money. You won't find a free (and trustworthy) that will crawl a site that large.
-
Hi Alick,
I tried that but I only have the free version so we're capped at 500 URLs. Also, the site:search provided 50,000 results, but I know we don't have that many pages. Are there any other tools?
Cheers,
Rhys
-
Hi,
Do a site: search on Google itself - like "site:google.com" - to return pages of SERPs containing the pages from your competitors site which Google has indexed.
I would also suggest you use screaming frog tool you will get more accurate value here.
Hope this helps.
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website crawl error
Hi all, When I try to crawl a website, I got next error message: "java.lang.IllegalArgumentException: Illegal cookie name" For the moment, I found next explanation: The errors indicate that one of the web servers within the same cookie domain as the server is setting a cookie for your domain with the name "path", as well as another cookie with the name "domain" Does anyone has experience with this problem, knows what it means and knows how to solve it? Thanks in advance! Jens
Technical SEO | | WeAreDigital_BE0 -
Migrating micro site into existing website
My company is planning to migrate an existing (ecommerce) micro site - which sits on its own domain - into their main ecommerce site. This means that the content will be moved from www.microdomain.co.uk to www.maindomain.com/category. Some products already exist on the main domain. The micro site is fairly small with just over 400 pages - I am planning to map each URL to the new URL (exact corresponding page) and create 301 redirects for each. Where any additional content does not exist yet on the existing main domain, we will create it and 301 redirect to it. The micro site currently ranks fairly well for some keywords - being such a specialised micro site, (some of) the keywords also form part of the domain name, however, they won't on the main page although they may form part of the URL (category). As an example (using a made up URL), our micro site www.bread-sticks.co.uk ranks on page 1 for the keyword bread sticks - we don't just sell bread sticks on www.bread-sticks.co.uk but also rolls and bread though, bread sticks is one category of very closely related categories. Say our main domain is www.supermarket.co.uk (selling a wide range of food / drink products. The micro site will be moving to www.supermarket.co.uk/baked-products/ - which is a category. Within that category, there are sub categories, i.e. bread sticks, rolls and bread which will sit under www.supermarket.co.uk/bread-sticks/ etc. What would be the best way for ensuring that our main domain would take over the rankings from our micro site, given that it will be sitting on our main domain as a category (one of many)? Can we expect www.supermarket.co.uk/baked-products/ or www.supermarket.co.uk/bread-sticks/ to replace www.bread-sticks.co.uk in the rankings simply by 301 redirecting? Thanks for your help!
Technical SEO | | ViviCa10 -
Mobile website indexing
Hi we have a mobile version of our website at mobile.gardening-services-edinburgh.com its been live for 5, maybe 6 months, it has its own mobile-sitemap.xml have tried submitting this sitemap to google and for some reason it does not index these pages any ideas, most welcome
Technical SEO | | McSEO0 -
Is there a maximum sitemap size?
Hi all, Over the last month we've included all images, videos, etc. into our sitemap and now its loading time is rather high. (http://www.troteclaser.com/sitemap.xml) Is there any maximum sitemap size that is recommended from Google?
Technical SEO | | Troteclaser0 -
Duplicate content. Wordpress and Website
Hi All, Will Google punish me for having duplicate blog posts on my website's blog and wordpress? Thanks
Technical SEO | | Mike.NW0 -
Website redirects
We consolidated websites. All the international sites have been brought under the roof of our mothership site based in the US: www.crisisprevention.com ... We mapped out all of the URLs and where they should be redirected. However, if someone types in, say, www.crisisprevention.co.uk it redirects to the mothership site, BUT the old URL hangs around no matter what page you navigate to. I feel like it has duplicate content ramifications or worse. I would like opinions on this, so I can take my findings to IT and figure out a solution. Here’s another example: http://www.positive-options.co.uk and another http://www.positive-options.com
Technical SEO | | spackle0 -
Should I change hosting / registration when buying an established website?
I am buying an established website that is currently ranking very well for several keywords. I have no intentions of changing anything on the site. My question is, will it matter if I change where the site is hosted, and the whois information (I want to change it from public to private)? i.e., I don't want to do anything to mess up the current rankings of the site - will these changes have any effect? Thanks for the help.
Technical SEO | | red6marketing0 -
Mobile website settings - I am doing right?
Hi, http://www.schicksal.com has a "normal" and a "mobile' version. We are using a browser detection routine to redirect the visitor to the "default site" or the "mobile site". The mobile site is here:
Technical SEO | | GeorgFranz
http://www.schicksal.com/m The robots.txt contains these lines: User-agent: *
Allow: / User-agent: Googlebot
Disallow: /m
Allow: / User-agent: Googlebot-Mobile
Disallow: /
Allow: /m Sitemap: http://www.schicksal.com/sitemaps/index So, the idea is: Only allow the Googlebot-Mobile Bot to access the mobile site. We have also separate sitemaps for default and mobile version. One of the mobile sitemap is here My problem: Webmaster tool is saying that Google received 898 urls from the mobile sitemap, but none has been indexed. (Google has indexed 550 from the "web sitemap".) I've checked the webmaster tools - no errors on the sitemap. So, if you are searching at google.com/m - you are getting results from the default web page, but not the mobile version. This is not that bad because you will be redirected to the mobile version. So, my question: Is this the "normal" behaviour? Or is there something wrong with my config? Would it be better to move the mobile site to a subdomain like m.schicksal.com? Best wishes, Georg.0