Measuring the size of a competitors website?
-
I think our website is too big, far too many indexed pages. I'd like to do some research on how big our competitors' websites are (how many indexed pages). Is there a way to do this?
Cheers,
Rhys
-
Thanks, both!
-
Xenu's Link Sleuth is free so you may want to check that out (not sure just now whether there are any limits with regard to website size) but I also recommend Screaming Frog - it's money well-spent, such a feature rich tool!
-
I highly recommend buying the license for Screaming Frog, at $100/year, you won't find a more valuable SEO tool for the money. You won't find a free (and trustworthy) that will crawl a site that large.
-
Hi Alick,
I tried that but I only have the free version so we're capped at 500 URLs. Also, the site:search provided 50,000 results, but I know we don't have that many pages. Are there any other tools?
Cheers,
Rhys
-
Hi,
Do a site: search on Google itself - like "site:google.com" - to return pages of SERPs containing the pages from your competitors site which Google has indexed.
I would also suggest you use screaming frog tool you will get more accurate value here.
Hope this helps.
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will Google crawl and rank our ReactJS website content?
We have 250+ products dynamically inserted and sorted on our site daily (more specifically our homepage... yes, it's a long page). Our dev team would like to explore rendering the page server-side using ReactJS. We currently use a CDN to cache all the content, which of course we would like to continue using. SO... will Google be able to crawl that content? We've read some articles with different ideas (including prerendering): http://andrewhfarmer.com/react-seo/
Technical SEO | | Jane.com
http://www.seoskeptic.com/json-ld-big-day-at-google/ If we were to only load the schema important to the page (like product title, image, price, description, etc.) from the server and then let the client render the remaining content (comments, suggested products, etc.), would that go against best practices? It seems like that might be seen as showing the googlebot 1 version and showing the site visitor a different (more complete) version.0 -
Rel=canonical on Godaddy Website builder
Hey crew! First off this is a last resort asking this question here. Godaddy has not been able to help so I need my Moz Fam on this one. So common problem My crawl report is showing I have duplicate home pages www.answer2cancer.org and www.answer2cancer.org/home.html I understand this is a common issue with apache webservers which is why the wonderful rel=canonical tag was created! I don't want to go through the hassle of a 301 redirect of course for such a simple issue. Now here's the issue. Godaddy website builder does not make any sense to me. In wordpress I could just go add the tag to the head in the back end. But no such thing exist in godaddy. You have to do this weird drag and drop html block and drag it somewhere on the site and plug in the code. I think putting before the code instead of just putting it in there. So I did that but when I publish and inspect in chrome I cannot see the tag in the head! This is confusing I know. the guy at godaddy didn't stand a chance lol. Anyway much love for any replies!
Technical SEO | | Answer2cancer0 -
SEO impact classifieds website
Hi, I'm part of an organization running a classifieds platform in Spain. (Mercadonline.es) We are hit by Google penalties since a few weeks, possibly caused by numerous errors we are experiencing. Most frequent errors are 404's and duplicate content (titles tags etc) since the nature of our website is dynamic. Many ads change daily, are added or removed, causing Googlebots (and others) to flag us and not being able to see our more unique content. Until what part of our platform should we be indexed? Since we have +34,000 pages indexed (mostly due to internal filter pages) I would need a systematic solution for us to display relevant and unique content, with enough usage of keywords that can bring us back up - we are actually ranked <50 on google for most of our main keywords. It is costing us precious time and money since we can only aquire our visitors (adwords etc) and not being to attract any organically. I can go in more detail with someone who can give me a bit more direction. Your answer is much appreciated! Ivor
Technical SEO | | ivordg0 -
Hreflang and canonical for multi-language website
Hi all, We're about to have a new website in different languages and locations, which will replace the existing one. Lets say the domain name is example.com. the US version will be example.com/en-us/ and the UK version will be example.com/en-uk/. Some of the pages on both version share the same content. So in order to solve it, we're about to use hreflang on each page + a canonical tag which will always use the US address as canonical address. My question is - since we are using canonical tag along with hreflang, is there a possibility that a user who is searching with Google.co.uk will get the canonical US address instead of the UK address? Or maybe the search engine will know to display the right localized address since (UK) i've been using hreflang? It is really important for me to know, because i'm afraid we will lose the high rankings that we have right now on google.co.uk. Thanks in Advance
Technical SEO | | Operad-LTD0 -
Duplicate Title - Shopping Website
My website is showing with errors due to duplicate titles, these errors add up to over 150 errors. The problem is a lot of my products are similar, just in different lengths, colors etc. For example
Technical SEO | | adelemaree
http://www.ugg.org.au/en/classic-candy-short-australia-ugg-boots/48-classic-candy-short-australia-ugg-boots-blue.html
http://www.ugg.org.au/en/classic-candy-button-short-australia-ugg-boots/31-classic-candy-button-short-chocolate.html
http://www.ugg.org.au/en/classic-candy-button-short-australia-ugg-boots/28-classic-candy-button-short-black.html How can I avoid this error without having to merge all my products and colors into one page? As that would make my catalogue look too small! Thanks in advance0 -
Keep getting random broken links pointed at our website
Over the past 6 months I've noticed an increase of 404 errors being picked up by Google Webmaster Tools. When I look at the errors table and review these errors the links are invariable made up likes being generated from what looks like scraper sites or sites with search results embedded into them. Below are some examples of them. Broken Link URL Generating the Link www.mysite.com/rwickenhauser http://www.webstatsdomain.com
Technical SEO | | adamlcasey
www.mysite.com/guest http://www.webstatsdomain.com
www.mysite.com/windows http://www.webstatsdomain.com
www.mysite.com/..app http://213.174.143.40/
www.mysite.com/ht.. http://de.aguea.com/ None of these pages exist on our website.
So do I need to setup redirects for these or can they be ignored?0 -
I have found a website on my dedicated server which is not mine
Hi i have found a website on my dedicated server that is not mine by using a number of tools including http://www.yougetsignal.com/tools/web-sites-on-web-server/ the site that i found that was not mine is buycostumes.com i contacted my hosting company and told them and this is the reply i got back We have checked the issue and it appears there have a bug/glitch in this site, because after we checked the mentioned domain we found that this website respond on IP which is not our and is not assisgned in your Dedicated server as you may preview below: Found 3 domains hosted on the same web server as buycostumes.com (66.45.3.44). buycostumes.com (linkback) target.com (linkback) www.certifigroup.com (linkback) Thus if you find any issues on your web server, please mention them in this ticket and we will be glad to provide you with further assistance on this matter. Please feel free to contact us if you have any further technical difficulties. Best regards, Now they have not said if they are going to do anything about this and to be honest i am getting fed up with the hosting company because i am being told that the slow speed i was receiving for my website www.in2town.co.uk was down to it taking a long time to reach my server before reaching my site. Now not sure if this is correct or not but with all the help i have received off semoz i have managed to increase the speed to my site but now i have found this problem. Can anyone tell me if i am being played and also can anyone recommend a professional UK hosting company. Also is this site affecting my spped and my site performance.
Technical SEO | | ClaireH-1848860 -
How to find a specific link on my website (currently causing redirects)
Hi everyone, I've used crawlers like Xenu to find broken links before, and I love these tools. What I can't figure out is how to find specific pieces of code within my site. For example, Webmaster Tools tells me there are still links to old pages somewhere on my website but I just can't find them. Do you know of a crawler that can search for a specific link within the html? Thanks in advance, Josh
Technical SEO | | dreadmichael0