SEOMOZ Crawling Our Site
-
Hi there,
We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist.
Thanks,
Geoff
-
That's a question best answered by a programmer and/or someone who is familiar with the code of your site. Sorry I can't help more!
-
Hi Adam,
Thanks - how do you change the navigation so each page is linked with a single URL?
Thanks
-
If SEOmoz is crawling all those duplicate/unnecessary URLs, the search engines probably are, too.
Best solution: Change the navigation on your site so that each page is linked to using a single URL.
-
Thanks Adam.
Have had a look and it looks like the crawlers are cycling through numerous unnecessarily crawled pages for each product based on the top and bottom nav bar on our website (i.e. it adds FAQ and Shipping Info for each product + all the categories we have created on the website). Attached is an image link of some lines from the CSV file.
I imagine this is what is making it chew up bandwidth. Any insight on how to avoid / change this would be much appreciated.
Thanks
Geoff
-
In addition to what Adam said, you can just search your site on Google and I come up with over 14,000 results:
Both ways should give you a good insight into what's being found.
-
Hi Geoff,
There are multiple reasons your site could have "extra" pages: tag pages, category pages, print this pages, duplicate pages, etc.
To see what pages SEOmoz is crawling:
- Go to http://pro.seomoz.org/campaigns
- Click View This Campaign for your site
- Click Crawl Diagnostics
- At the top right, Export as CSV
- You'll get a spreadsheet listing all the URLs that SEOmoz crawled
Hope that helps!
~Adam
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Open Site Explorer detects links from a site that redirects to it? How is this possible?
I was checking external links to a site: scte-brasilien.de/ and was wondering why there are so many links pointing from another domain (reisen-nach-brasilien.com/) although it redirects to scte-brasilien.de/? So when checking the redirecting domain OSE knows its redirecting... The URL you've entered redirects to another URL. We're showing results for scte-brasilien.de/ since it is likely to have more accurate link metrics. See data forreisen-nach-brasilien.com/ instead? How is this possible? best regards
Moz Pro | | inlinear
Holger0 -
SEOMoz Question
Hi, I have taken over SEO on a real estate site with an internal blog. Unfortunately there are loads of duplicated pages and titles in the blog. It was suggested that all should be rel=canonical so not to show up. In my last crawl here though they still do. So question is if SEOMoz crawls and sees them is Google also seeing them? Also would it be best to move the blog off site so this does not cause anymore damage and just link to it from the main site? Thanks for your comments
Moz Pro | | AkilarOffice0 -
How do YOU use site explorer?
I normally use open site explorer to identify links that competitors of my clients have and sometimes this gives me what I call 'some low hanging fruit' to go after. (and of course links that are more challenging to get) I don't know why this didn't occur to me sooner. If my client is a chiropractor why not look at the links for 50 or 100 of the top rankings chiropractic sites all over the US? This would HAVE to uncover a wealth of blogs to comment on that have good authority, great industry associations, publications, forums - a whole wealth of items. It made me wonder how many people use site explorer like I have been (top 3-4 competitors that your client has) or identifying links pointing to LOTS of competitors? How do you use it? Couldn't you almost base an entire link building campaign using OSE? Why would this be a bad idea if not? Just some random thoughts. THE WEEKEND IS ALMOST HERE - Have a great day everybody! 🙂
Moz Pro | | Mrupp441 -
Bug with seomoz level display?
Is anyone else seeing people's moz levels shown incorrectly? Spotted this yesterday, but I am getting the same today. Lots of people showing as Aspirant who shouldn't be.
Moz Pro | | matbennett1 -
Ranking Differs between SEOmoz and Google Webmaster?
My Google webmaster ranking for keywords differs significantly from SEOmoz. For example, Webmaster say 8 and SEOmoz says 1...thoughts?
Moz Pro | | mjcarrjr0 -
SEOMOZ can we trust you?
I was thinking today what are the chances that SEOMOZ can look into my campaigns and see what keywords I am targeting, strategies etc. and then sell that information to others? There has to be some type of clause when you sign up correct? But who knows because I'm not sure. Having Rand and Matt Curts talking via twitter to each other kind of concerns me. Not that I'm worried that Google will know my strategies because we don't do any black-hat linking and anything shady but how much power does SEOMOZ really have and can they leak any information to other companies or Google? Scary stuff that's why I don't put all my eggs into one basket, I spread them out. Thoughts? Thanks in advance - Brett Shaffer
Moz Pro | | EmpireSEO1 -
SEOmoz crawler bug?
I just noticed that a few of my campaigns have number of pages crawled 1. Can someone tell me what this is.... from 5 campaigns 2 have only one pages crawled from which one is an online shop with over 2000 products 🙂
Moz Pro | | mosaicpro0 -
How to force a recrawl of a site?
Hi, I made changes in my site. I would like to see the result of the crawl diagnostic. I know the crawl is happening every week, however, is there a way to force a re-crawl in order not to have to wait 5 days? Cheers,
Moz Pro | | nuxeo0
