SEOMOZ Crawling Our Site
-
Hi there,
We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist.
Thanks,
Geoff
-
That's a question best answered by a programmer and/or someone who is familiar with the code of your site. Sorry I can't help more!
-
Hi Adam,
Thanks - how do you change the navigation so each page is linked with a single URL?
Thanks
-
If SEOmoz is crawling all those duplicate/unnecessary URLs, the search engines probably are, too.
Best solution: Change the navigation on your site so that each page is linked to using a single URL.
-
Thanks Adam.
Have had a look and it looks like the crawlers are cycling through numerous unnecessarily crawled pages for each product based on the top and bottom nav bar on our website (i.e. it adds FAQ and Shipping Info for each product + all the categories we have created on the website). Attached is an image link of some lines from the CSV file.
I imagine this is what is making it chew up bandwidth. Any insight on how to avoid / change this would be much appreciated.
Thanks
Geoff
-
In addition to what Adam said, you can just search your site on Google and I come up with over 14,000 results:
Both ways should give you a good insight into what's being found.
-
Hi Geoff,
There are multiple reasons your site could have "extra" pages: tag pages, category pages, print this pages, duplicate pages, etc.
To see what pages SEOmoz is crawling:
- Go to http://pro.seomoz.org/campaigns
- Click View This Campaign for your site
- Click Crawl Diagnostics
- At the top right, Export as CSV
- You'll get a spreadsheet listing all the URLs that SEOmoz crawled
Hope that helps!
~Adam
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My moz only one page was crawled
I recently moved my shopping cart from one provider to another and today moz only crawled one page, could this be because maybe google has not indexed it yet or should i be concerned? I pointed the DNS at the new cart monday night if that helps. I would have expected it to be indexed by now
Moz Pro | | SmartVapes0 -
Not provided Data in SEOMoz reports
Hi How do SeoMoz reports deal with 'not provided' data I see my total visits from organic search for a month are same as total of both my branded and non branded keyword traffic combined yet GA is reporting 157 visits from non provided data so is SEOMoz being very clever and finding a way to decipher this not provided data and allocate it accordingly in the reports ? Or if not what ? Many Thanks Dan
Moz Pro | | Dan-Lawrence0 -
Why is my site not ranking?
I have done various link building strategies, and used all white hat seo methods. SEOMoz even shows that my site is better than all my competition, but yet it doesn't rank ANYWHERE on Google. Can someone point me in the right direction as to why this is happening? I have attached the opensitexplorer screenshot showing my competition. 2013-02-13_1456_zpsb86ab0e1.png
Moz Pro | | locallyrank0 -
Is my Site Spam?
Recently google dropped our site a big time. Can some body tell me if my site is spammy. Our visibility was 67% and one of our top competitor had the visibility of 72%. www.aa-rental.com
Moz Pro | | tanveer10 -
SEOMoz Q&A having some issues?
Apologies if this has been asked. I was browsing Q&A and got a Ruby error page, reloaded and everything was fine. However a bunch of the topics are kind of messed up, and it looks like a chunk of them are missing now. For example my 'questions I've answered' section of My Q&A only has the newest question and the oldest question, and when I sort the SEOMoz Tools category by newest I get one recent post and then the next one is from November of last year. You guys having database problems?
Moz Pro | | icecarats0 -
Is it possible to exclude pages from Crawl Diagnostic?
I like the crawl diagnostic but it shows many errors due to a forum that I have. I don't care about the SEO value of this forum and would like to exclude any pages in the /forum/ directory. Is it possible to add exclusions to the crawl diagnostic tool?
Moz Pro | | wfernley2 -
How to Link build for our site
Hello, A client of mine has a site that gives advice to people. It's moderately competitive. It's a content site. Using Open Site explorer, I looked up competitor's top pages and backlinks, and there were things such as sites that were heavy on directories, sites heavily on editorial mension, sites on juicy topics that I don't want to cover, and the rest was pretty much junk. Just so that you know, I only link build with quality websites that look strong, long term, have a nice design, are high quality, and are not a directory (with a few exceptions) What's the standard way to link build with a site like this? We've got 7 articles of good content that is from me spending hours learning about the subject and using my skills as a life coach. Right now I'm just adding quality content and social media. I am not a top expert in the field, but I'm learning as much as possible. How do we link build?
Moz Pro | | BobGW0 -
What tools should i use to research the following site
Hi i am researching the following site http://www.telegraph.co.uk/travel/ and i would like to know what keywords they are using to gain traffic and i would like to know what tools i need to use to research how they are achieving great success with google. i am new to the semoz tools so i would like help in using them to study this site and other sites, can anyone please let me know what i need to do and what tools i need to use to research this site and find the links to the site and how they are achieving great success many thanks
Moz Pro | | ClaireH-1848861