SEOMOZ Crawling Our Site
-
Hi there,
We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist.
Thanks,
Geoff
-
That's a question best answered by a programmer and/or someone who is familiar with the code of your site. Sorry I can't help more!
-
Hi Adam,
Thanks - how do you change the navigation so each page is linked with a single URL?
Thanks
-
If SEOmoz is crawling all those duplicate/unnecessary URLs, the search engines probably are, too.
Best solution: Change the navigation on your site so that each page is linked to using a single URL.
-
Thanks Adam.
Have had a look and it looks like the crawlers are cycling through numerous unnecessarily crawled pages for each product based on the top and bottom nav bar on our website (i.e. it adds FAQ and Shipping Info for each product + all the categories we have created on the website). Attached is an image link of some lines from the CSV file.
I imagine this is what is making it chew up bandwidth. Any insight on how to avoid / change this would be much appreciated.
Thanks
Geoff
-
In addition to what Adam said, you can just search your site on Google and I come up with over 14,000 results:
Both ways should give you a good insight into what's being found.
-
Hi Geoff,
There are multiple reasons your site could have "extra" pages: tag pages, category pages, print this pages, duplicate pages, etc.
To see what pages SEOmoz is crawling:
- Go to http://pro.seomoz.org/campaigns
- Click View This Campaign for your site
- Click Crawl Diagnostics
- At the top right, Export as CSV
- You'll get a spreadsheet listing all the URLs that SEOmoz crawled
Hope that helps!
~Adam
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Joined SEOMOZ but non changed.
hi. i am new SEOMoz and i joined pro monthly and create 2 campagnes but there are every thing shows me 0 and i guess nothing changed with before.. please see this.. | | StickerApt |
Moz Pro | | bratt
| Domain Authority | 1 |
| Domain MozRank | 0.00 |
| Domain MozTrust | 0.00 |
| External Followed Links | 0 |
| Total External Links | 0 |
| Total Links | 0 |
| Followed Linking Root Domains | 0 |
| Total Linking Root Domains | 0 |
| Linking C-Blocks | 0 |
| Followed Links
vs
NoFollowed Links****Followed Linking Root Domains
vs
NoFollowed Linking Root Domains | | ubdomain Metrics | canadastickerking | stickeryou | stickybusiness |
| 4.31 | Transparent 5.26 | 4.56 |
| 3.78 | Transparent 5.43 | 4.87 |
| 71 | Transparent 38,649 | 631 |
| 73 | Transparent 38,814 | 1,265 |
| 3,805 | Transparent 235,124 | 26,337 |
| 7 | Transparent 243 | 115 |
| 7 | Transparent 286 | 161 |
| | | | | | StickerApt |
| Subdomain MozRank | 0.00 |
| Subdomain MozTrust | 0.00 |
| External Followed Links | 0 |
| Total External Links | 0 |
| Total Links | 0 |
| Followed Linking Root Domains | 0 |
| Total Linking Root Domains | 0 |
| Followed Links
vs
NoFollowed Links****Followed Linking Root Domains
vs
NoFollowed Linking Root Domains | "No followed (0%)"
"No nofollowed (0%)" | Total Branded Keywords Manage brand rules Non-branded Keywords Week ending: 9/9 Change 9/16 9/9 Change 9/16 9/9 Change 9/16 --- --- --- --- --- --- --- --- --- --- Organic Search Visits 45 -18% 37 7 -14% 6 38 -18% 31 URLs Receiving Entrances Via Search 8 25% 10 4 -50% 2 4 100% 8 | Non-Paid Keywords Sending Search Visits | 20 | -10% | 18 | 6 | -33% | 4 | 14 | 0% | 14 | can any one help me what should i do with SEOMoz? all the keyword i set up tells me not in top 50 ranking.. i set up 12keyword for start.. and aslo i there is craw erros on my site but it can not be fixed because the page that getting erros are automatic quote pages and made php. and it should be duplicate pages it share with other pages, hard to explain.. but you will see what i am talking about it has duplicate title and content.. it should be like that.. http://www.stickerapt.com/quote.php i can not change duplicate erros is it gonna effect page rank? please help..ㅡㅡ0 -
How do I force a crawl?
In the campaign overview it reads that 0 pages were crawled. Also got an email saying that a comprehensive audit will be done in 7 days. But the 'crawl in progress' wheel disappeared. I think it stopped, and I need to submit that report to substantiate buying the tool! How do I force a crawl?
Moz Pro | | ilhaam0 -
Open Site Explorer Update
What is taking OSE so long to update? The update schedule said the next update was going to be on Dec 28th.
Moz Pro | | Robbie8299
If you open OSE it says "Last Index Update: November 28th, 2011" Today in January 1st. Any thoughts as to why the delay?0 -
Crawl Issues
My website - qtmoving.com - has 26 articles and when the SEOmoz did a crawl it only found 13 articles. Can someone please give me some insight as to why not all pages are being crawled.
Moz Pro | | CohesiveMarketing0 -
What the hell...spam on SEOMOZ!
I received this in my Private Messages section: My name is Fatima,i saw your profile at/www.seomoz.org/today and became intrested in you,i will also like to know you the more,and i want you to send an email to my email address so i can give you my picture for you to know whom i am.Here is my email address (fatimababy06@yahoo.com) I believe we can move from here I am waiting for your mail to my email address above.Fatima(Remeber the distance or colour does not matter but love matters alot in life) How can somebody spam like this on protected forum?
Moz Pro | | IM_Learner2 -
What causes Crawl Diagnostics Processing Errors in seomoz campaign?
I'm getting the following error when seomoz tries to spider my site: First Crawl in Progress! Processing Issues for 671 pages Started: Apr. 23rd, 2011 Here is the robots.txt data from the site: Disallow ALL BOTS for image directories and JPEG files. User-agent: * Disallow: /stats/ Disallow: /images/ Disallow: /newspictures/ Disallow: /pdfs/ Disallow: /propbig/ Disallow: /propsmall/ Disallow: /*.jpg$ Any ideas on how to get around this would be appreciated 🙂
Moz Pro | | cmaddison0 -
Only Crawling 1 page?
Hi Guys, Any advice much appreciated on this! Recently set up a new campaign on my dashboard with just 5 keywords. The domain is brammer.co.uk and a quick Google site:brammer.co.uk shows a good amount of indexed pages. However - first seomoz tool crawl has only crawled 1 url!! "Last Crawl Completed: Apr. 12th, 2011 Next Crawl Starts: Apr. 17th, 2011" Any ideas what's stopping the tool crawl anymore of the site?? Cheers in advance.. J
Moz Pro | | lovealbatross0
