SEOMOZ Crawling Our Site
-
Hi there,
We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist.
Thanks,
Geoff
-
That's a question best answered by a programmer and/or someone who is familiar with the code of your site. Sorry I can't help more!
-
Hi Adam,
Thanks - how do you change the navigation so each page is linked with a single URL?
Thanks
-
If SEOmoz is crawling all those duplicate/unnecessary URLs, the search engines probably are, too.
Best solution: Change the navigation on your site so that each page is linked to using a single URL.
-
Thanks Adam.
Have had a look and it looks like the crawlers are cycling through numerous unnecessarily crawled pages for each product based on the top and bottom nav bar on our website (i.e. it adds FAQ and Shipping Info for each product + all the categories we have created on the website). Attached is an image link of some lines from the CSV file.
I imagine this is what is making it chew up bandwidth. Any insight on how to avoid / change this would be much appreciated.
Thanks
Geoff
-
In addition to what Adam said, you can just search your site on Google and I come up with over 14,000 results:
Both ways should give you a good insight into what's being found.
-
Hi Geoff,
There are multiple reasons your site could have "extra" pages: tag pages, category pages, print this pages, duplicate pages, etc.
To see what pages SEOmoz is crawling:
- Go to http://pro.seomoz.org/campaigns
- Click View This Campaign for your site
- Click Crawl Diagnostics
- At the top right, Export as CSV
- You'll get a spreadsheet listing all the URLs that SEOmoz crawled
Hope that helps!
~Adam
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are the SEOMoz queries banned by Google?
Hi: In our company, we have notified that Google is banning our IP due to massive queries under the same IP. It is caused mainly by local softwares installed in PCs or some plugins of Firefox. We think that SEOmoz don´t causes this due to its tool uses its own IP and servers, not ours. We would like your opinion about this. Thank you,
Moz Pro | | Equipo_SEO_HUB_Interaction0 -
Only few pages (308 pages of 1000 something pages) have been crawled and diagnosed in 4 days, how many days till the entire website is crawled complete?
Setup campaign about 4-5 days ago and yesterday rogerbot said 308 pages were crawled and the diagnostics were provided. This website has over 1000+ pages and would like to know how long it would take for roger to crawl the entire website and provide diagnostics. Thanks!
Moz Pro | | TejaswiNaidu0 -
SEOMOZ Crawl Test
Guys I really have an issue that i know have but cannot see if that makes sense. Basically 3 months ago i did a site wide 301 from economyleasinguk.co.uk to www.economy-car-leasing.co.uk Every thing looks good get all the correct header responses , all canonicals work perfectly , Google webmaster tools is updated fetch as google bot shows the old site is 301 I tried the seomoz crawl test today on the old domain and got this message Oh no! Looks like the page you were trying to access is temporarily down which at first thought ok because the site was not there it wont do it on an old 301 domain, however i tried it on a domain i know has just been 301'd and i got this message The URL http://www.site1.com/ redirects to http://site2.com/. Do you want to crawl http://site2.com/ instead?
Moz Pro | | kellymandingo
Would you like to:
Continue with www.site1.com
Continue with site2.com I really do not know what to do, its either the redirect script is missing something however its doing what it should or the server is a problem but again its doing what it should so why would SEOMOZ not be able to crawl the old URL like it example site above. Now the strange thing is Open Site Explorer does see the 301 and asks if i want to check the new URL instead Ps the redirect is done using PHP redirect which i am asking him to change to a htaccess as its now on a apache server and was wondering if this could be an issue, all pages go to correct pages as requested Thanks in Advance1 -
Getting your site totally indexed by SEOMOZ
Hi guys! Ijust started using SEOMOZ software and wondered how it could be that my site has over 10.000 pages but in the Pro Dashboard it only indexed about 1500 of them. I've been waiting a few weeks now but the number has been stable ever since. Is there a way to get the whole site indexed by SEOMoz software? Thanks for your answers!
Moz Pro | | ssiebn70 -
Crawl Report Technical Issue
I'm having a problem with our campaign especially the crawl report. Because the last update is Nov 4 and isn't supposed to be updated weekly? I already submit a helpdesk support ticket and even send a seperate e-mail regarding this issue but until now the report is still not updated. Anybody here can help me raise this issue/ Thanks.
Moz Pro | | shebinhassan0 -
SEOMOZ Competitive Domain Analysis
Hi, I am just trying to get to the bottom of why I am seeing such wildly different figures in the SEOMOZ link analysis. The SEO company that we use is saying that we have around 331 external links to our website, when I check on Yahoo site explorer, it returns 264 external links, but in my SEOMOZ campaign manager, it says I have 4 external links? I fully appreciate that none of these methods are going to be wholly accurate, and are going to produce different results, but I can't understand why the SEOMOZ results are so vastly different. I also know for a fact there are a number of Followed links that I have added to a couple of websites, that Google has now indexed (as they appear in search results), but don't appear in the campaign manager results. If SEOMOZ use a different method to assess an external link, then no problem - my main concern here is that as a Newbie I am doing something wrong within the SEOMOZ tools, and don't want to take action based on results that may be incorrect because I have set things up incorrectly? Any help is greatly appreciated. Many thanks, Gareth
Moz Pro | | gdavies090319770 -
How long does a crawl take?
A crawl of my site started on the 8th July & is still going on - is there something wrong???
Moz Pro | | Brian_Worger1
