SEOMOZ Crawling Our Site
-
Hi there,
We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist.
Thanks,
Geoff
-
That's a question best answered by a programmer and/or someone who is familiar with the code of your site. Sorry I can't help more!
-
Hi Adam,
Thanks - how do you change the navigation so each page is linked with a single URL?
Thanks
-
If SEOmoz is crawling all those duplicate/unnecessary URLs, the search engines probably are, too.
Best solution: Change the navigation on your site so that each page is linked to using a single URL.
-
Thanks Adam.
Have had a look and it looks like the crawlers are cycling through numerous unnecessarily crawled pages for each product based on the top and bottom nav bar on our website (i.e. it adds FAQ and Shipping Info for each product + all the categories we have created on the website). Attached is an image link of some lines from the CSV file.
I imagine this is what is making it chew up bandwidth. Any insight on how to avoid / change this would be much appreciated.
Thanks
Geoff
-
In addition to what Adam said, you can just search your site on Google and I come up with over 14,000 results:
Both ways should give you a good insight into what's being found.
-
Hi Geoff,
There are multiple reasons your site could have "extra" pages: tag pages, category pages, print this pages, duplicate pages, etc.
To see what pages SEOmoz is crawling:
- Go to http://pro.seomoz.org/campaigns
- Click View This Campaign for your site
- Click Crawl Diagnostics
- At the top right, Export as CSV
- You'll get a spreadsheet listing all the URLs that SEOmoz crawled
Hope that helps!
~Adam
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Not Cached By Google
Hi My name is Apoorwa and i have my own website, My website is not cached by google, Why is this problem happening with my site.can somebody help me please? its urgent....this is my site - http://www.holifestival.org/Please assist me.......
Moz Pro | | Packersmove0 -
Hoe to crawl specific subfolders
I tried to create a campaign to crawl the subfolders of my site, but it stops at just 1 folder. Basically what I want to do is crawl everything after folder1: www.domain.com/web/folder1/* I tried to create 2 campaigns: Subfolder Campaign 1: www.domain.com/web/folder1/*
Moz Pro | | gofluent
Subfolder Campaign 2: www.domain.com/web/folder1/ In both cases, it did not crawl and folders after the last /. Can you help me ?0 -
Where is the crawl test tool located in this new site?
Hi there, Where is the crawl test tool located in this new moz site? Formerly it was, http://pro.seomoz.org/tools/crawl-test Hope you can help. Thanks:)
Moz Pro | | steveovens1 -
SEOMOZ Support: Domain Name Change in SEOMOZ
My client is changing their domain name. They are redirecting (301) all of their URLs to the new domain. Do I need to do anything in SEOMOZ to preserve the campaign?
Moz Pro | | DynoSaur0 -
How do YOU use site explorer?
I normally use open site explorer to identify links that competitors of my clients have and sometimes this gives me what I call 'some low hanging fruit' to go after. (and of course links that are more challenging to get) I don't know why this didn't occur to me sooner. If my client is a chiropractor why not look at the links for 50 or 100 of the top rankings chiropractic sites all over the US? This would HAVE to uncover a wealth of blogs to comment on that have good authority, great industry associations, publications, forums - a whole wealth of items. It made me wonder how many people use site explorer like I have been (top 3-4 competitors that your client has) or identifying links pointing to LOTS of competitors? How do you use it? Couldn't you almost base an entire link building campaign using OSE? Why would this be a bad idea if not? Just some random thoughts. THE WEEKEND IS ALMOST HERE - Have a great day everybody! 🙂
Moz Pro | | Mrupp441 -
Has anyone else not had an SEOmoz crawl since Dec 22?
Before the holidays, I completed a website redesign on an eCommerce website. One of the reasons for this was duplicate content. The new design has removed all duplicate content. (Product descriptions on 2 pages) I took a look at my Crawl Diagnostics Summary this morning and this is what I saw: Last Crawl Completed: Dec. 15th, 2011 Next Crawl Starts: Dec. 22nd, 2011 Thinking it might have something to do with the holidays. Although, I would like to see this data as soon as possible. Is there a way I can request a crawl from seomoz ? Thanks, John Parker
Moz Pro | | JohnParker27920 -
Help with SEOmoz API
Hi guys, I'm trying to make API requests from my webserver via PHP. I'd like to retrieve data from the SEOmoz URL Metrics API. Unfortunately I always get the error response "unauthorized" even when I copy and paste the Sample Valid API Signature generated by your system into the browser. Is Signed Authentication not longer supported? I even tried the sample PHP Code SignedAuth.php but there's the same problem, too. If signed authentication is not longer available, do you have a code example for the basic http authorization? Thanks, Brandon
Moz Pro | | thegreatpursuit1
