Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I reduce Too Many On-Page Links? I am looking for best method through which I can reduce by on page link.
Hello, As I have the Pro Account in SEOMOZ . I have created the campaign for my website and I have seen the warring for on page analysis for Too Many On-Page Links. As per my knowledge in past it's matter that you can put maximum 100 links per page but now is it still matter or harm if pages has Too Many On-Page Links? And if yest then please let me know the best method to reduce my On-Page Links with out doing any major changes in website
On-Page Optimization | | jemindesai0 -
Sold Property Page
I have a real estate agent property website. What should I do with the sold property page. Should I remove the page entirely since the page has no more use for user. Currently the page is still exist, but only with word "This property is currently not available/sold"
On-Page Optimization | | odihost10 -
Page architecture
We have some good content on our site, particularly relating to UK employment law. One section on unfair dismissal is split into 9 pages - there is a fair amount of legal detail. The question is whether we should combine it all into one "mother of all unfair dismissal" page just to satisfy the Google monster or keep in as it is. Some of the individual pages rank on page 1 already. If we change the architecture are 301 redirects the best way to handle the changing urls? The other more important issue is whether it is easier to read it all on one page or split it. Keeping G happy may not actually keep our users happy. As the content is quite dense we want to ensure we don't overload people. Any thoughts appreciated.
On-Page Optimization | | dexm100 -
Duplicate Page Content for Product Pages
Hello, We have one website which URL is http://www.bannerbuzz.com & we have many product pages which having duplicate page content issue in SEOMOZ which are below. http://www.bannerbuzz.com/backlit-banners-1.html
On-Page Optimization | | CommercePundit
http://www.bannerbuzz.com/backlit-banners-10.html
http://www.bannerbuzz.com/backlit-banners-11.html
http://www.bannerbuzz.com/backlit-banners-12.html
http://www.bannerbuzz.com/backlit-banners-13.html We haven't any content on these pages, still getting duplicate page content errors for all pages in SEOMOZ. Please help me how can i fix this issue. Thanks,0 -
Product page optimalisation
Throughout the years our website kept on growing this has led to product pages that have so much sub-pages that nobody is able to really get a good idea of the product. We are working on a new website where the visitor is central. Together with a usability partner we have down sized the preferred data to fit on one page with a tabular system with a maximum of 4 taps. My question will this affect our find ability if we go from 10 to 15 sub-pages to one main page
On-Page Optimization | | TiasNimbas0 -
Page Analyzer & Page 1
I follow the recommended things from the Page Analyzer or Grader, and I am like position #40, so how do I get to page #1 as a minimum.
On-Page Optimization | | sansonj0 -
Duplicate Page Content and Duplicate Page Title
Hi All, I'm new in SEOMoz and have some questions after I have already spend 2-3 days trying to resolve the problems identified from Crawling one of my clients websites. I get quite a lot of Duplicate Page Conntent and Page Titles warnings and trying to find a workaround through the forums and posts. I continuously get this error on most of my pages: URL: http://domain.com/benefits with the same Page but with a WWW in front URL: http://www.domain.com/benefits Any advice will be highly appreciated. Thanks, Athos
On-Page Optimization | | athosk0 -
SEO for spanish website
Hi, A client has given us the site http://www.comtranslations.com/Home.html for optimization. He wants to optimize only the spanish part ( the link is on the top right ). By clicking on the link Espanol, the url opened is - http://www.comtranslations.com/Principal.html. He wants seo for this website for spanish keywords. The keywords are - Traducción
On-Page Optimization | | seoug_2005
Traductor
traducir español inglés
traducción My question is how do we go about this ? Shall we purchase a software that translates spanish to english ? Thanks0