Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it OK to put a Blog Post and a Page within the same folder on a Wordpress hosted website?
Our education company website (hosted on wordpress) has evolved into having content on key topics distributed across both blog posts and pages. For example, "top-pharmaceutical-companies" lends itself to being published as a page. However other content "top-pharmaceutical-companies-usa-2016" lends itself to being published as a blog post as it's more temporal in nature. Now we'd like to establish topical domain relevance for the root keyword "pharmaceutical companies" and build a folder www./ourcompany/pharmaceutical-companies/ But when we look through our blog content, we notice we have "Blog Posts" that would be an excellent fit for certain folders within our "Page" url structure. So would it be OK to amend these blogs post urls addresses to place them within the folder structure of the pages.
On-Page Optimization | | GetReskilled0 -
Why are http and https pages showing different domain/page authorities?
My website www.aquatell.com was recently moved to the Shopify platform. We chose to use the http domain, because we didn't want to change too much, too quickly by moving to https. Only our shopping cart is using https protocol. We noticed however, that https versions of our non-cart pages were being indexed, so we created canonical tags to point the https version of a page to the http version. What's got me puzzled though, is when I use open site explorer to look at domain/page authority values, I get different scores for the http vs. https version. And the https version is always better. Example: http://www.aquatell.com DA = 21 and https://www.aquatell.com DA = 27. Can somebody please help me make sense of this? Thanks,
On-Page Optimization | | Aquatell1 -
SeoMOZ meta descriptions
Anyone notice that seoMOZ doesn't use meta descriptions? Any idea why not?
On-Page Optimization | | SoulSurfer80 -
How do you fix on page SEO ?
I have been trying to push my foundation website in organic search results for competitive keywords , i have been not been so consistent in raking our website in top search results of Google. can some one recommend the guidelines and activities which can really push my websites to Google first page. More Info: about our foundation We are the worlds largest school meal run ngo in the world feeding over 1.3 million school children in India Wesite url www.akshayapatra.org
On-Page Optimization | | AkshayaPatra0 -
On Page Optimization Reports
How is it determined which terms and associated urls are chosen when SEOmoz tracks your On-Page Report Card? I'm receiving a lot of F Grades for terms I'm not really interested in and a lot of terms I'd like to be tracked aren't. Is there a way I can manually choose which terms and pages I'd like to be shown?
On-Page Optimization | | ClaytonKendall0 -
Website review needed
Hello, We need advice about seo for website http://www.oluadeonline.com/ The keywords provided are - Make Money Online, Online Services, Work At Home, Online Money, Online Business, Online Work Its seems impossible to rank the site for these keywords as there is too much competition
On-Page Optimization | | seoug_20050 -
Duplicate pages
Hi, I am using a CMS that generates dynamic urls that according to the SeoMoz tool will be indexed as duplicate pages. The pages in questions are forms, blog-posts etc. that are not crucial to achieve ranking for. I do worry though about the consequences of having 20 (non-duplicate)pages with static urls and about 100 pages that are duplicates with dynamic urls. What consequences will this have for the speed that the robots crawl the site and could there be negative effects on ranking for the entire domain?
On-Page Optimization | | vibelingo0 -
Page Authority
I have recently optimised a set of images for a client of ours: I'm looking through all the PA of these newly optimised images, and have varying PA {from SEOmoz toolbar} I understand that internal linking will pass link juice, and obviously external links will add to the overall PA. I have several pages with a PA of 36: { Fairly deep pages} Yet they have no external or internal links going to them. My question is "How can a page gain any authority when it has no visible links pointing at it?" Obviously there must be a link pointing at it {internally} as Google wouldn't have crawled the page right? Also lets say all the keywords are of equal competitiveness would the keywords with highest PA rank higher than those on O PA pages. Many Thanks
On-Page Optimization | | Yozzer0