Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it OK to put a Blog Post and a Page within the same folder on a Wordpress hosted website?
Our education company website (hosted on wordpress) has evolved into having content on key topics distributed across both blog posts and pages. For example, "top-pharmaceutical-companies" lends itself to being published as a page. However other content "top-pharmaceutical-companies-usa-2016" lends itself to being published as a blog post as it's more temporal in nature. Now we'd like to establish topical domain relevance for the root keyword "pharmaceutical companies" and build a folder www./ourcompany/pharmaceutical-companies/ But when we look through our blog content, we notice we have "Blog Posts" that would be an excellent fit for certain folders within our "Page" url structure. So would it be OK to amend these blogs post urls addresses to place them within the folder structure of the pages.
On-Page Optimization | | GetReskilled0 -
Duplicate Page Titles in Crawl Errors (although Google is rewriting in serps ??)
Hi Im working on a client/project and crawl report is showing thousands of dupe page titles In the case of the blog/news section its aprox 50 since aprox 50 posts and they all have the same meta-title: "Brand News | Brand" as opposed to: "Title Unique to Page/Topic/KW Relating to Content | Brand" Since these are the main content pages we want to rank (in addition to the main site category pages) then i have instructed dev must prioritise populating these pages meta-titles with the actual post/article titles, as per the latter version of the above example. (I should mention that i have requested they fix all dupe titles but main content pages are the priority). Whilst this will reduce the number of dupe titles in crawl error/warning report which is a good thing, is it actually likely to increase the ranking of these news/content pages given that Google does seem to be rewriting the titles correctly in the serps based on the page content ? Many Thanks in advance for your input
On-Page Optimization | | Dan-Lawrence0 -
How to rank well on 2 keywords - 2 separate pages or 1 combined page
Hi, I have a website about allergy. We ar developing new content, and through keyword research I have discovered that "dog allergy" and "cat allergy" are both very common searches. However, the cause, and symtoms are very alike for these 2 types of allergy so it would make sense to combine the two allergies on one page. So my question is: What do I choose to increase my chances to ranke the best I can for both "cat allergy", and "dog allergy"? Should I develop 2 separate pages for cat & dog allergy or should I do a combined page? (We would of course review the texts so no duplicate content/text would be used if we chose to have 2 pages) I would be so greatful for your advice!! Kind regards, Jeanette
On-Page Optimization | | Mylan-GDM0 -
On-Page Analysis Question
Hi, I have a question about the On-Page Analysis report. I am tracking two different keywords for our campaign: "Private Dining" and "Private Dining Sacramento". We are ranked 8th for Private Dining Sacramento but we have an On-Page analysis rating of F. While on the other hand we are not ranked in the top 50 for Private Dining but have an A on-page report. When looking at the on-page report it makes sense that we have an F for Private Dining Sacramento as we don't use that keyword anywhere on the page. We only use Private Dining. However, we are still ranked for Private Dining Sacramento and not for Private Dining. Should we update our keywords/text to use the Private Dining Sacramento keyword instead of the Private Dining? If we add Sacramento will we also get credit for Private Dining because it will still be part of all H,P and A tags we use? Sampe Report | Keyword | Grade | Google US |
On-Page Optimization | | Three29
| URL | Current | Change | Rank | Change |
| | Private Dining /private-dining | A | No-change-icon | no data |
| | Private Dining Sacramento /private-dining | F | No-change-icon | 8 | No-change-icon |0 -
Can you 301 redirect to a page that has other pages 301 to it?
Two years ago updated url page to include better keywords and used a 301 redirect from the old page to the new. so www.example.com/keyword-1st-generation.html now points to ... www.example.com/keyword-2nd-generation.html That moved the pages up in ranking, but now have better kw for the url, so is it okay to redirect the /keyword-2nd-geration-html to www.example.com/keyword-3rd-generation.html And what is a good length of time before removing the 1st-generation url? It's been 3 years and there is no chance of using it again. Plus, no sign of it in analytics.
On-Page Optimization | | AllIsWell0 -
Pages crawled dropped like a stone
set up a new campaign seomoz crawled the site and did about 1500 pages the last crawl it now only lists 1 page I can see anyway to see why, the site has not changed so why has the number of pages dropped?
On-Page Optimization | | spiralsites0 -
Why would my homepage be ranked lower (Page Rank 2) than my other pages on the site (PR3) ?
Why would my homepage be ranked lower (Page Rank 2) than my other pages on the site (PR3) ?
On-Page Optimization | | dmurtagh0 -
Page without content
Hey Everyone, I've started an SEO On Page analysis for a web site and I've found a lot of duplicate content and useless pages. What do I have to do? Delete this useless page, redirect or do canonical tag? If I have to delete what is the best way to do? Should I use GWT to delete? or just delete from the server? This URL for example: http://www.sexshopone.com.br/?1.2.44.0,0,1,13,0,0,aneis-evolved-boss-cock's.html [admin note: NSFW page} There is no content and it is duplicate in reference of this: http://www.sexshopone.com.br/?1.2.44.0,0,1,12,0,0,aneis-evolved-boss-cock's.html [admin note: NSFW page} and the correct page of the product is: http://www.sexshopone.com.br/?1.2.44.0,423,anel-peniano-evolved-boss-cock's-pleasure-rings-collar-white-reutilizavel-e-a-prova-d'agua-colecao-evolved.html [admin note: NSFW page} What is happening is that we have 8.000 pages like this. Useless and without any content. How do I proceed? Thanks!
On-Page Optimization | | luf07090