Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Missing H1 headings
Hi,
On-Page Optimization | | Colin12
I have an enquiry in relation to missing H1 headings. I am getting a number of these errors reported for collection pages where I am using a full width banner image at the top of the page with no visible description underneath. In these circumstances is it advisable to somehow include a H1 heading or can I ignore it? https://osullivanspharmacy.ie/collections/deep-heat0 -
2 Similar Pages
Hello, I have two very similar pages. The first page is an apartment rental page with a map, rental listings and some neighborhood data below. The neighborhood data includes useful info about the area with photos, text about the area, crime rates, avg. rental rates, etc. The second page is a neighbourhood guide that includes virtually the same data as the rental page, but in longer form ie. more photos, more text, etc. I want the rental page to rank, while ranking the neighborhood page is not important as it would be used more for link bait. But since the information on the two pages is the same, I don't want them to compete with each other. I'm thinking of putting a cannonical tag on the neighborhood page pointing back to the rental page. Is that the correct thing to do in this instance? Thanks for your help. J
On-Page Optimization | | torontojon0 -
Why am I not seeing sitelinks for this website?
http://laleablack.com/ It's been around for a few months - I have other sites showing sitelinks in the search results. but when I search for "laleablack" "lalea and black" I don't see anything...anyone have any idea why? Links have proper anchortext etc
On-Page Optimization | | smlsml0 -
Duplicate Page Content
Hi there, We keep getting duplicate page content issues. However, its not actually the same page.
On-Page Optimization | | HamiltonIsland
E.G - There might be 5 pages in say a Media Release section of the website. And each URL says page 1, 2 etc etc. However, its still coming up as duplicate. How can this be fixed so Moz knows its actually different content?0 -
Which page to rank for a Keyword? Home Page or Deep Page?
So, we have a situation where there is one particular keyword we want to rank for. We have been up and down over the years, at our best probably position 4-5, and now at 20ish. Thats for our home page of course, which the majority of our linking is probably pointing at. We also have a sub page which is optimised for that particular service. The term is "web design brisbane".
On-Page Optimization | | MauriceKintek
So as you can imagine, Web Design is in itself a service and we offer others. Should we optimise our home page for it and remove the sub page?
Keep the sub page because its one our services and optimise both?
Do some kind of canonical thing?
Change our interlinking? All our competitors home pages seem to be the ones that rank, and it feels and looks better in results if its the home page, but if switching up to our sub page is better im all ears. Also if our sub page is somehow hurting or leaking SEO from the home page, id like to know as well. Would prefer to not have to provide a link, due to competition but if someone wants to know we can always PM.0 -
Understanding SEOMOZ and how to make improvements to my website
I want the same as everyone else the best position in Google. My campaign is set up as sub-domain. I am trying to get an understanding of what make the improvements to the following. What are the most important metrics and if I can work to improve them what makes those improvements. Currently I have a good position in Google for my most productive phrase but feel I need to make improvements to ensure I keep that position. I am adding regular unique content and looking for relavant quality links but it does not seem to be making much difference to the metrics below. Domain Authority 16 Domain MozRank 2.83 Domain MozTrust 0.86 External Followed Links 16 Total External Links 55 Total Links 668 Followed Linking Root Domains 8 Total Linking Root Domains 9 Linking C-Blocks 3 Subdomain MozRank 2.95 Subdomain MozTrust 1.73 External Followed Links 15 Total External Links 54 Total Links 667 Followed Linking Root Domains 8 Total Linking Root Domains 9 Which are the important metrics and can I improve my site to increase my Moz position therfore improving Google results
On-Page Optimization | | PhilSmith230 -
Opinions please on Duplicate page titles & too many on-page links warnings.-
Hello folks, I'm a total SEO newbe but totally enjoying
On-Page Optimization | | CSC
using SEOmoz to learn more. We have ecommerce sites and the 1st crawl flags – as appears typical too many on-page links. We display up to 20 products (each with three links!)
and I’m trying to push to have fewer but meeting resistance from colleagues.
We have links duplicated all over the site believing it eases navigation. My question is just how critical is the number of products displayed
and the resulting volume of links to SEO results? Also we currently have collections of products displayed
across several pages which of course have the same page title and this is flagged
as a duplication error. I wonder if product auto-scrolling help as this means only a certain number of products are displayed at one time on one page thus reducing links and the need for duplicate page titles? My superiors are resisting change (perhaps nervous of spoiling
what already works) and I need to know where to direct my persuasive powers! Many thanks in anticipation, Spence0 -
Is reported duplication on the pages or their canonical pages?
There are several sections getting flagged for duplication on one of our sites: http://mysite.com/section-1/?something=X&confirmed=true
On-Page Optimization | | Safelincs
http://mysite.com/section-2/?something=X&confirmed=true
http://mysite.com/section-3/?something=X&confirmed=true Each of the above are showing as having duplicates of the other sections. Indeed, these pages are exactly the same (it's just an SMS confirmation page you enter your code in), however, they all have canonical links back to the section (without the query string), i.e. section-1, section-2 and section-3 respectively. These three sections have unique content and aren't flagged up for duplications themselves, so my questions are: Are the pages with the query strings the duplicates, and if so why are the canonical links being ignored? or Are the canonical pages without the query strings the duplicates, and if so why don't they appear as URLs in their own right in the duplicate content report? I am guessing it's the former, but I can't figure out why it would ignore the canonical links. Any ideas? Thanks0