Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sitelinks to internal pages
Hi Moz Community, Is it only possible to get sitelinks to your main domain in the SERPs or can you also get them to internal pages in the SERPs? i.e. if this were to pop up in the SERP: www.mysite.com/page1
On-Page Optimization | | Brian_Dowd
Can I get sitelinks such as the examples below:
Sitelink1: www.mysite.com/page1/page1a
Sitelink2: www.mysite.com/page1/page1b
Sitelink3: www.mysite.com/page1/page1c
etc. Just curious and I haven't really find anything like that on the internet. How can I set them up? Thanks.0 -
Do i optimise only the page that i want to rank or do i have to optimise my entire website.
I'm looking at 2 keywords and i have optimized my main page towards it. My website consist of other pages like, "About Us, Contact Us, Testimonials". My SEO title and Meta Description for these pages are just the basic. Q: My question is, do i have to optimize the pages that i do not want to rank? Is there any way to utilise these pages to help my main page to rank?(besides using them for internal links)
On-Page Optimization | | kevinbp0 -
Webs Pages not correct
First of all I am a beginner but that will show with this question. I have moved my website from Wix to Web.com. I had the dns I think that is correct re-direct my site to web.com. Since then some of the pages from wix still show up even though they are no long apart of my site. Secondly when I go to google and type in cestoday.com some of the pages gives me a 401 or 404 error code, and others do not link to the proper page on my website. They go to my web-site but not the correct page. What do I do. Thank you Dave
On-Page Optimization | | redsman944
cesyes@hotmail.com In reference to my last question about moz not being able to see my pages and giving me an "F" I thought you might find it interesting to know that that same page once moved got an A amazing.0 -
Should a website have the same standard headings for product pages? And how does that affect SEO?
Hello! I hope my question is self-explanatory but let me give you an example. Suppose I have website X selling A, B and C and I have a dedicated page for each of those products. For SEO practises, Is it better to have the same webpage structure for all product pages (i.e. the same headings and sections) or is it better to vary the structure of the page? Or is it really not important? Again going back to the example, is it better for product page A, B and C to have the same sections in the same order (i.e. Pricing, Product Details, Product Reviews) or is it better to keep it as a flowing content page (product page A has same information in B and C but all those pages are different with different headings and questions asked depending on the product)? Many thanks!
On-Page Optimization | | MarketingDude0 -
Pages crawled dropped like a stone
set up a new campaign seomoz crawled the site and did about 1500 pages the last crawl it now only lists 1 page I can see anyway to see why, the site has not changed so why has the number of pages dropped?
On-Page Optimization | | spiralsites0 -
Duplicate Page Content for Product Pages
Hello, We have one website which URL is http://www.bannerbuzz.com & we have many product pages which having duplicate page content issue in SEOMOZ which are below. http://www.bannerbuzz.com/backlit-banners-1.html
On-Page Optimization | | CommercePundit
http://www.bannerbuzz.com/backlit-banners-10.html
http://www.bannerbuzz.com/backlit-banners-11.html
http://www.bannerbuzz.com/backlit-banners-12.html
http://www.bannerbuzz.com/backlit-banners-13.html We haven't any content on these pages, still getting duplicate page content errors for all pages in SEOMOZ. Please help me how can i fix this issue. Thanks,0 -
Multiple silos/products/landing pages. How to design the root page for conversion?
Hi everyone, First post. Tried a few awkward searches on the topic but I must be using bad keywords. I'm re-designing a site that has multiple products and matching multiple audiences. This means we have multiple sillos for multiple groups of keywords with the supporting pages for each silo landing page. Currently I'm working on updating the look and text of those landing pages for each silo to increase conversion. This leaves me with the root web page. We get quite a lot of search traffic from people searching our brand name - so this results in clicks straight through to our root domain. There are no product specific landing pages because it could be any one of the 3-5 different personas we have hitting the site from that source. Does anyone have any good examples of where a site has had multiple products and needed to segregate their audience on a root top page? I'd like to see some examples and hear peoples thoughts. At the moment I'm thinking I need to fill that page up with trust factors as to why people should use us as a company, along with navigational elements in relation to each and every product so they can click through to the proper landing page. The main way I can see on executing that is to have a rotating banner with the same tag line "this is what we do" but be alternating between banners relating to each product.. with their own click through button to go to the respective landing page. Thoughts anyone? Example of sites doing this well?
On-Page Optimization | | specific0 -
Optimization of home page
Hi there I have an issue which, despite searching hard, I simply cannot find the right solution for. We have an index page that used to rank pretty well for a main industry keyword. However following a revamp of the site last year the kw slipped and no longer brings in decent traffic levels. The problem seems to be that the old static site had a sprinkling of variable anchor text links that brought value to the home page. Instead of the main anchor being "home" we would revert to "main keyword" and variations across the site sometimes in t he content but mainly on the nav bars. However the new CMS design structure restricts us considerably with anchor distribution and so instead we opted for the site logo on the masthead to have an ALT tag for "main keyword" but so as not to game google too much we added .."home" to the tag. Probably pointless but we figured it could do no harm. This ALT text is site wide Problem now is that we have lost the spread of internal nav bar anchors and variety etc. We have slipped in the serps for "main keyword" and I cant help thinking we are not maximising the anchors as we should. So what Im coming to is this.... How can we tell if Google is picking up the ALT tage anchor as the main anchor to rank the site at the expense of all internal text anchors. Despite retaining lots of embedded anchors - according to the Moz metrics these are not being picked up because OSE suggests the ALT tag anchor is taking precedence. The serps probably support this view as well. Should we: a) Vary the masthead ALT if there is no way of avoiding this being the most important link / anchor on the page b) Remove the ALT anchor and instead opt for content links high on the page (we do have nav bar links saying "Home" site wide as well which may overrid the embedded links?) c) Leave the ALT alone and still push for content anchors as described in b) What is the best way to handle this..? Best wishes and thanks Morch
On-Page Optimization | | Morch0