Moz crawling doesn't show all of my pages
-
Hello,
I'm trying to make an SEO on-pages audit of my website.
When using the SEO crawl, i see only one page while i have much more pages in this website (200+). Is it an issue with the sitemap ? The robot.txt ?
How can i check and correct this. My website is discoolver.com
-
Hi there,
Sam from Moz's Help Team here!
So, I took a look at your site, and unfortunately I believe this is a result of our crawler's inability to work with Javascript. Any client side loaded content will not be rendered to our crawler as we are only seeing what is rendered by your server.
Our crawler works by parsing the source code of your site, looking at HTML elements. If your site is primarily Javascript, then the data you get back with regards to the crawl report won't be completely accurate because of this. There's no real workaround that I can recommend for this one, since it is a technical limitation of our tools, but there are some good blog posts and discussions in the Q&A about this if you head over to our Help Hub.
While the tools and data that rely on our crawl of your site may not return the best results because of that Javascript, your keyword rankings and link profile should work just fine.
You might also want to check out a few tools that are compatible with Javascript, like Botify or Screaming Frog.
I'm really sorry I can't be of more help here; Please let me know if there's anything else I can help with, or if there are any follow-up questions you might have!
-
Thanks for the answers.
I believe the issue is about a rel=canonical from the homepage which links to an external website...
Can this tag prevent a good website crawl ?
-
So just looking it appears you don't have a robots.txt file or a sitemap. At least one that I could easily find.
This Moz post might help when creating one.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My site links to my clients not showing in Moz
I've got a web design business with a site that's been out there for 17 years - www.biztoolsone.com. I've got links to my clients' sites in my portfolio page. The backlinks list for my clients, when I check them in Moz, don't show the links from this site. They always showed in the list until a few months ago. The site has a DA of 59 & PA (for home page) of 50. The spam score is 0/17. I'm having a tough time telling my clients that the Moz backlinks list is accurate when that site doesn't show in the list. The same is true for www.fayettevillenc.biz - this is one I created years ago to generate backlinks. It's got a spam score of 4/17, so I'm a little more understanding of that one. Thanks for any guidance!
Link Explorer | | wcksmith10 -
What's the Story on Mozscape Updates?
Hey gang, As you may be aware, we were considerably late with our last index release. You have my sincere apologies for that and the apologies of the entire team. In the interest of transparency, I want to try to explain what's been going on. Since stepping down as CEO, I've been asked to take on a few roles in the company. One of those is product architect (basically the product owner) of our Big Data team, who produces the Mozscape link index. For several years, that team has been almost exclusively focused on getting us closer to a near real-time indexing system that does not have scalability issues. Mozscape is currently smaller than our major competitors, and we're also often slower. Our metrics (PA, DA, MozRank, MozTrust, Spam Score, Social Data, etc) have been the unique value we provide, but it's not enough. We need to be competitive on size and freshness. Building a raw link index (without processed metrics like PA/DA et al) is hard, but it's possible. Building a link index with those metrics is really tricky, and requires computer science knowledge and skills far beyond the scope of my understanding. That's what our team's been working on, and they've made some progress, but it's been slow, hampered by unknown unknowns, and materially hurt by a lack of experienced talent we can hire to help (we've had open job posts for years now). In the meantime, our historic Mozscape index structure keeps encountering challenges - this latest round is still somewhat unexplained (we believe there's hardware issues compounded by how the system is architected to handle large domains, but there may be other issues). The team's struggled to split time between keeping the old Mozscape running and hunkering down to finish the new system. I'm trying to help them balance things as best I can, and we're going to be putting effort toward making sure we get index releases out on time. However, to do that, we'll need to scale down size, and then rebuild back up. We think we can do this while also improving the prioritization of which links we crawl (e.g. deeper on important domains that link out, less so on deep pages that don't link anywhere) so the index overall improves. However, I don't want to minimize the risks - we may have some slow updates, some smaller indices, and some less-than-ideal data in the next one or two indices while we work to remedy this issue. I HOPE we don't, and that things actually get better immediately, but we can't promise that until the work gets finished. TL;DR - Mozscape V2 is in development and will let us as big and faster as any link index. In the meantime, current Mozscape's having issues & we're making smaller indices in an attempt to diagnose and repair. As always, thanks for your understanding, continued support, and if you have any questions, feel free to leave them below. I realize that this level of service/product quality is NOT OK, and I'm doing everything in my power to fix it.
Link Explorer | | randfish8 -
Why moz is not showing My Website External Backlinks?
Google Webmaster showing my external links 41
Link Explorer | | djsamrock
Majestic showing my external links 28
Ahrefs showing my external links 13
&
Moz is showing my external links 0... Will anyone please explain Why moz is not showing My Website External Backlinks?0 -
Open Site Explorer Top Pages https Title tag issue
Hello, everybody. I've noticed this strange thing in Top Pages reporting of Open Site Explorer. If i do a report for any website which doesn't have security certificate installed, such as http://www.hyperlinksmedia.com I get normal results, with title tag shown for every page, but, if i do a report for website with Security certificate, such as https://www.hyperlinksmedia.com, report is coming back for non-secure URL version (http:...), so, it says "No Title" for any pages. I wonder, if it influence PA and DA scores. Although, if i run full crawl test, then i shows all meta and title tags for SSL version of a website's url. Thanks!
Link Explorer | | seomozinator0 -
Why Moz Doesn't See or Count Our Backlinks?
Hi Moz Community! We have been working hard to improve our Moz metrics, measuring against a high ranking competitor to help us set our goals. Our Majestic and Webmaster Tools find tens of thousands of external backlinks pointing to our domain. That's all well and good. Moz's Open Site Explorer, however, only finds 900 total links - including internal links! This being the case, we have worked diligently to build a variety of great external backlinks, creating Bitly links and encouraging clicks on those through social promotion. Yet, our competitor has over 7,000 external backlinks in Moz's index, while ours is not growing relative to their number of backlinks. Can anyone share with us what they do to tell Moz about their backlinks? We already know we have many more backlinks than our competitor, from trusted domains with good authority, yet it seems Moz is not discovering them. We just want to understand how to use these Moz metrics to create meaningful calls-to-action. Otherwise, it seems like a gargantuan waste of time, and our team has difficulty getting buy-in from our company to put time and assets toward tasks based on our Moz numbers!
Link Explorer | | RegistrarCorp0 -
Why does facebook, twitter, pinterest show up for links?
Hello, As I have used site explorer on competition sites, I am curious what I need to do to have my facebook, twitter, and pinterest links to shop up as a backlink? I see some other sites have it, even if its a nofollow, but curious if I am doing something wrong? I have had accounts since last year. Just curious if I am doing something wrong, Thank you Chris
Link Explorer | | Berner0 -
Do 'Just Discovered' Links get added to the main link index?
Hi, I was wondering if the 'Just Discovered' links get added to the main link crawl index? It would seem to make sense for them to do so, as this would enable the enable the link index to be more up to date than it would otherwise be. Observing the link index it would seem that at the moment it does not do this and they are totally separate indexes (based on personal observation). Thanks
Link Explorer | | James770 -
Is there an option that's more precise over Open Site Explorer?
I've had folks explain to me before that OpenSiteExplorer is just an estimation, etc. but there are some fairly easy statistics that seem to be different and it makes me nervous that either its wrong or I'm doing something wrong. As you can see in the image, moz isn't read social metrics correctly. It actually used to be pretty spot on, but as you can see with Google+'s its off. Maybe not a big deal right? But for my clients new Moz Analytic print outs it makes somewhat of a difference. Any help? UlwQs3w
Link Explorer | | jonnyholt0