How Is a Page Crawled by Moz When Moz Says 'No Links'?
-
As above, really. I've crawled a new client's site to find the Moz crawler has identified a handful of 404 errors.
The Moz crawler says these pages have '0 linking domains', and OSE has no data for these pages.
So how are these pages being crawled by Moz and what should I advise my client?
-
Thanks again, Peter.
-
That's great, thanks Keri.
-
Hi, no problem. Keri has answered your query re the Moz crawler not finding them.
Peter
-
If you download the CSV report, we list the referring page there. We know it isn't obvious and we're working on making this a little easier to find.
-
Hi Keri,
Thanks for getting back to me (thanks to Peter above, also).
So what is the best way of me finding where these 404 pages are linked from?
I am 99% sure they are internal links that I will ask the client to remove, but would like to avoid manually checking every link on the site if possible.
-
Hi Peter,
I searched for their domain name and about the expected number of results came up.
I searched for these individual URLs that Moz is reporting using the site: command, and they were not returned.
They are also neither in the site's sitemap (HTML page or XML file).
So how is SEOmoz Crawler finding them?
-
We have two completely different crawling systems. We have on crawler for your website, where you enter in your URL and we crawl your site looking for problems. We have a different crawler for OSE that goes and looks at the web in general and crawls for links. The two crawlers do not talk to each other, so it is actually understandable that OSE may not have data on the site. OSE doesn't have the same index size as Google, so we don't see all the links that are actually out there (though we're working all the time on improving this!).
-
Hi, if you put site:xxx.com into Google where xxx.com is your client's domain name, does Google report anything?
Peter
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Fastest way to get new backlinks recognised by Moz?
Hi all, I have seen a few discussions / advice pieces around this, but wondered what the current view is on the fastest way to get new backlinks recognised by Moz? I have seen people say that the content needs to be shared (e.g. on Twitter), and advice around creating Bitly links. I am not convinced that these techniques are working. On a related note, my website is very large. I have around 2.5m pages. Most of these pages are 3-4 clicks from the home page, but I have a small amount of important content much higher up in the site (1-2 clicks from the home page). Does Moz crawl pages nearer the home page more frequently, or does it make a pass of all pages on the site before it starts at the beginning again? If it makes a complete pass of all pages, then presumably making my site smaller will help to get Moz to see new content more quickly? Is there any other value to be gained in making a large site smaller? Many thanks in advance, Neil.
Link Explorer | | NHARRIS10 -
Why doesn't Moz crawler follow robots.txt?
It is crawling the entire site, and there is stuff we do not want it to. Please advise.
Link Explorer | | Tylerj0 -
Spam Score and crawling of my site
Hello, I'm trying to analyze the spam score of my site which is 9/17 Actually I have few backlinks and all of them have a low spam score (max 4/17, just one). I think there's some kind of issue with the crawler since I get strange spam factors: Large Site with Few Links (likely true, I recently deleted a lot of tags used once) Low Number of Pages Found (wasn't it a "Large Site"??) Low Number of Internal Links (I got a considerable number) No Contact Info (I have a link to my facebook in the menu and a "contacts" page) Thin Content (It's just a blog with min 300 words per post, why thin?) Site Link Diversity is Low (likely true) Ratio of Followed to Nofollowed Subdomains (likely true) Low MozTrust or MozRank Score (true) Ratio of Followed to Nofollowed Domains (likely true) Can you please help me to understand it, is it a crawling problem or similar? If needed I will post the url of the website. Thank you so much Marco
Link Explorer | | MarcoBP0 -
Why isn't OSE showing any of my links?
My domain uses a redirect of all traffic to https. The site is https://www.tallslimtees.com. I've been working on it this year and know there are several good, topical links coming in. But OSE shows nothing. Any idea why this would be the case? How can I see all of my links and the data on them?
Link Explorer | | DanDeceuster0 -
Error getting your data in moz ose
Why am I receiving a "There was an error getting your data" in moz ose? Everything worked fine yesterday but now I'm having trouble getting link metrics for my site.
Link Explorer | | TitanDigital1 -
How long will it take for the changes we've made to reflect in Moz OSE spam score data?
I signed up for Moz to see the spam flags our site had triggered. As soon as I found out, we worked on it and have been trying to correct our mistakes but it's been more than a month and we've managed to neutralise zero flags. I would appreciate if someone can clarify how long the OSE data takes to refresh. Also, how do you combat the following three specific flags: Ratio of Followed to Nofollowed Subdomains Ratio of Followed to Nofollowed Domains Low Number of Pages Found Crawl only gets a valid response to a small number of pages. Thanks.
Link Explorer | | Oziti0 -
Does Feedburner URL of the Home Page Carry Link Equity?
Hi There, During an SEO Audit, I found that OSE categorizes Feedburner URL of root domains under link-equity passing and followed. For example, the following link has been categorized under link-equity passing and followed: http://feeds.feedburner.com/SpoonflowerBlog I have heard that a lot of SEOs saying feedburner links don't carry any link juice. If that's true, then why does OSE categorize feedburner URL of root domains under link-equity passing and followed? I would appreciate if someone from the Moz staff could take some to answer this. Thanks.
Link Explorer | | TopLeagueTechnologies1 -
Why does facebook, twitter, pinterest show up for links?
Hello, As I have used site explorer on competition sites, I am curious what I need to do to have my facebook, twitter, and pinterest links to shop up as a backlink? I see some other sites have it, even if its a nofollow, but curious if I am doing something wrong? I have had accounts since last year. Just curious if I am doing something wrong, Thank you Chris
Link Explorer | | Berner0