Can Google see all the pages that an seomoz crawl picks up?
-
Hi there
My client's site is showing around 90 pages indexed in Google. The seomoz crawl is returning 1934 pages.
Many of the pages in the crawl are duplicates, but there are also pages which are behind the user login.
Is it theoretically correct to say that if a seomoz crawl finds all the pages, then Google has the potential to as well, even if they choose not to index?
Or would Google not see the pages behind the login? And how come seomoz can see the pages?
Many thanks in anticipation!
Wendy
-
Well, that could be your easy solution. Make sure they're all set not to be indexed, then you'll be able to (mostly) ensure Google won't crawl them, and they'll probably disappear from your moz crawl report as well. As far has how moz is finding them to begin with behind your login wall, sorry, I have no idea.
-
The pages behind the login? No not yet - they are a new client, so I am just auditing at the moment to identify what we need to do
Many thanks for your replies!
-
This may be an obvious question, but to you have those pages set to noindex?
-
Hi Marisa
seomoz are crawling unecessary pages, (they return pages ignored by screaming frog for example)
BUT my concern is that if Google can also see them, even if they choose to ignore them my client maybe getting slammed for duplicate issues or the pages behind the login may suddenly appear in the index.
We'll get no index / no follow added, and fix the dupes, but am really interested as to how seomoz sees behind the login
-
Here's the real question: Do you WANT Google to see all these pages, or is SEOmoz crawling unnecessary pages?
-
Great, many thanks Nakul - they are a new client so am waiting on getting access to WMT - will go through with a fine tooth comb! Just seems really weird with regards to the pages behind the login ...
-
Wendy, if SEOMOZ can see it, I am sure Google can see it as well. I would login to your webmaster console and check the index status. Do you have an XML sitemap submitted for your website ? Once you do, you'll have a more accurate read on the number of pages you submitted and how many of them are indexed. The new index status Google introduced last month also lets you see pages Google ignored for multiple reasons.
I hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Special Characters in URL & Google Search Engine (Index & Crawl)
G'd everyone, I need help with understanding how special characters impact SEO. Eg. é , ë ô in words Does anyone have good insights or reference material regarding the treatment of Special Characters by Google Search Engine? how Page Title / Meta Desc with Special Chars are being index & Crawl Best Practices when it comes to URLs - uses of Unicode, HTML entity references - when are where? any disadvantage using special characters Does special characters in URL have any impact on SEO performance & User search, experience. Thanks heaps, Amy
Moz Pro | | LabeliumUSA0 -
Why is my MOZ report only crawling 1 page?
just got this weeks MOZ report and it states that it have only crawled: Pages Crawled: 1 | Limit: 10,000 it was over 1000 a couple of weeks ago, we have moved servers recently but is there anything i have done wrong here? indigocarhire.co.uk thanks
Moz Pro | | RGOnline0 -
Where has the old seomoz crawl tool gone? I can't seem to find it
I'm looking for the (SEO)moz crawl tool - but can't find it. Where has it gone?
Moz Pro | | SearchMotion0 -
1 page crawled ... and other errors
1. Why is only one (1) page crawled every second time you crawl my site? 2. Why do your bot not obey the rules specified in the robots.txt? 3. Why does your site constantly loose connection to my facebook account/page? This means that when ever i want to compare performance i need to re-authorize, and therefor can not see any data until next time. Next time i also need to re-authorize ... 4. Why cant i add a competitor twitter account? What ever i type i get an "uh oh account cannot be tracked" - and if i randomly succeed, the account added never shows up with any data. It has been like this for ages. If have reported these issues over and over again. We are part of a large scandinavian company represented by Denmark, Sweden, Norway and Finland. The companies are also part of a larger worldwide company spreading across England, Ireland, Continental Europe and Northern Europe. I count at least 10 accounts on Seomoz.org We, the Northern Europe (4 accounts) are now reconsidering our membership at seomoz.org. We have recently expanded our efforts and established a SEO-community in the larger scale businees spanning all our countries. Also in this community we are now discussing the quality of your services. We'll be meeting next time at 27-28th of june in London. I hope i can bring some answers that clarify the problem we have seen here on seomoz.org. As i have written before: I love your setup and you tools - when they work. Regretebly, that is only occasionally the case!
Moz Pro | | alsvik1 -
What could be the reason that seomoz only shows crawl results for my homepage?
Hi there I am running three campaings for three different sites. The first site crawl is successful with a ful report. However the other two only shows results for the homepage, i.e. only a single page crawled by the mozbot. What could be the reason for this? Thanks, Gerrie
Moz Pro | | marketingmen0 -
What's the future of SERP Tracking? And... Is SEOMoz's SERP Rank Tracking in compliance with Google Adwords API Terms of Service?
My question is: Is SEOMoz's SERP Rank Tracking in compliance with Google Adwords API Terms of Service? Background: The reason I ask is because Raven Tools is now removing their SERP Reporting tool because it uses scraped Google position data. So, it looks like SEO's will either have to find a new rank tracking tool or find new ways to traffic the effects that rankings have on a website traffic volumes. For instance, there is a way to get the position a search results was in Google when it was clicked. We could create a secondary profile in Google Analytics for each client and use a custom filter to record the position that the keywords was in when the search result was clicked ( http://www.seomoz.org/blog/show-keyword-position-using-filters-and-advanced-segments ) Or perhaps we'll have to use Google Webmaster Tools' SEO Report to get data somehow ( http://support.google.com/analytics/bin/answer.py?hl=en&answer=1308626 ) What are your thoughts on this? As you know, ranking data is still a great way to show clients if they are gaining or losing visibility in the search engines. It helps SEO's to report how effective their efforts have been. Because other ranking software companies uses Adwords API data to show the keyword search volume and advertiser competition of a keyword, they can not or eventually will not be able to use scraped ranking data any more. But, if another rank tracking tool out there doesn't need to be in compliance with the Adwords API TOS because they don't use that API to show search volume and advertiser competition, they can still technically provide their ranking data and not be violating any TOS, right? I'm just trying to understand the best way to continue reporting impact of organic keyword rankings on a website. Does the SEOMoz SERP Tracker comply with Adwords API? Is there another rank tracking tool out there that already is using Average Position data from the GWT SEO Report tool? Should we all just stop reporting rankings to clients altogether? Scott
Moz Pro | | OrionGroup2 -
How Does On Page Analysis work
Hi guys, I just need to run something past you. when I look at my on page analysis I have 5 key terms I am focusing on. For instance one of them is "computer backup". According to the report the current grade is 'F' when looking at site page "/" which I assume is the home page.
Moz Pro | | cubetech
When I do a lookup on other pages of the site it gets a ranking of A. Which is good. But since the hompage ranking went from A to F my rankings have definitely been affect. So i guess my questions are: does "/" mean the hompage, or all pages overall. What should I really be looking at here. I am assuming that you select certain pages to target certain key words. Should i be focusing like this, or more to the "/". Thanks Guys hoping to clear this one up.0 -
Can I exclude pages from my Crawl Diagnostics?
Right now my crawl diagnostic information is being skewed because it's including the onsite search from my website. Is there a way to remove certain pages like search from the errors and warnings of the crawl diagnostic? My search pages are coming up as: Long URL Title Element Too Long Missing Meta Description Blocked by meta-robots (Which is how I want it) Rel Canonical Here is what the crawl diagnostic thinks my page URL looks like: website.com/search/gutter%25252525252525252525252525252525252525252525252525252525 252525252525252525252525252525252525252525252525252525252525252 525252525252525252525252525252525252525252525252525252525252525 252525252525252525252525252525252525252525252525252525252525252 52525252525252525252525252525252525252525252525252Bcleaning/ Thank you, Jonathan
Moz Pro | | JonathanGoodman0