How much content does Google Crawl on your site?
-
Hi,
We've had a debate around the office where some people believe that Google only crawls the first 150-200 words on a page and some people believe that they priority content that is above the fold and other people believe that all content has the same priority. Can you help us?
Thanks,
Matt -
Google actually crawls 150kb, excluding css files, images, etc.
150kb is much more than 200 words, and the experiment suggested by Mr Bennett proves it.
-
They definitely crawl more than that, and it's easy to prove as well.
Pick a long page, such as the Wikipedia page about London. Choose a block of text from near the bottom of that page, I've selected this:
in the south-western suburb of Wimbledon.[252] Other key events are the annual mass-participation London Marathon which sees some 35,000 runners
If you search for that text you will see the Wikipedia page in the results. If they only crawled the first 200 words they wouldn't have been able to find that result.
Prioritising is harder to demonstrate (and probably also to define!). However it is generally believe that greater importance is given to text towards the top of the page. That is logical if you consider how the majority of documents are structured.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Too much internal linking?
Hi everyone, Too much of anything is not good. In terms of internal linking, how many are too many? I read that the recommended internal links are about 100 links per page otherwise it dilutes the page's link equity. I have a concern about one of our websites - according to search console, the homepage has 923 internal links. All the pages have a corresponding /feed page added to the page URL, which is really weird (is this caused by a plugin?). The site also has an e-com feature, but it is not used as the site is essentially a brochure and customers are encouraged to visit the shop. I assume the e-com feature also increases this number. On the other hand, one of the competitors we are tracking has 1 internal link site-wide. Ours is at 45,000 site-wide. How is it possible to only have 1 internal link? Is this a Moz bug? I know we also need to reduce our internal links badly, however, I'm not sure where to start. I don't know how these internal links are linked together - some aren't in the copy or navigation menu. When I scan the homepage links using 'check my links', the total links identified for the homepage is only 170. kAeYlTM
On-Page Optimization | | nhhernandez0 -
Can bots crawl this homepage's content?
The website is https://ashleydouglas.com.au/ I tried using http://www.seo-browser.com/ to see if bots could see the content on the site, but the tool was unable to retrieve the page. I used mobile-friendly test and it just rendered some menu links - no content and images. I also used Fetch and Render on Search Console. The result for 'how google sees the page' and 'how a visitor sees the page' are the same and only showing the main header image. Anything below isn't shown. Does this mean that bots can't actually read all content on the page past the header image? I'm not well versed with what's going on with the code. Why are the elements below the header not rendering? Is it the theme? Plugins? Thank you.
On-Page Optimization | | nhhernandez0 -
Consolidating a Large Site with Duplicate Content
I will be restructuring a large website for an OEM. They provide products & services for multiple industries, and the product/service offering is identical across all industries. I was looking at the site structure and ran a crawl test, and learned they have a LOT of duplicate content out there because of the way they set up their website. They have a page in the navigation for “solution”, aka what industry you are in. Once that is selected, you are taken to a landing page, and from there, given many options to explore products, read blogs, learn about the business, and contact them. The main navigation is removed. The URL structure is set up with folders, so no matter what you select after you go to your industry, the URL will be “domain.com/industry/next-page”. The product offerings, blogs available, and contact us pages do not vary by industry, so the content that can be found on “domain.com/industry-1/product-1” is identical to the content found on “domain.com/industry-2/product-1” and so-on and so-forth. This is a large site with a fair amount of traffic because it’s a pretty substantial OEM. Most of their content, however, is competing with itself because most of the pages on their website have duplicate content. I won’t begin my work until I can dive in to their GA and have more in-depth conversations with them about what kind of activity they’re tracking and why they set up the website this way. However, I don’t know how strategic they were in this set up and I don’t think they were aware that they had duplicate content. My first thought would be to work towards consolidating the way their site is set up, so we don’t spread the link-equity of “product-1” content, and direct all industries to one page, and track conversion paths a different way. However, I’ve never dealt with a site structure of this magnitude and don’t want to risk messing up their domain authority, missing redirect or URL mapping opportunities, or ruin the fact that their site is still performing well, even though multiple pages have the same content (most of which have high page authority and search visibility). I was curious if anyone has dealt with this before and if they have any recommendations for tackling something like this?
On-Page Optimization | | cassy_rich0 -
Site Structure question?
Hey guys, Sorry for posting this again but the last thread got a bit too wayword. I'll sum it up better here. We're producing a WordPress theme every 3-6 months. Each is differently niched (eg: ecommerce, restaurant, magazine, etc...) Which option is better for our products going forward (even the ones we've yet to launch...eg...which method will get future projects more "trust juice" from google): A: create a subfolder for each theme eg: http://bigbangthemes.net/TicketLab_WP/wordpress-ticket-system & http://bigbangthemes.net/Showoff_WP/landing-page/ **This is currently what we're doing.**B: have them all under bigbangthemes.net/wordpress-themes/ eg: bigbangthemes.net/wordpress-themes/wordpress-ticket-system & bigbangthemes.net/wordpress-themes/showoff-startup-agency-theme Thanks for the help!
On-Page Optimization | | andy.bigbangthemes0 -
Simple on-site SEO - bet practice for keywords in content
Hello, The Moz on-page grader will give a grade of A if the keyword appears exactly in the content at least one time. If there are 500 words and a lot of it is about the main keyword, what have you found to be important to look for beyond the on-page grader - beyond the one exact instance of the keyword? I'm specifically talking just about keywords in the content. My guess is that it needs to occur 3 or 4 times in different forms and at least once exactly, but the on-page grader doesn't require it. What have you found?
On-Page Optimization | | BobGW0 -
On site SEO review please
I'd appreciate it its anyone could take the time to review my on site SEO and suggest improvements. it's an adult dating site at http://www.local-sex-search. All pages can be found at http://www.local-sex-search.com/sitemap
On-Page Optimization | | SamCUK0 -
Google Places Problem
This may have been answered before but I have 2 questions. When I placed a business in Google Places, the "generic" ranking fell off the map. I now just have the 1 line Google places reference and that is all I can find. How can I get around that and get my 4 line description to show again? Do I have to delete my Places account? Before the Google Places account was built, the company was moving up the SERP ranks, now he is on pg 1 for Places but the other SERP positions have disappeared. This is true for all the keywords we are targeting. If there is not a Places reference he shows on Pg 3-5 (given the website is 4 weeks old, I think this is not bad). For the same client, he that services many of the surrounding communities. How do I get Google to recognize the various towns he services during a search? He places well for his "home" town but not at all for the other towns. if it helps any, the website is www.myairstat.com. Thanks for the help. Scott
On-Page Optimization | | scott5180 -
Are Content in Inline Javascript and Collapsible Considered Cloaking to Google?
Hi, I would like to save space in my website and do not want my other products to be pushed down below the first fold. In order to do that, I have decided to add content inside inline javascript or using collapsible. For collapsible, I may be using "show/hide" button or "read more" button to show the whole content. So does content in Javascript and collapsible considered hiding from Google? If it is, then I have to think of other options. Thanks.
On-Page Optimization | | globalsources.com0