Ajax Module Crawability vs. WMT Fetch & Render
-
Recently a module was built into the homepage to pull in content from an outside source via Ajax and I'm curious about the overall crawability of the content.
In WMT, if I fetch & render the content it displays correctly, but if I view source all I am seeing is the empty container. Should I take additional steps so that the actual AJAX content displays in my source code, or am I "good" since the content does display correctly when I fetch & render?
-
Hey Scott,
You're good. If you see it in the fetch and render, you're seeing it as Googlebot sees it.
Google has had capabilities to crawl Ajax content for some time now. And while you don't see the content when you view source, that's not a big a problem. As long as the content is on the page at load Google should not have a problem indexing it. So after load, you should look in the Inspect Element section, that's how Google will see the content.
Where there may be crawlability issues is in content that requires user action to display. Google is inconsistent with how many actions they will attempt in order to index content.
-Mike
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Forced to remove Categories with high volume & revenue
Hi everyone I've been forced to remove level 4 & 5 categories (e.g. example.com/level-2/level-3**/level-4/level-5/**) from our website, even though they're getting plenty of traffic, revenue and are ranking for some of our keywords. The argument is customers were using refinement/filters more than clicking into categories, and a new backend system is coming into the business and these need to be removed anyway. We've done this before and seen a drop in visibility, revenue & traffic in these areas, but we're going ahead with another batch of removals anyway. I was wondering if anyone has any experience in fixing a problem like this? I've been told the categories will not be returning and have to 301 them, so need to find a workaround to get eligible for ranking for these Keywords again. I've been looking at using the refinements to make it look like a category (change URL to a clean one, update Page Title, Meta Description, H1, remove text from core page, when refinement is clicked) but not sure what kind of knock-on effects this will have, if it even works! Hope you can help! I've probably missed some details so let me know if you need more info!!! Thanks
Intermediate & Advanced SEO | | Frankie-BTDublin0 -
Proxy Servers & SEO
Does putting a blog on a proxy server (the pointed at the main site) hurt SEO? i.e. can Google tell? And if they can, does it matter? My server people won't use PHP on their servers but we want a Wordpress blog. So their suggested solution is that they put the blog on a proxy server and point it at the ourdomain.com/blog subfolder on our site. So to all intents and purposes it's hosted in the same place. They assure me this is normal practice and point out that our (main site) images are already being sourced from a CDN. Obviously we'll deal with Google not seeing two separate versions of the same site. But apart from this, is there any negative effect we could suffer from in SEO terms?
Intermediate & Advanced SEO | | abisti20 -
ExampleSite.com vs ExampleSite.com.br
What would you say to a client who is concerned he'd have to run around buying his .com.??? in alot of other countries. Thanks!
Intermediate & Advanced SEO | | 945010 -
Display:None CSS & SEO
Hi A while back I was told that using the display:none tag to hide content you want minimised is bad for onpage SEO - is this the case? It's not that we want to hide it from Google, we just don't want it taking up a huge amount of space on product pages. I have found some of these on our site, and want to know how bad they are. Is the content under the tag going to be ignored? Thank you
Intermediate & Advanced SEO | | BeckyKey0 -
Avoiding Duplicate Content with Used Car Listings Database: Robots.txt vs Noindex vs Hash URLs (Help!)
Hi Guys, We have developed a plugin that allows us to display used vehicle listings from a centralized, third-party database. The functionality works similar to autotrader.com or cargurus.com, and there are two primary components: 1. Vehicle Listings Pages: this is the page where the user can use various filters to narrow the vehicle listings to find the vehicle they want.
Intermediate & Advanced SEO | | browndoginteractive
2. Vehicle Details Pages: this is the page where the user actually views the details about said vehicle. It is served up via Ajax, in a dialog box on the Vehicle Listings Pages. Example functionality: http://screencast.com/t/kArKm4tBo The Vehicle Listings pages (#1), we do want indexed and to rank. These pages have additional content besides the vehicle listings themselves, and those results are randomized or sliced/diced in different and unique ways. They're also updated twice per day. We do not want to index #2, the Vehicle Details pages, as these pages appear and disappear all of the time, based on dealer inventory, and don't have much value in the SERPs. Additionally, other sites such as autotrader.com, Yahoo Autos, and others draw from this same database, so we're worried about duplicate content. For instance, entering a snippet of dealer-provided content for one specific listing that Google indexed yielded 8,200+ results: Example Google query. We did not originally think that Google would even be able to index these pages, as they are served up via Ajax. However, it seems we were wrong, as Google has already begun indexing them. Not only is duplicate content an issue, but these pages are not meant for visitors to navigate to directly! If a user were to navigate to the url directly, from the SERPs, they would see a page that isn't styled right. Now we have to determine the right solution to keep these pages out of the index: robots.txt, noindex meta tags, or hash (#) internal links. Robots.txt Advantages: Super easy to implement Conserves crawl budget for large sites Ensures crawler doesn't get stuck. After all, if our website only has 500 pages that we really want indexed and ranked, and vehicle details pages constitute another 1,000,000,000 pages, it doesn't seem to make sense to make Googlebot crawl all of those pages. Robots.txt Disadvantages: Doesn't prevent pages from being indexed, as we've seen, probably because there are internal links to these pages. We could nofollow these internal links, thereby minimizing indexation, but this would lead to each 10-25 noindex internal links on each Vehicle Listings page (will Google think we're pagerank sculpting?) Noindex Advantages: Does prevent vehicle details pages from being indexed Allows ALL pages to be crawled (advantage?) Noindex Disadvantages: Difficult to implement (vehicle details pages are served using ajax, so they have no tag. Solution would have to involve X-Robots-Tag HTTP header and Apache, sending a noindex tag based on querystring variables, similar to this stackoverflow solution. This means the plugin functionality is no longer self-contained, and some hosts may not allow these types of Apache rewrites (as I understand it) Forces (or rather allows) Googlebot to crawl hundreds of thousands of noindex pages. I say "force" because of the crawl budget required. Crawler could get stuck/lost in so many pages, and my not like crawling a site with 1,000,000,000 pages, 99.9% of which are noindexed. Cannot be used in conjunction with robots.txt. After all, crawler never reads noindex meta tag if blocked by robots.txt Hash (#) URL Advantages: By using for links on Vehicle Listing pages to Vehicle Details pages (such as "Contact Seller" buttons), coupled with Javascript, crawler won't be able to follow/crawl these links. Best of both worlds: crawl budget isn't overtaxed by thousands of noindex pages, and internal links used to index robots.txt-disallowed pages are gone. Accomplishes same thing as "nofollowing" these links, but without looking like pagerank sculpting (?) Does not require complex Apache stuff Hash (#) URL Disdvantages: Is Google suspicious of sites with (some) internal links structured like this, since they can't crawl/follow them? Initially, we implemented robots.txt--the "sledgehammer solution." We figured that we'd have a happier crawler this way, as it wouldn't have to crawl zillions of partially duplicate vehicle details pages, and we wanted it to be like these pages didn't even exist. However, Google seems to be indexing many of these pages anyway, probably based on internal links pointing to them. We could nofollow the links pointing to these pages, but we don't want it to look like we're pagerank sculpting or something like that. If we implement noindex on these pages (and doing so is a difficult task itself), then we will be certain these pages aren't indexed. However, to do so we will have to remove the robots.txt disallowal, in order to let the crawler read the noindex tag on these pages. Intuitively, it doesn't make sense to me to make googlebot crawl zillions of vehicle details pages, all of which are noindexed, and it could easily get stuck/lost/etc. It seems like a waste of resources, and in some shadowy way bad for SEO. My developers are pushing for the third solution: using the hash URLs. This works on all hosts and keeps all functionality in the plugin self-contained (unlike noindex), and conserves crawl budget while keeping vehicle details page out of the index (unlike robots.txt). But I don't want Google to slap us 6-12 months from now because it doesn't like links like these (). Any thoughts or advice you guys have would be hugely appreciated, as I've been going in circles, circles, circles on this for a couple of days now. Also, I can provide a test site URL if you'd like to see the functionality in action.0 -
Shopify Product Variants vs Separate Product Pages
Let's say I have 10 different models of hats, and each hat has 5 colors. I have two routes I could take: a) Make 50 separate product pages Pros: -Better usability for customer because they can shop for just masks of a specific color. We can sort our collections to only show our red hats. -Help SEO with specific kw title pages (red boston bruins hat vs boston bruins hat). Cons: -Duplicate Content: Hat model in one color will have almost identical description as the same hat in a different color (from a usability and consistency standpoint, we'd want to leave descriptions the same for identical products, switching out only the color) b) Have 10 products listed, each with 5 color variants Pros: -More elegant and organized -NO duplicate Content Cons: -Losing out on color specific search terms -Customer might look at our 'red hats' collection, but shopify will only show the 'default' image of the hat, which could be another color. That's not ideal for usability/conversions. Not sure which route to take. I'm sure other vendors must have faced this issue before. What are your thoughts?
Intermediate & Advanced SEO | | birchlore0 -
Google Places & Multiple Listings
Our client used to have a listing in each city, but after updating the addresses they were forever under review. Google said that businesses serving customers at their locations can only list their primary office. Back when this client had multiple city listings, all addresses but one were UPS boxes. If they are to change back to "No, all customers come to the business location," can they once again submit a listing for each city using these addresses? Yes, I realize they are UPS boxes, but they insist on being listed for each city.
Intermediate & Advanced SEO | | elcrazyhorse0 -
Panda/Penguin & more than one services site in niche
Hello, My friend has a personal development training site. I have been advised not to make separate personal coaching sites for the owners of the training sites. Do you have experience that Panda/Penguin could penalize for separate sites in a similar niche? Do you need any more info to give a good response? Thank you.
Intermediate & Advanced SEO | | BobGW0