Welcome to the Q&A Forum

Hey Scott,

You're good. If you see it in the fetch and render, you're seeing it as Googlebot sees it.

Google has had capabilities to crawl Ajax content for some time now. And while you don't see the content when you view source, that's not a big a problem. As long as the content is on the page at load Google should not have a problem indexing it. So after load, you should look in the Inspect Element section, that's how Google will see the content.

Where there may be crawlability issues is in content that requires user action to display. Google is inconsistent with how many actions they will attempt in order to index content.

-Mike

Hey Cyndee,

Your issue has to do with how this is coded. Let me explain.

Here's what your paginated numbers at the bottom look like in the code:

<a title="2" data-bvcfg="3520493" name="BV_TrackingTag_Review_Display_PageNumber_2" data-bvjsref="http://improvements.ugc.bazaarvoice.com/0048-en_us/414441/reviews.djs?format=embeddedhtml&page=2&scrollToTop=true" <strong="">href="javascript://">2</a>

Notice that the "href" parameter of the anchor tag has no direct URL and because of that Google doesn't crawl to the next page in the series because there's no actual link. What would be ideal is if you had the actual URL to the second page so that it is accessible to Google as the href tag. Granted, Google will likely come back to these pages with the more feature-rich crawler and be able to access the content, but that could potentially take a long time or in fact never happen. I believe this is a function of how BazaarVoice operates, although I haven't had enough experience with it to know. A view-all page would help you get around the problem, but again, I'm not sure how that works with regard to BazaarVoice.

You can also use rel-prev and rel-next to connect the pages, but that directive often has spotty results.

-Mike

Hey Carla,

I'm not entirely sure what you're saying with:

"one of the crawl reports is showing tons of 404's for the "smartphone" bot and with very recent crawl dates. If our website is responsive, and we do not have a mobile version of the website I do not understand why the desktop report version has tons of 404's and yet the smartphone does not. I think I am not understanding something conceptually."

You say that the smartphone bot is seeing tons of 404s and the desktop report is showing tons of 404s, but the smartphone does not. If you can clarify that, I can probably better answer your question.

However, the answer is likely that Google may decide not to crawl URLs that it has already identified as 404s in one context. That is to say if they identify URLs on the mobile device as 404s they will know not to crawl them if they encounter them on desktop and vice versa.

-Mike

Hey Alan,

Not sure if your issue was related to the DDoS attack or not, but I was able to replicate it now. Make sure that there is no spacing before the URL when you paste it into the on-page grader.

To track rankings for keywords over time you'll have to set up a Campaign which it sounds like you have. Once you've done that you can find the rankings in the Search section of the dashboard. This is also where you can filter by the label you've tagged your keywords with.

Alternatively you can set up keywords rankings in the rank tracker https://moz.com/researchtools/rank-tracker but you won't get the tagging or email report functionality.

-Mike

Hey David,

Quick question: It's been over a month. Are you still having this issue?

From what you've said it sounds like the site verification failure is likely what caused this. However it is worth noting that even if Google ingests a sitemap it doesn't necessarily mean they will index the pages. I would suggest you review your site log files to see if Google is indeed crawling those new URLs, if they are they are likely discarding them because you are fulfilling your crawl allocation with more important pages and these new pages haven't been given any link or social activity.

-Mike

Hey Hardley,

I concur with Kevin's point below. Giving all of your traffic to YouTube isn't the best idea. Phil Nottingham's research has shown that an extremely low number of users click through to links in YouTube descriptions. Naturally, YouTube is the 2nd biggest search engine on the web and people will find you within it, but you should consider self-hosting your videos so the traffic lands on your website. Wistia is a great solution for that.

All that said, YouTube pages pretty much follow the same rules as any other page on the web with regard to Google. On-page optimization with keyword-relevant copy, titles and metadata and off-page optimization in the form of links. Do those things as you normally would and your videos will rank in Google. The on-page factors and engagement metrics lead to its ranking within YouTube. YouTube also has its own Keyword Research tool so you can identify the terms that need to be placed in your tags, descriptions and titles.

-Mike

Hey Chris,

I agree with William. You should always take a peek at the relevant keyword research tools to see what vocabulary people are using to discuss your topic, but don't worry too much about having a post title that mirrors those phrases exactly. Even if the phrase is long tail, Google's keyword tool is just an estimation so there may be more people searching for a given term than you think. Check out UberSuggest and also look at keyword trends in Social Media using Bottlenose's Sonar tool. Those may give you more of an indication of what keywords people are using.

Definitely don't go haywire on shoehorning keywords into titles and headings, but do be sure what keywords that a given post owns so that you can go back through your site and link to those new posts from old posts with the relevant keywords.

-Mike

Hey Arthur,

Your CSS is not the issue here. None of your content is is hidden or "cloaked." You can see this when you disable CSS and JavaScript with the Web Developer plugin to get a sense of what the text crawler would see.  However the issue is definitely that the content is indeed thin and you're not offering anything beyond what can be found in the product specifications sheet on the manufacturers website.

You should absolutely consider some more text-based content. You may consider having user generated content in allowing people to  comment on the differences between the two and what they like better. Responses would be inherently keyword-rich and the approach allows you to scale it across numerous pages.

However the UX is very nice. Best of luck and let us know if you might be interested in sharing your progress.

-Mike

Is this from all channels? Or are you just specifically looking for the increase in referral traffic? I need a little more information to determine what's going on here. Feel free to direct message me some more details if you're not comfortable sharing them publicly.

-Mike

Hey Tom,

Can you confirm that there has been no penalty on your blog domain? If not, you should be careful because penalties are known to hop 301s and JohnMu recently revealed that penalties will follow content without a 301.

That said I would prioritize what you redirect based on what has links and shares. There's no reason to set up redirects for page that no one is visiting or linking to. Go with your first option, but 410 them to get them out of the index faster.

-Mike

Hey Mark,

In answer to your question, yes Google does see more value in body links than in navigation links. That is because they are pretty good at detecting what is the reoccurring code across the site. As Dave said the navigation itself is a question of usability, so build it in a such a way that users can easily get to the right pages quickly.

With the body links, however, it is about branding the other pages throughout the site for relevance. As you might imagine, it's a little harder to create relevance from just a list of links. Whereas if you have a paragraph of text surrounding a link you can programmatically identify context.

All that is to say, what you've read is right. There is more value in the body links for search engines due to context and there is more value for the user in the navigation due to user experience. Account for both, but don't go overboard with either.

-Mike

You should fill out a Change of Address in Webmaster Tools as well and then confirm that all of the old URLs are out of the index. The redirects however should remain in place as long as you have links pointing from the old site.

Hey Chris,

If your domain is onedirection.net and people are linking to your images on cdn.onedirection.net you are seeing some benefit, just not as much as if they were on onedirection.net root domain. If that's the case that's why they are coming up in your Just Discovered links.

If however cdn.onedirection.net is your CDN completely irrespective of your domain and your site is on another domain like justinbieber.com then nope you won't get any link equity from that.

-Mike

Hey Gary,

Is your traffic OK? Have you been banned?

If neither of those things are issues I wouldn't worry about toolbar PageRank.

-Mike

Hey Olivier,

You could detect the user agent and hide the button. The difference isn't substantial enough to be called cloaking.

Or you could make the button not actually a button tag, but another tag with that traps clicks with a JS event. I'm not sure Google's headless browser is smart enough to automate that. I would try this first and if it doesn't work switch to the user agent detection idea.

Let us know how it goes!

-Mike

I think that's a great idea actually.

Hey Jared,

As you might imagine I have done this. In fact we did it with the iAcquire.com redesign when I was still there. Basically we built something custom to grab the social counts and display them in our custom social share buttons. Our Creative Director wanted to make custom share buttons because they are otherwise different sizes and it looks silly. Then we just set the old URL in one place in the CMS and it pinged SharedCount.com's API to get all the different share counts and display them.

A quick update to my post, from what I understand, social shares are hopping 301s for Google +1s only right now.

As far as your issue with the new shares being attributed to old URLs, that's something I've thought about as well. You could potentially build a similar solution that we had, but allow the new social shares to go with the new URL and check the number of social shares between the new and old URL whenever the page loads. Once the new URL surpasses the old one's share count just skip grabbing the old URL's share counts altogether and use the new URL for both displaying share counts and capturing new shares.

Let me know if that helps. I'm sure we can perfect a solution for you, but it will involve getting a little creative.

-Mike

Hey Gagan,

So I think you're question is will content on your staging site still get indexed despite using robots.txt? The answer is yes, sometimes that does happen especially if a lot of people link to it. The best way to keep content out of the index would be to use the meta robots tag with noindex, nofollow. Search engines are much better about adhering to those than robots.txt.

Let us know if you run into any problems!

-Mike

Yes Open Site Explorer does allow you to compare up to 4 URLs to a given URL of your choosing.

I'm not quite understanding the purposes of having two pages here. I took a quick glance, but it looks like the 2 pages are two different views of the same content. I would just consolidate the pages into one by 301 redirecting the text-based view to the graphical view that way everyone wins.

-Mike

Depending on the site they may be valuable. Soundcloud for example has nofollow attributes on links so there's no direct link equity passed through those links.

Hey Edward,

The second link is indeed in Google. You can see it cached at this link. It also has links in OpenSiteExplorer.

The first one however is not. I've checked several different link indices and it would appear that the page does not have any links to it. I've also crawled the site and the page never appeared. So basically this is an orphaned page meaning there is no crawl path to it and no crawler will ever see it.

Hope that helps.

-Mike

Hey Josh,

I'm leery of the "301 redirect add-on" and that would be the first thing that I double-check. I'd pull all of the links that you guys have run all the link targets through a web crawler such as Screaming Frog to make sure that they aren't your 404s. My main concern is that your link equity has not been maintained.

I'd also look at the quality of the links as was mentioned later in this thread, then I'd start looking at whether your content is what we'd consider thin.

There's a lot of moving parts here, but without more information on the site I can't give you a more definitive answer.

If I'm understanding you properly that is not a good idea. Google will potentially not be able to crawl your content that way. Regardless of the pagination you still want to be able to surface your content for search. You might consider using push state and pop state to give crawl paths and maintain the quick loading user experience. Be sure to make it gracefully degrade for lesser browsers.

Hey Charles,

A spammy link is a spammy link whether it comes from .edu sites or not. .Edu sites don't get any special considerations in the ways of spam. If the links make sense in context of the site that you are getting them from, then I wouldn't worry. However if you're getting a ton of list site links from pages that have nothing to do with your space at all, then I would worry about those especially if they are sitewides or on pages that have other clear indications of spam like adsense or too many ad blocks.

-Mike