Welcome to the Q&A Forum

The mobile first index is not live as of today

This is from March of this year

http://searchengineland.com/googles-mobile-first-index-still-months-away-271851

"Don’t freak out. That’s what Google’s Gary Illyes repeated this week — at least three times — to SEOs and webmasters who might be concerned about the upcoming switch to a mobile-first index.

“The team behind the mobile-first index wants it to launch this year,” Illyes told our SMX West conference on Wednesday. “We’re still experimenting. We don’t have a timeline. It could be a few months or quarters, but it’s definitely not weeks [away]. Don’t freak out, especially if you have a responsive site.”

Later in June of this year

http://searchengineland.com/googles-mobile-first-index-likely-not-coming-2018-earliest-277074

"Google is “probably many quarters away” from launching its mobile-first index. So said Gary Illyes, Google webmaster trends analyst, during a crowded session Tuesday afternoon at our SMX Advanced conference in Seattle.

“It’s going to be a big change, but don’t freak out,” Illyes said.

SEOs and webmasters have been wondering and waiting for a couple of years now for news on when the mobile-first index will roll out. Illyes wasn’t able to give an exact answer to that question today.

“We don’t have a timeline for the launch yet,” Illyes said. “We have some ideas for when this will launch, but it’s probably many quarters away. Our engineers’ timeline was initially end of 2017. Right now, we think more 2018. ”

The point I am making here is that you need to follow the current Google guidance as Roman mention so that you can rank now. Yes, you will probably need to tweak things once the mobile first index comes out, but until it does and Google then updates guidance on how you should setup the canonical and alternative tags as suggested.

You other option to act on now is to convert to a responsive site, work with one set of URLs and make sure that when your responsive page shows that it passes mobile friendly tests etc. You may end up with a slower page, but you would need to test and see how much of a difference makes. If the page is still fast overall, you should still be good.

WIthout a specific example, there are a couple of options here. I am going to assume that you have an ecommerce site where parameters are being used for sort functions on search results or different options on a given product.

I know you may not be able to do this, but using parameters in this case is just a bad idea to start with. If you can (and I know this can be difficult) find a way to rework this so that your site functions without the use of parameters.

You could use canonicals, but then Google would still be crawling all those pages and then go through the process of using the canonical link to find out what page is canonical. That is a big waste of Google's time. Why waste Googlebots time on crawling a bunch of pages that you do not want to have crawled anyway? I would rather Googlebot focus on crawling your most important pages.

You can use the robots.txt file to stop Google from crawling sections of your site.  The only issue with this is that if some of your pages with a bunch of parameters in them are ranking, once you tell Google to stop crawling it, you would then lose traffic.

It is not that Google does not "like" robot.txt to block them, or that they do not "like" the use of the canonical tag, it is just that there are directives that Google will follow in a certain way and so if not implemented correctly or in the wrong sequence can cause negative results because you have basically told Google to do something without fully understanding what will happen.

Here is what I would do.  Long version for long term success

1. Look at Google Analytics (or other Analytics) and Moz tools and see what pages are ranking and sending you traffic. Make note of your results.

2. Think of the most simple way that you could organize your site that would be logical to your users and would allow Google to crawl every page you deem important. Creating a hierarchical sitemap is a good way to do this. How does this relate to what you found in #1.

3. Rework your URL structure to reflect what you found in #2 without using parameters. If you have to use parameters, then make sure Google can crawl your basic sitemap without using any of the parameters. Use robots.txt to then block the crawling of any parameters on your site. You have now ensured that Google can crawl and will rank pages without parameters and you are not hiding any important pages or page information on a page that uses parameters.

There are other reasons not to use parameters (e.g. easier for users remember, tend to be shorter, etc), so think about if you want to get rid of them.

4. 301 redirect all your main traffic pages from the old URL structure to the new URL structure.  Show 404s for all the old pages including the ones with parameters.  That way all the good pages will move to the new URL structure and the bad ones will go away.

Now, if you are stuck using parameters. I would do a variant of the above.  Still see if there are any important or well ranked pages that use parameters. Consider if there is a way to use the canonical on those pages to get Google to the right page to know what should rank. All the other pages I would use the noindex directive to get them out of the Google index, then later use robots to block Google crawling them.  You want to do this in sequence as if you block Google first, it will never see the noindex directive.

Now, everything I said above is generally "correct" but depending on your situation, things may need to be tweaked. I hope the information I gave might help with you being able to work out the best options for what works for your site and your customers.

Good luck!

We have actually been doing this on one of our sites where we have several thousand articles going all the way back to the late 90s.  Here is what we do / our process (I am not including how to select articles here, just what to do once they are selected).

1. Really take the time to update the article. Ask the questions, "How can we improve it? Can we give better information? Better graphics?  Better references? Can we improve conversion?"  2) Republish with a new date on the page.  Sometimes add an editor's note on how this is an updated version of the older article.  3) Keep the same URL to preserve link equity etc or 301 to new url if needed 4) mix these in with new articles as a part of our publication schedule.

We have done this for years and have not run into issues. I do not think Google sees this as spammy as long as you are really taking the time to improve your articles. John M. and Gary I. have stated unequivocally that Google likes it when you improve your content. We have done the above, it has not been dangerous at all. Our content is better overall. In some cases where we really focused on conversion, we not only got more traffic, but converted better. Doing this will only benefit your visitors, which usually translates into Google liking the result.

I would ask, why take a few months where you only recycle content, to just mixing it up all year long?  If you were going to designate 3 months of the year to just update content, then why not take the 3rd week of the month each month or every Wednesday and do the same thing instead.  You accomplish the same thing, but spread it out. Make it a feature! Flashback Friday etc.

Bonus idea - make sure you get the schema right

We have something new with our process. Previously, we only marked up the publication date in schema. So when we republished, we would change the publication date in the schema as well to the new pub date.  Now that Google requires a pub date and last modified date in schema we have changed our process. When we republish content, we will leave the original publication date as the publication date marked up in schema and then put the new date that the article is being published marked up as last modified in schema. This is a much more clearer and accurate representation to Google as what you are doing with the article.

We are also displaying the last modified date to the user as the primary date, with the publication date made secondary. The intent here is that we want to show that this is an article that has been recently updated to the user so they know the information is current.

To get this to work properly, we had to rework how our CMS interacts with content on both published date and last modified date, but in the end, I think we are giving better signals to Google and users on the statuses of our articles.

I would just follow and say that using Google or any other company that large is not the best example for most businesses.

Google does not need to worry about SEO. It has one of the most authoritative websites on the internet. They have baked in traffic due to android, chrome and map users. Yes the sub domain works for them from a product and technical standpoint. They have different groups using different technologies different servers/resources so they need the subdomains to make that all work.

As I read the question, this person is not in that situation. This brand needs link equity to be shared. Links are still super important, I don't see why they would want to dilute that by using sub domains. Everything they want to accomplish can be done w a sub folder.

Nah - I say subdirectory

You get all the control, plus all the link equity is shared across all site. Rising tides raises all ships. I have done this with independent separate brands under a single brand even. It works.

Yes you can rank for subdomains. But the link equity is not shared. Unless you want all that extra work.

More reading

https://moz.com/community/q/the-great-subdomain-vs-subfolder-debate-what-is-the-best-answer

https://moz.com/blog/subdomains-vs-subfolders-rel-canonical-vs-301-how-to-structure-links-optimally-for-seo-whiteboard-friday

You did not lose the traffic - it is just being classified as something else. I am going to predict that you saw an increase in Organic traffic. Right?

http://searchengineland.com/60-direct-traffic-actually-seo-195415

Above is a classic article of how Groupon deindexed page to look at this and they found for their traffic 60% of direct traffic was organic. Your % will vary but you get the idea.

Your specific issue is due to the change in http to https - as mentioned in the Groupon article https://tools.ietf.org/html/rfc2616#section-15.1.3 specifies that when moving from http to https referral data is supposed to be stripped out.

You did not lose traffic but maintained it or gained it. If I could borrow $20 from EGOL I would bet that your revenue increase is possibly due to the fact that your site is now 100% https and people trust it more, which can impact purchase intent.

Cheers!

Here is what I would do

1. Disavow the domain that is linking to you from the adult site(s).

2. The fact that Google search console is showing that you have an internal page linking as well makes me want to know a) have you always owned this domain and maybe someone previously did link internally like this or b) you may have been or are hacked

In the case of b) this can be really tricky.  I once had a site that in a crawl it was showing sitewide links to various external sites that we should not be linking to. When I looked at the internal pages via my browser, there was no link as far as I could see even though it showed up on the crawler report.

Here was the trick. The hacker had setup a script to only show the link when a bot was viewing the page. Plus, we were running mirrored servers and they had only hacked one server. So, the links only showed up when you were spidering a specific mirrored instance as a bot.

So thanks to the hacking, not only were we showing bad links to bad sites, we were doing this through cloaking methodology. Two strikes against us. Luckily we picked this up pretty quick and fixed immediately.

Use a spidering program or browser program to show a user agent of Googlebot and go visit your pages that are linking internally. You might be surprised.

Summary

Googlebot has a very long memory. It may be that this was an old issue that was fixed long ago. If that was the case, just show the 404s for the pages that do not exist, and disavow the bad domain and move on. Make sure that you have not been hacked as this would also be why this is showing.

Regardless, the fact that Google did find it at one point, you need to make sure you resolve. Pull all the URLs into a spreadsheet and run Screaming Frog in list mode to check them all to make sure you fix all of it.

You need to figure out what is going on with u.couponshop.co.uk as there are about 108 pages listed that get mixed up when you search

site:couponshop.co.uk/

If you go to page 9 you then get your https://www.couponshop.com results. What is interesting is out of the 13,000 pages listed the u.couponshop URLs are listed before the www ones.  This is an issue.

Those old URLs need to be redirected to the proper URLs on the current site if those URL get traffic or have links to them.

site:u.couponshop.co.uk/

Can be used to find the u.couponshop pages.  You will need to look in your server logs to see how often Google is crawling them. You may find that Google is crawling them a fair amount and so you need to get them taken care of. You also want to see if there are any sites linking to these URLs as well to determine if they are of value to you.

Somewhere on your server you are serving up pages with redirects and Google is crawling them and cataloging them and may not be paying as much attention to your new website.

Hello!

If you want to do this. You need to setup your rel alternate and canonical links

https://developers.google.com/webmasters/mobile-sites/mobile-seo/separate-urls

I am not sure if the https vs http designation is that big of a deal as you are already setting up a separate set of URLs with the www. and the m.

What is interesting here is that with the new mobile first update occurring, I am not sure that this page will eventually be updated to have the canonicals point to the mobile version vs the desktop version as mentioned in the link above. Likewise, the https is favored for ranking so there may be another reason to canonical that direction, but you would need to test and see. You may find that due to the mobile first initiative and Googles preference for https that your m. pages might do better.

Generally, I would find a way to move away from the m. setup and simply run a responsive site on https://www - that is going to get you the best bang for your buck.

Thanks for the comments Gianluca, I am not as up on international stuff and so appreciate the input.

If I am reading your pages correctly, /farsi is the exact same page as your home page, just translated into Farsi.

Here are some things to think about.

1. you have no explicit declaration of what language is on this page in your markup. Google can probably figure it out, but you should mark it explicitly. When I brought up your page in Chrome, Chrome listed that the page was in Persian. It may be that persian is the same as farsi (I dont know) but, little details around being very explicit with the language designation can help.

2. If the /farsi section of your website is an exact translation of the English version of the website, you need to use the hreflang to explain this to Google.  Google may not be 100% sure of the relationships of the two sets of pages.  They are "duplicates" with the difference being in the type of language and so you want to make sure that Google understands the type of duplication so it knows what to do with it.

3. Are you hoping that persons in the US will be searching in farsi on Google to find your site? Are you hoping that someone in Iran will search for key terms in farsi and Google will show your farsi pages? You need to add some markup to also show what countries you are targeting with each section, depending on your keyword goals.  Just to add to what Dimitri posted, I would say a technical SEO audit with someone who understand international markup.

Generally, the .com URLs are more common and therefore seen as more "normal" to users and more likely to be trusted.

It does not mean that the .training will not work, just that it is different and that may cause other issues outside of ranking. You may remember that when .info came onto the scene, it was used by spammers so much that now it has kind of a black cloud over it.

Of course the trick with the foo-training.com domain is that you have to make sure people put in the dash. Consider issues around what if they leave the dash out, should we overtly mention the dash to remind people (think radio and TV and other ads) etc etc etc.

Overall, remember that you do not have to have "foo training" in the URL to rank well. Yes, it can help, but it really does not give you the power it used to have in the old days due to people using it as a way to spam the search engines to get keyword rich anchor text.

I would lean towards the URL with the dash (if you think it will not confuse folks to have the dash) or see about getting another .com URL and getting your keyword in the slug or in other places. I would defer to other Moz folks to see if they have had much success with the newer TLDs.

Another thing to tell your friend. If 301s did not work, the whole web would implode.

This would suck for website owners as they would never be able to change anything with a URL without losing traffic. It would suck for Google as they would have perfectly good pages that move drop out of the index for no other reason that it moved, This would mess up their search results as good quality pages would drop out of the index and this would suck for users as they would not be able to find good quality pages in Google. Google does not want to upset users as they need to users to keep using Google. If website owners were penalized in this way, they could get frustrated and stop producing good content.

It is kind of a "chicken or the egg" type of explanation, but I think you get the point.

All of that said, there can be issues with how 301s are used, 301s can be used incorrectly, this can mess up rankings, and that is true. But done right, 301s work, and they have to so that everyone (users, webmasters, Google) can benefit from a better web.

Short answer - no.   It is true that Google looks at your internal link structure to see what the most important pages are on your site. So, if you have a link in your footer that is on every single page of the site, you are telling Google that this is an important page for your site.

That is only half the equation.

If the page you link to is a sitemap (a page with a bunch of internal links on it) then this will help Google find and crawl those pages, but since the sitemap is simply a list of links, it will not rank for anything etc.

The sitemap is simply a tool to help Google find all the pages on your site. It does not mean those pages will rank well. If you want a page to rank well, you have to have good content on it and also good external links to it (to keep this point simple).

If this is an HTML sitemap that you have setup, I would suggest also setting up an XML sitemap on your site and submitting that via Google Search Console.  https://support.google.com/webmasters/answer/156184?hl=en  You can then not only help Google further crawl your site, but Search Console gives you feedback on if there are any errors in the XML sitemap and what your ratio is of pages that Google finds in the XML sitemap to how many Google has indexed.

I think I understand what you are trying to do, but I may be off with the semantics. The xml sitemap (at it's most basic level) is just a centralized list of links for pages that are on your site to help the search engines find pages without having to crawl all of the pages on your site. They do not contain any anchor text in them.

If you are wanting to crawl your site and find out what all your anchor text is internally, I would use a tool like Screaming Frog.

If you are trying to extract information from your sitemap.xml then you would need to use a combination of a text editor and excel.  Say your URL structure includes product and brand information and you have all the URLs in a sitemap.  You can pull that document into a text editor and then use the "Find and replace" function to get rid of all the extraneous code and then you can be left with just the URLs on one line.  You can then use the excel function "Text to columns" to then split out the URL into sections based on slashes or dashes.  If you have Screaming Frog, it will extract all the URLs for you and then you can split out the info in excel as well.  Use Google to find examples of how to run the tools mentioned above in detail.

If I am totally off, please give a more detailed explanation of what you are trying to do and that will help as well.

Hi Satish,

Not worth the effort. I think it will confuse folks more than anything as your home page is normally in the root directory. If you want your home page to rank for a KW, then just do everything else you should be doing to rank your home page.  Take time to update the content on your home page and build more internal and external links, much better payoff.

Good luck.

You may need to login to Google Search Console and check and see if you have a manual action of some sort.

https://www.seroundtable.com/google-rich-snippet-manual-actions-increase-21600.html

I did some digging, there are some cases where WP plugins have added so much markup it is overdone and that can cause issues.  I did not see the one you are using (Schema plugin v1.6.1) mentioned, but that does not mean it caused the issue.

Additional info

https://www.seroundtable.com/google-rich-snippets-homepage-19205.html

http://www.thesempost.com/wordpress-recipe-plugins-causing-rich-snippet-manual-actions/

Look at the keyword you are trying to rank for.  Perform several searches and see what currently ranks in the SERPs

Two things to look at

1. Are you going to look the same or different from everyone else?

If everyone who is ranking right now uses the title "Top 10 Violin Teachers of Boston" then go with something like "Hire Violin Teachers, Tutors from Boston".   Likewise, if everyone else uses "Hire Violin Teachers, Tutors from Boston", then go with something like "Top 10 Violin Teachers of Boston" You get the idea.  If you want to get clicks, you need to stand out from the rest a bit.

2. What is the user really looking for? What is the searcher intent?

Google is big on searcher intent. Forget what you and the owners think. What is the problem that potential students are trying to solve? Have your client go ask a bunch of his students / parents of students. Are they looking for "the best"? Are they looking for "the cheapest"? Are they looking for "the most experienced"? Are they looking for "most experienced with teenagers" etc.  Use that with the KW volume to drive your decision.

Here is an example.

Let's say the keywords are "violin teachers Boston". It has good search volume and goal completion rates.

You have the owner talk to clients and find they really looking for Violin instructors with the most experience.

You then see that all the pages in the serp use the listicle approach - top 10 violin teachers in Boston, the best 5 instructors for violins in Boston" etc  The results seem to indicate that people are looking for the best/experienced violin teachers in Boston, but the pages that are ranking are trying to fulfill this intent by using a list approach.

Brainstorm with ideas around the keywords "experienced violin teachers in Boston", that don't include lists and try that out. You should now have the basic keywords covered for volume/conversion/ranking purposes, you have searcher intent covered, and you have something that can help you stand out a bit on page 1 among all the other titles.  Make sense?

Cheers!

Just a correction here.  I agree with all the items above, with one very, very, very, very, very important change.

DO NOT set the corrected urls to disallow in your robots.txt

If you do not allow Google to crawl the pages, Google will not see that the links were removed, that the page is now 4xx, etc. If you were to disallow all those pages, all the clean up work that you have done will not be seen by Google and would be for naught.

If you later want to disallow those pages, that would be fine, but you need to let Google see your clean up work first.

Hello!

When you state "website health" improved, what do you mean?

• Did you check your schema with the Google Rich Snippets tester and it validated? https://search.google.com/structured-data/testing-tool
• Did you go into Google Search Console and check for errors and validate that Google is finding the snippets? https://www.google.com/webmasters/#?modal_active=none

My suspicion is that the schema plugin is maybe doing other things you did not intend it to do. Did it add a noindex tag? Did it setup strange canonicals?

Without you specifically giving a page to look at, screen shots from any of the above (or other tools) or what is the specific plugin you are using, it is otherwise very difficult to figure out.

Cheers

I know that you may not want to post this, but the URL of the site would help.

Generally speaking, Google will sometimes decide to use all kinds of alternative information about your site if Google deems that this is "better for the user".  As you may know, Google will show different title tags and descriptions in the SERPs, if it feels that is more appropriate.

That said, this is pretty major with Google changing the domain name. As it looks like your .co.uk is being chosen vs the .com it seems like Google feels like the .co.uk is the better "local" version for this audience.  Is your audience in the UK? Did you change anything in Search Console to indicate that the .co.uk site is the preferred site for a UK audience?

You need to look at any and all signals that you are giving to Google and see if any of them contradict what your 301s are showing. Here is what I would look at

1. Manually go through the 301s. Use a tool via your browser or a website that will show you what is happening in the 301. The 301s might actually be 302s (and that would explain a lot).

2. Use the site: command in Google.  See what is indexed and what is not.

3. Check the serps for the KWs that are ranking.  Click through them and see if you notice anything strange

4. Run a spider on all the .com and .co.uk URLs.  Is there some sort of strange canonical setup that you do not know about? Did you do something funky with the href langs? Are the redirects 302 or maybe taking 2 and 3 strange hops?

5. Check google search console.  You should have accounts for both domains, even though you only have 1 site active. If not, get an account setup for each domain. What is Google showing you?

6. Do you have any strange stuff in xml sitemaps?  What about robots.txt?

7. Double check with your devs that "nothing" has been done recently.  You might be surprised.

Good luck!

You need to 404 the old XML sitemaps.  Google loves to hang onto data in old sitemaps unless you 404 them.

These should help

https://www.lockedowndesign.com/prevent-google-from-crawling-old-sitemaps/

https://support.google.com/webmasters/answer/183669?hl=en

Hi everyone,

I am doing an external link audit for a site that contains a large number of public business profiles in it. As a part of each profile, there can be a website listed for the business, as well as a FB page and twitter etc.  Pretty standard.

There are thousands of business profiles on the site and I noticed that in a small group of links to FB profiles there seem to be typos.  These are not the actual businesses that I found, but I have taken common public profiles and modified to show what I am seeing to give examples.

https://www.facebook.com//nissanusa/

https://www.facebook.com//pizzahutus/

https://www.facebook.com//WhiteHouse/

https://www.facebook.com//TrekBicycle/

Notice the double slash.  You can put as many slashes as you want and it makes no difference, e.g.

https://www.facebook.com///////////////////////////////////////////////////////////TrekBicycle/

FB shows a 200 for all of pages when it should show a 404.  There is no 301 to the correct URL. Facebook also does not canonical these other pages to the one slash URL.

You could call this a potential duplicate content issue due to typos. These types of pages would be important for brand related searches for a business. Google may be smart enough to ignore them, or maybe the typo does not happen often enough that it does not really matter. I am just surprised that FB does not 404 or 301 these pages.

When I checked my personal FB page URL and some of my friends, this does not happen. FB shows a 404 if you add extra slashes to personal pages. So, the duplicate issue seems to only be with business type FB pages.

Curious about what the group thinks or if they have seen similar situations like this one.

Thanks!

Just to be clear on what Logan said. You have to allow Google to crawl your site by opening up your robots.txt to Google so it can see your noindex directive that is on each of the pages. Otherwise Google will never "see" the noindex directive on your pages.

Likewise, on sitemap.xml.  If you are not allowing Google to crawl the sitemap (because you are blocking it with robots.txt) then Google will not read the sitemap, find all your pages that have the noindex directive on them and then remove those pages from the index.

A great article is here

https://support.google.com/webmasters/answer/93710?hl=en&ref_topic=4598466

From the mouth of Google "Important! For the noindex meta tag to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see the noindex tag, and the page can still appear in search results, for example if other pages link to it."

The other point that logan makes is that Google might list your site if there are enough sites linking to it. The steps above should take care of this, as you are deindexing the page, but here is what I am thinking he is referencing

https://www.youtube.com/watch?v=KBdEwpRQRD0

Google will include a site that is blocked in robots.txt if enough pages link to it, even if they have not crawled the url.

You can go into Search Console and find all the links that they say are pointing to your site. You can also use tools like CognitiveSEO or Ahrefs, Majestic or Moz etc and gather up all of those sites to find links to your site and include those in a disavow file that you put into Search Console and tell Google to ignore all of those links to your site.

Secret bonus method.  Putting a noindex directive in your robots

https://www.deepcrawl.com/knowledge/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/

This allows you to manage your noindex directives in your robots.txt.  Makes it easier as you can control all your noindex directives from a central location and block whole folders at a time.  This would stop Google from crawling AND indexing pages all in one page and you can just leave the rest of the site alone and not worry about if a noindex tag should or should not be on a certain page.

Good luck!

I could write out how to implements rel next prev but it would be better to look at these articles

https://webmasters.googleblog.com/2011/09/pagination-with-relnext-and-relprev.html

https://moz.com/ugc/seo-guide-to-google-webmaster-recommendations-for-pagination

Hello there,

If I understand your question, it looks like you need to look fundamentally at your site structure.  If your most important pages are hub type pages this can help users easily find and navigate items wpages within the site as well as helping Google to find pages. If site structure is done properly your internal links will naturally reflect this structure that will highlight your most important pages.

For example, if you sell shirts, pants and shoes.  Your main shirt, pant and shoe pages will have many subpages with various types of shirts, pants and shoes.  As all products link up into the main pages of shirt, pants and shoes, those 3 pages will be seen with a lot of internal links.  Users will be able to see this as they navigate the site and you are giving a clear structure for Google to crawl.

Good luck!

"P.P.S. Just noticed that there is noindex x-robots-tag in headers"

That will do it.  You are telling Google to take all of your pages out of Google. You set that at the web server level and so you will need to get into your apache or nginx setup

https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

Get on this ASAP!

Howdy!

TLDR - I would estimate Google bot desktop to run at about about 980 pixels, but there is an easy way to test, just mess around with your site by adjusting the width of the browser and see if you can duplicate what you see in Google fetch and render.

http://www.w3schools.com/browsers/browsers_display.asp  97% of browsers have a width of 1024 or greater.  Therefore, if you use that minimum of 1024, your width would be appropriate for pretty much everyone. That said, you might want to go with 980 as the width to account for things like scrolling bars and the fact that most people do not browse in full screen. This is a pretty standard starting point for width.

When you use fetch and render - Google uses one of it's bots depending on the type of page https://support.google.com/webmasters/answer/6066468?hl=en

When Google talks about responsive design https://developers.google.com/webmasters/mobile-sites/mobile-seo/responsive-design it notes, "When the meta viewport element is absent, mobile browsers default to rendering the page at a desktop screen width (usually about 980px, though this varies across devices)."  In other words in some Google documentation they are giving a nod to the 980 pixels being a "standard desktop width"

Having that in mind, I would look at your site and see if you can tell if this jives.  If you have setup the page to look "normal" at greater than 980 pixels, say 1200 pixels, set your width to 1200 pixels in your browser. Then play with the width of the browser and see if you can get it to match what you see in Google fetch and render. If your site looks the same as what you see in fetch and render and your browser is at 980 pixels, then you have a confirmation of the Googlebot desktop viewport size.

You could also setup a simple page and put several images on separate rows that are 950px 980px 1000px 1200px etc wide. Run fetch and render and see what happens, but I like my first suggestion better.

Have fun!

I think you guys are missing the point.  Anything after the hashtag is ignored. As far as the crawler is concerned, all the links to page 2,3,4,5 are all the same URL - that is why the crawler does not see the other pages.

There is no issue with canonical or how it interacts with the rel next prev.  My point on the canonical was simply for illustrative purposes and looks to be implemented correctly.

Separate from the canonical the rel next prevs are setup incorrectly and that needs to be fixed once the issue with how the paginated pages are linked to using the URL with the hashtag parameters.

Several ways to go about this. Here is my take.

Buying links - "For only $50 you can get 10,000 links on sites with a PR of 3 or more". People were setting up junk sites and generating a lot of low quality links. It worked for a while, but then Google figured it out and penalized folks for it. Now the reverse has happened, people are having to pay to get links removed or for someone to help them with a disavow, also people have used negative link techniques to build bad links to competitors to hurt them (this has happened to one of my sites).

Buying followers - Similar on twitter, pay to get followers! People setup fake accounts and then sell them so that people can say they have a big number of followers.

Buying likes and shares - I have no idea who Quuu is. No idea who these people are that they use to promote/share something. For the sake of conversation, let's assume that every single person who shares content for Quuuu is legit and that I can pay to have 30,000 shares - will they buy from me? are they really relevant to my niche? On the Quuu website they list that they have followers categorized by topic, but again, unless you are a large company with a wide appeal 99% of the people that they reach will not be relevant IMHO.  Why else do you get a reach of 10k people for $10? You are going to only touch a very small percentage of that 10k who will be interested and buy from you. You can probably curate those people just as easily through your standard social media work etc.

For all the examples above, relevancy is just flushed down the toilet. Sure I have 10,000 links or followers or shares - but if none of those people are interested in buying my product, I have gained nothing but a failed ranking strategy or exposure to people who do not care about my company and my product.

EGOL - can a I have a copy of your continuity plan? I just realized that I need to update my will.

If it looks like a bad set of links, smells like a bad set of links and tastes like a bad set of links - it is probably a bad set of links. If you have control of the other sites, clean them up, otherwise this is a simple fix with a domain based disavow for the sites in question.

View source on both pages.

http://www.computerhope.com/issues/ch000746.htm

Or use the handy Moz bar to view the descriptions

Both your title and meta are exactly the same - aka they are duplicates

view-source:http://www.key.co.uk/en/key/workbenches
<title>Workbenches & Work Stations from Key</title>

view-source:http://www.key.co.uk/en/key/workbenches?page=2
<title>Workbenches & Work Stations from Key</title>

You can remedy this by simply adding "- Page #" at the end of your title and description, where # is whatever page in the pagination you are at.

The reason why the other pages in your pagination are not showing up with the duplicate issue is that you are hiding them from Google.

When I am on Page 2 and I click on the buttons for page 3,4,5 etc - here are the links that are shown

Page 3 http://www.key.co.uk/en/key/workbenches?page=2#productBeginIndex:60&orderBy:5&pageView:list&

Page 4 http://www.key.co.uk/en/key/workbenches?page=2#productBeginIndex:90&orderBy:5&pageView:list&

Page 5 http://www.key.co.uk/en/key/workbenches?page=2#productBeginIndex:120&orderBy:5&pageView:list&

These are the links that people can click on to navigate at the bottom of the page.  Everything behind the hash is ignored by Google. It is a clever way to hide parameters, but when Google looks at this it is just seeing links to the exact same page. Likewise, on that page you have a canonical link to page 2, so even if Google could see the parameters you are giving it a directive that tells Google that Page 2 is the only page that exists.

I can see that you are using rel next prev to designate Page 3 as Page 3 http://www.key.co.uk/en/key/workbenches?page=3 etc, but you are not coding the rel next prev properly by putting it up in the header with the meta tags.

In summary

1. You have duplicate title and meta tags for all your paginated pages
2. You are not linking to your paginated pages properly within the user navigation
3. You are incorrectly using rel next prev

Just to make sure I understand.  Can you clarify the sequence of the changes and for how long?  Do you know if one set of URLs has links to it or was ever indexed.

Let me explain.

It sounds like you had a site that was using http and was an asp site. So you had URLs like

http://www.website.com/file.asp  (we will call this URL type A)

You then converted to https so the URLs were like

https://www.website.com/file.asp  (we will call this URL type B)

You then updated to a PHP site so now with URLs are like this

https://www.website.com/file.php (we will call this URL type C)

You can setup 301s to go from A to B  and then another set to go from B to C.  Your question is can you setup a 301 to go from A to C, the answer is yes. You should do this.  Anytime you can reduce the number of hops the better.

What you need to think about is, well, that about the A to B and the B to C redirects?   Well, I would say at a minimum, you need to eliminate the A to B 301s as you have now decided to skip the "B" and go right to C.  That works.  What about the B to C 301 redirect?   It depends.   If you had version B of the website out for a while, and it was indexed by Google and you have links that are built to B version URLs, then yes, you need to leave the B to C redirects.  You don't want to lose any of that equity.

Likewise, let's say you have a version D of the site that comes out a year later.  You have lots of links into the C version of the site.

https://www.website.com/file.html

You then need the A urls to 301 to the D URLs (and get rid of the A to C 301s), you need the B URL to 301 to the D URLs and so on. In other words, go through another process of cleaning up the 301s and reducing the hops.

Why do all this.  Two reasons.  There will still be links to the A, B, C versions of the site.  Google will still find them and crawl them and you want to get credit for those links to your site.  Also, Google keeps an internal log of URLs and will check them from time to time, even if no one is linking to them.  You want Google to find the right URL.    In either case, if Google hits a version A URL, it would then have to go to version B via a 301 and then to version C.  It can do it, but it would rather have 1 hop.

Side note.  Try not to use global 301s, where you just 301 a bunch of pages to the home page.  That does nothing for you as far as link equity.  Try and make the 301s a 1 to 1 relationship as much as possible.

Take a look at this video and this backs up what I just said.  The number of hops is discussed at about 3 min in.  The whole video is worth watching https://www.youtube.com/watch?v=r1lVPrYoBkA

Agree with EGOL.  Even if you are giving "credit" by linking to the website.  Even if you put the legal points aside, you are just filling up the internet with copies that add no value to the internet.

Focus on the unique content you are creating.

Agree with Dave's comments.  1) Get the syntax updated on your canonical links at a minimum.  2) Yes your canonical solution will "work", but it is not best practice.  This "solution" is really a last resort.  I would try and push to move away from using canonicals this way. You optimally want 1 URL.

Just to add some color, a great / classic video on this was made by Matt Cutts.  He gives all kinds of examples where you could have duplicate URLs, i.e. www vs non www subdomain, sorting parameters added onto the URL, different file extensions,  capitalization changes, etc.  He then gives 3 options to fix them.

1. Best practice: Fix your site where you only have one URL per content item and link to it consistently (Best solution)

2. Use 301 redirects to consolidate to one URL (Next best solution)

3. Use a canonical link, if you cannot do 1 or 2. (Last resort)

Note that Matt says that they treat a canonical as a strong suggestion (it is treated similar to a 301), but they do not always have to follow it.  He repeatedly says, use the first two options, and would NOT recommend a canonical as your best or first option.

My favorite quote is at 2:24 in the video, "Developers keep SEOs in business"

What your developer may notice is that Matt does say that using a canonical link for consolidating http and https will work. No one here would say that it would not, it is just not optimal.  Sure, you can use a pair of scissors to cut your lawn, "it will work".  It doesn't mean it's the best idea.  I would think any developer worth his/her salt would want to have "clean code" and having duplicate URLs is not "clean" by SEO standards

Ok, so now you need to go back to the developer or your manager with an argument that is stronger than just, "Well, some random dude on the Moz forum said that Matt Cutt's from Google said it was preferred not to use a canonical link even though it would work".  I would never want to leave you in such a position. Here is what will/can happen over time if you stay with your current setup.

1. Report consolidation issues.  When you look at GA for traffic or OSE for links, any spidering tool for technical issues, social sharing counts, you now have split data for any given page potentially.  Sure there are ways around this, but now you have to spend all your time "fixing" reports that should not be broken to start with.   Trust me, this will come back to bite you on the bum and will cripple your efforts to show the efficacy of your SEO work.  Now who really wants that?

2. Link juice consolidation issues.  With any redirect - you lose a bit of link juice.  If you have links to both sets of URLs, any single page is not getting as much credit as it should.

3. Down the line 301 redirect bloat.  If you ever change anything and need to setup a 301 redirect, now you have to setup 2 of them and having too many 301s can negatively impact server performance.

One last thing.  If you can get the URLs consolidated into one using 301s etc.  Go with the https   That is the way that we are headed with the web and so you might as well get going in that direction.

Good luck!

I am not aware of any studies showing impact of what you call "not really keywords".  Even if there were, I would ignore them as you have to ask, "Do these words mean anything to your users?"  I work in some verticals where when you use a term like "wiki" or "faq" people think I am either talking about some type of weaved basket or I just said they were overweight, respectively.  I think it is good to start from a common lexicon, with words like "wiki" etc, but you still have to ask that initial question.

We went through this exercise about 2 years ago on a site as we wanted to setup an evergreen section for the top 40 keyword concepts.  Each concept would have a single page that would contain an original article that was at least 1,000 words long, custom videos, images, etc.

We asked a similar question.  Should we call the section (and folder in the URL) "wiki", or "encyclopedia" or "info", etc?  We did not want to call it a "wiki" as we had experts editing and writing and it was not open to the public to edit, even though for the sake of organization and layout, it looked like a wiki and provided information like one.  This is when we found out that our users were generally not as familiar with wikis and the comment with the weaved basket came out.

We did some keyword research and found that the word "help" made sense to users and also with the way Google looked at our topics when you used help in conjunction with the key terms we were trying to rank for. So we ended up using a URL that was domain.com/help/keyword-topic-of-article-slug.html  It has worked pretty well for us and the traffic has followed. We have our daily blog content and ecommerce sections in separate folder.  Makes it easy with GA and Search Console as we can look at each section easily and determine what is going on.  This was key when we had some issues with organic traffic as once we noticed overall traffic changes, we drilled down into the sections, found the issues and fixed.

TL;DR - Find out what words make sense to your users and use those, ideally they would also be keywords that you are trying to rank for. After than, just follow best practices for URL guidelines.

Great answer!  As an example, we have had issues on a site with negative link building from competitors and update 1x to 2x a year.

First rule of hiring an SEO firm, don't hire a firm that reaches out to you randomly and tells you that you have something broken and they can fix it.  FYI, I also know a prince in Nigeria who needs access to your bank account and would be happy to pay you for it.

That said, the whole pages on site vs pages ranking in Google is a valid measure.  If most of the pages of your site are not ranked in Google, then that may be an indicator that you have a lot of low quality pages and this could be an issue.  This type of metric is a good one for check in the health of your site.

That said, your have to figure out if this matters to your site or not and it is pretty easy to do. Various ways to look at this.

Log into Search Console and if you have a sitemap submitted on your main dashboard for sitemaps you will see URLs submitted vs URLs indexed. At a basic level, how many pages did you show to Google and then how many pages did Google say, yes, these are good enough to be indexed.

Use ScreamingFrog or the MozCrawl, see how many URLs are found by crawling your website.  How does this total compare to what you show in your sitemap?  Do you have a bunch of pages that are just tag cloud pages or a page that is just a resorted or printer friendly version (aka duplicate) of another page?  You probably need to block access to those (and make sure they are not in your sitemap).

ScreamingFrog has a API to connect to your GA data.  You can then also pull what type of search traffic do these pages get.  This gets back to your ranking question.  If you have a page that Google is ranking for something, it should get some search traffic to it.  Depending on how much traffic you get overall to your site, you may need to look at a 3 to 6 month time period.  You can now say, out of all of the URLs on your site, if there is a group of URLs that is not getting any traffic from Google for the past 6 months, they are probably not ranking for any key words.  Why is that?  Look at those pages.  Are they crappy pages with very little or duplicate content?  Updating a meta tag will not do anything if you have a crappy page to start with.  Maybe you need to noindex them or 404 them and get them off your site.  Maybe you need to add content, write more, include some graphics etc.   Maybe you have duplicate or similar pages you need to combine into one super page and redirect the other page to it.

Good luck!

Just as a follow-up to Tom's great post.  If you were wanting to test a robots.txt setup, especially if you were using a wildcard or using an allow combined with a disallow, Google Search Console under the Crawl section has a robots.txt Tester.  You will see your most recent robots.txt file there that Google has a copy of.  You can then modify that version and then enter a URL at the bottom to see if everything is set correctly or not.  It is pretty handy, especially if you have a big robots.txt file.   Note that this tool does not change how Google crawls your site or your robots.txt file, it is just for testing.  Once you find the configuration that works, you would still need to update the robots.txt on your server.

It is my understanding it checks its own version and it does this weekly (Moz staff please chime in to correct and/or clarify further).  You can run a page report "on the fly" using Moz as well.  Update your page then run the grader to see the changes.

I think the key point there is not the schema markup per se, but the fact that the reviews can be original content on that and that can help the page rank.  That is the angle you want to start with with the Schema being the icing on the top.   Improving the thin pages is the key first step.  Lots of stuff out there on how Google does not like thin/duplicate content.

Markup for reviews will not help you rank for SEO directly.   That said, if the page already ranks, then Schema markup for reviews that then shows in a rich snippet helps with your CTR on that position (and this helps solidify your position as Google sees the action on your result).

I cant find the reference, but wanted to mention, there was a recent article showing that you actually get a higher CTR in position #2 with some sort of rich snippet vs something in #1 position without a rich snippet.    See if you can find it (or maybe someone else here can) for some reason my Google skills are failing me right now and I cannot find it. I must need more coffee. :-).

I cant release the actual data for client privacy reasons, but I can say from my own experience, we had a bunch of our pages on page 1 (usually top 3) for various local searches with review data marked up and the snippets would show for the reviews.  There was a change in how the schema validated and we saw a ~6% drop in traffic to those pages when the snippets on the search results went away - this was with no significant ranking changes.  We corrected the markup, Google crawled the site like crazy within 24 hours and we had our snippets and traffic back.  It was pretty amazing to watch and validated everything I had seen others talk about.

Separately, I just got back from PubCon.  Ton of people talking about using markup, most notably Duane Forrester from Bing. He spent half his keynote talking about advances in machine learning and the other half talking about how we all needed to use Schema.org.  In other words, computers are getting smarter and smarter, but if you can, please help them out with markup.

Hope this helps you figure things out.

The turn around is hard to predict.  I would ask as many Moz forum members to do what you and I did from a desktop and mobile device.

Looks like Google is pulling from a US based Yelp profile for a company with the same name.  Strange considering this is Google.com.au

http://www.google.com.au/imgres?imgurl=http://s3-media1.fl.yelpcdn.com/bphoto/zNjdBcEiNvc0YuXfuJvOJw/ls.jpg&imgrefurl=http://www.yelp.com/biz/community-first-credit-union-jacksonville-8&h=250&w=250&tbnid=ctUt5usQbxTaNM:&q=cfcu&tbnh=151&tbnw=151&usg=__mX0EGbrF8xnXjSeNBMRlSGiUn9U=&docid=Tww1toT0IL7f8M&itg=1

I checked your logo markup on your site http://www.communityfirst.com.au/Assets/1333/CFCU_logo_cent_RGB_square.png and it checks out ok in the structured data testing tool https://developers.google.com/structured-data/testing-tool/

Here is what you need to do.  If you look at your search result, right under the logo and Wikipedia snippet there is a link labeled "Feedback".  Click that and follow the instructions to let Google know about the error.

Remember these are all computers making assumptions and nobody is perfect.

Invisible to who?  If they are invisible to the search engines (e.g. blocked via robots.txt) they could not be harmful, but if you gain any link or other quality equity through those pages, your will not get any additional domain authority or organic traffic to them.

That said, if the hidden categories were resorts of the main categories on your site then you would want to hide them from the search engine as they would be seen as low quality duplicate content.  Otherwise if they were visible they could negatively impact your site.

If Hubspot is preventing the redirects then that is your issue.  Google ranked (and sends traffic to) the Hubspot pages.  Unless you can setup the 301s from the old site to the new one, your new site will just look like duplicates of the old one.

Another option to see if you cannot get the 301s to work is to setup canonical links from the old content to the new.  Google most often treats this like a 301.    This could be a good interim step before getting 301s setup.   If Google updates with the new URLs in the SERPs and if you can get most of any external links pointing from the old blog to the new blog, you could then simply shut down the old block.  The canonical link would have updated the SERPs and the external links to the old blog have been updated to the new one.  Therefore, your old blog posts would not have any value to you (as they are not ranked in Google and have little or no links to them) and it would be ok to terminate.

Good luck!

Agree with Martijn.  Technically, Google does not want any pages in the sitemap that 4xx or 3xx or have any other result than a 200.

I would say this, if your site is being rebuilt, having a sitemap that is accurate and that updates when you update the site is a basic requirement.   The fact that there is a high cost for fixing this issue is baloney. It sounds like the devs did not build the site correctly the first time if they do not have a way to update the sitemap automatically.

You could generate the sitemap yourself

You can use tools like

http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/

Or read tutorials on how to use Screaming Frog to create a sitemap

http://www.hmtweb.com/marketing-blog/dirty-sitemaps-how-to-download-crawl/

Frankly, the annual cost of Screaming Frog (about $150 a year) gets you so much more than just sitemaps.  Buy Screaming Frog, have it generate your sitemap and ask the devs to upload it.  If you have a site with several thousand pages, just running Screaming Frog monthly would help you find issues on your site that is well worth the cost.    Search "Screaming Frog" here in the forums and you can see that this is one of the "swiss army knives" of technical SEO.