Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
I actually have come across a handful URLs that are NoIndex, I'll DM you a list once complete.
I can't be certain this is the root of the problem (I've never seen this error in the crawl report), but based on the error you said you're getting, I believe it's a great starting point.
Hi John,
For simplicity sake, I'll guide you over to a very similar conversation I had yesterday. https://moz.com/community/q/consistant-citations-for-local-seo
Hope that thread helps!
Hi Rhys,
The page remove tool in GSC is for temporary removal, but since the pages are actually gone, it's likely they won't come back. I actually just did this a few days ago for the same reason.
Blocking the pages with robots.txt will not remove them from the index. Robots.txt disallows only prevent crawling, not indexing.
Hi Jarred,
Issues like this are usually connected to hrefang tag. I noticed you've got those in place, but the country element is missing. Try updating your American (.com) and French (.fr) hreflang tags to the following:
The additional specificity might be what you need to override that mistake by Google and get your meta descriptions back to normal. Hopefully this works, as I didn't see anything else on either TLD that would cause this.
Hi,
On this new page, you mostly talk about the other 2 types of oak beams, so there's really not enough differentiation in this content. At this point, you're kind of working up hill against yourself.
The better way to tackle this would have been to generate content for your broadest keyword first, show Google that your site is about oak beams. Then write content for specific types.
Hi Ryan,
You should use a 301 redirect rule to send HTTP to HTTPS URLs.
Do you have different profiles set up in Search Console for secure and non-secure versions? If not, you should do that as well. This will help you properly analyze data points and monitor indexing of both versions. You want to see HTTP indexed URLs approach (and eventually reach) zero, and your HTTPS indexed URLs should increase as Google discovers them via 301 redirects.
Hi,
Regarding the first set of URLs: I took a look at a handful of those URLs, and it's entirely possible that you're getting duplicate notices on those. Rogerbot flags any 2 pages as duplicates if the source code of those URLs matches at 90% or more. So it's not identical, but not different enough that search engines can discern. Most of the products you've listed there have no content, or a very small amount, meaning that when you consider the rest of the code involved with that page, it mostly matches the homepage.
Regarding the second set: I ran those URLs through Screaming Frog and don't see any canonical tags. Keep in mind, just because URLs aren't indexed in search engines, doesn't mean Rogerbot doesn't have access to them.
*Update - on further digging, I think I found the source of all of your duplicate issues. Both www and non-www versions of your URLs are accessible. One of them should redirect to the other, doesn't matter which, but both should not render.
This isn't going to count as duplicate content. Duplicate content is more of a concern if someone is stealing entire pieces of content from your site and posting it on theirs.
What's happening here is actually a good thing, you're getting extra visibility and people are referencing your phone number from their site. This is great! Don't do anything to deter additional phone calls from prospective customers!
Will,
I'm not familiar with the CMS you're using, but to answer your question about rel=canonical, no, that is not an instance of when to use that tag. Canonical tags are used for times when duplicate content is unavoidable, such as sorting a product category page and having different URL parameters based on the sort type.
Heyo!
I would not recommend your homepage exists at any URL other than example.com for the following reasons:
I'm sure other could drum up more reasons, but the ones I've listed here should be enough to dissuade you.
You can typically fare much better by giving non-branded keywords interior pages that are specific to that topic rather than the broader homepage content. This will increase the likelihood of people finding what they're looking for and is a better way to tailor your content to your audience and to algorithms.
Hey Tiffiny,
Typically your homepage is going to have much stronger PA than any other page, which means more pull on search engine algorithms. If your site started out selling baby headbands, chances are that was (and may still primarily be) the focus of content on your home page.Shifting algorithm focus to another page can be a lengthy and difficult process!
You've probably also got some links pointing to your home page with 'baby headband' anchor text - or variations on that - which is also contributing to your home page out-ranking your headband-specific page. As a whole, regardless of anchor text, you probably also have more links pointing to the home page.
I wouldn't worry too much about it though, if you've got 2 organic results for that keyword, you're even more likely to get the click!
That is correct, you should be using rel=next/prev for markup on paginated sections. But after noticing the www and non-www issue, I don't think your problem is related to canonicals or prev/next.
Regardless of what you're doing with canonical tags or prev/next, you pages should never be accessible at both www and non-www versions. You're going to be at a duplicate content risk as long as both versions exist.
Matthew,
For branded search, this box is generated using the information from your Google My Business listing. Do you have that listing claimed, verified, and accurate?
Hi,
Moving website A to a sub-domain of website B is going to be counterproductive, in that search engines hold different metrics for sub-domains than they do for their bigger brother, the primary domain. It's near impossible to advise what to do without any knowledge of the client and their division. But strictly from an SEO perspective, you stand to gain more by relocating website A to website B, but as a sub-directory rather than sub-domain - so long as the migration follows best-practices for such a task.
As with most things in SEO, it depends. Depends on the search volume of the keyword, how many sites you're competing with, how many of those sites are well optimized, how many links your page/site have, how authoritative/trustworthy your site is, and so on. There are a lot of variables in the mix here, so it's not feasible to provide a numerical answer.
Sorry for the confusion... I just meant that since that's your primary product, and at one point your only product, the page still has a strong focus on that topic. You need that baby headbands category page, and of course, you don't want to wipe all focus from your home page, so there's not a whole lot you can do here.
Some things you might try:
Google is less concerned about the actual structure of your URLs, and more concerned that you pick a horse and ride it, which you've done by canonicalizing 2 variations to the third. In your example, the third URL is perfectly fine, since it will always remain constant. The other 2 can change depending on how someone navigates to that product. I'd keep it the way you have it.
Hi,
There's a couple things you'll need to do. First is to set up a Search Console account for each TLD version of your domain. Once you've setup and verified, you'll want to specify the target country for each. To do this, go to Search Traffic on the left nav, then click International Targeting, then head to the Country tab. Set each for its respective domain.
Next, you'll need to implement hreflang tags. They work similar to canonical tags, but instead of telling search engines the preferential version of the content, these tags point them to the other country/language versions. Everything you need to know about this tag can be found here: https://moz.com/learn/seo/hreflang-tag.
These two actions will help the country-specific versions of Google serve up the appropriate versions of your content and avoid getting dinged for dupes.
Hope that's helpful!
Hi,
Here are a few steps you should take to ensure you hire a legitimate, white-hat link building agency:
Good luck in your search!
You're welcome, have an awesome day!
That's correct. I started doing this as a way to determine which 301s can be removed over time, but I come across new uses for it all the time!
I looked at your source code and searched for product names and couldn't find any. This tell me that your products are being pulled in with JavaScript, which I confirmed by disabling/enabling JS, products don't show with it disabled. Since JS is populating your product lists, and that's the only real difference on each page, Moz is flagging it as duplicate because the source code is more than 90% the same. In order to fix this, you'll have to either pull in products a different way, or figure out a way to create unique content for each page that isn't rendered via JS.
You can leverage that phone number by populating a field on the form based on what phone number displays.
For example, you may have the following dynamic phone configuration:
555-1234: Google Organic
555-3456: Google Paid Search
555-5678: Email Marketing
When someone submits a form, there is a hidden field that looks at what phone number is displayed and then populates the field with the coordinating marketing channel. As far as execution of this logic, I'm afraid I can't help, our Dev team handled all that. But it's pretty simple logic, so I think most developers would be able to write this relatively easily.
Hi,
These pages are essentially nothing but a quote request form, I'm wondering if they see that it's a form and therefore have flagged it as an important page to put in the six-pack. These TX and NY pages also appear to be almost exact duplicates of your Get A Free Quote page, if there's not a business case for having multiples of the same page, I'd say get rid them. Your best bet is to demote them in Search Console, maybe after demoting them a few times Google will get the point.
Hi,
I wouldn't recommend the noindex approach here. Three of your four URLs are parameter'd versions of the homepage; meta tags are applied at the page level, not by parameters. This means your noindex tag would apply to the homepage - no good!! You could put a nofollow attribute on the links that point to those parameter URLs, but don't expect it to have a huge impact on your SEO efforts. Your /members/ folder might be worth de-indexing via a noindex tag, depends on what is contained in that folder. It sounds like it's probably account-level pages, if that's the case, you can and should keep those pages out of the index.
Hope that's helpful!
Hi,
It sounds like you might need to implement hreflang tags. Have you done so already? Without being able to specify each country-level shop in Search Console via ccTLD, this is your best bet to ensure the right pages show in the right versions of Google. Here's Moz's guide to using hreflang tags. For each of your country-specific subdirectories, you'll need to add the appropriate tag which will specify the country and language of the page. This helps keep Swiss pages out of Google.de, etc, etc.
Hi,
Google will count this as duplicate content, regardless of 'affiliate' status or not. You've got a few options here:
I think option 3 is the most viable, since you won't have to ask the affiliates to make any operational changes.
Hi James,
My choice: Love Live Chat Pop Ups
Reason: Compare your conversion rates for users that chat and users that don't. We have one eComm client who saw 3-4 times the conversion rate for users whose session included a live chat vs. those who didn't chat.
It doesn't have to be all or nothing either. You say you're recommending disabling it and that it automatically pops up 1.66 times per session - you can curb that a bit and only show it based on site engagement. For example, you could show it only after a user has been on a product detail page for X amount of seconds. Or after Z amount of products viewed.
Hi,
I've not seen any comment from Googlers regarding this debate. I realize I'm keeping this in the Moz-sphere, which isn't quite what you're looking for, but this quote is from Moz's domain setup guide:
"Since search engines keep different metrics for domains than they do subdomains, it is recommended that webmasters place link-worthy content like blogs in subfolders rather than subdomains. (i.e. www.example.com/blog/ rather than blog.example.com) The notable exceptions to this are language-specific websites. (i.e., en.example.com for the English version of the website)."
I think that quote is pretty compelling towards the subdirectory side of this quandry. I also recommend checking out the comments on the Whiteboard Friday link you posted, there is plenty of evidence there as well.
Unfortunately, this debate will probably go on forever until we get definitive word from Google.
Yea, I wouldn't worry too much about those. You should however, specify the purpose of those parameters in Google Search Console - it'll help Google better understand their function. Here's a good guide on how to do that: http://www.shoutmeloud.com/google-webmaster-tool-added-url-parameter-option-seo.html
Hi Ramendra,
Based on what you said, it sounds like both versions of your site exist and are indexed, and you want to mitigate your duplicate content risk. If that's accurate, here are my recommendations on this:
Hi,
In a perfect world, yes, all your links would be pointing to the most accurate version of your URLs possible. But Google has also recently stated that they're indexing HTTPS by default whenever possible. If you've got some that would be easy to update, go for it, but I wouldn't sink too much time into it as I don't believe you'll see too much of an impact from doing so.
Luke,
This technically wouldn't cause any ill-effects on you SEO efforts since your URLs aren't changing. However, according to our good friend, Mr. Cutts, auto translation via Google Translate isn't recommended and can be seen as spamming. https://www.youtube.com/watch?feature=player_embedded&v=UDg2AGRGjLQ
Nick,
I've inherited a number of sites with this same problem, some of them quite large sites with rampant singe-keyword landing pages that required a lot of cleanup. In my opinion, in the long run it's definitely worth the effort to prune where you can and get same/similar content combined into one higher quality page. I've had pretty good success in combining pages into topical focused LPs.
There's been a lot of talk in the SEO world over the last year about "topics, not keywords", and this discussion is exactly where that idea comes into play. With Google's increasing effort at better understanding keyword intent, they've gotten really good at determining synonymous phrases. As a result, sites with single-keyword LPs are suffering (or will eventually) due to content cannibalization. Let's say you have a LP for "blue widgets" and another for "widgets that are blue", eventually those 2 pages are going to compete with each other and Google will choose to rank neither very well.
If you do go down the road to combining pages, here's a few tips:
Hope that helps, happy optimizing!
Heyo!
That redirect is most likely causing your problems. If search engines can't access the homepage of that subdomain, it's going to be difficult for them to find other URLs on that subdomain.
Search engines are also going to have a hard time finding the robots.txt and XML sitemap if the homepage of the subdomain is redirecting. Turn that redirect off and make sure you've got a good robots and sitemap file in place and you should be good to go. You'll also want to make sure Google knows about the sitemap by submitting it in Search Console.
I'm going to have to disagree with 411. According to Google's Search Raters Guidelines document that was released last year, they're taking into account your site/company's overall reputation. This means they're not only looking at Google Reviews, but third party review/reputation sites as well. It's part of their initiative to improve the quality of results. It doesn't matter how many links you have or how good your site structure and on-page SEO is if the service or product you provide is terrible and gets bad reviews everywhere you look.
For a summary of what's in these guidelines, check out this summary: https://moz.com/blog/google-search-quality-raters-guidelines. The whole guidelines document is worth reading, but be sure to check out the Reputation section and the 5 clues of what makes a page low quality in this post.
For future reference, if you've got the Moz bar installed in your browser, all you need to do is click the pencil/marker icon and select the "Nofollowed (pink)" option. All nofollow links on that page will be highlighted.
Hmm...only other thing I can think of is your that XML sitemap may contain these additional URL strings, but I assume you've already got clean URLs there.
Hi Rena,
Issues like this pop up frequently in Moz QA. Here's what's happening - the only difference on these pages is the name of the product, the meta data, and the URL. This being the case, the majority of the source code for each page is the same, when you take into account all the code for the template (header, footer, etc). Moz sets a threshold of 90% for source code match, any 2 pages that have at least 90% same source code get flagged as duplicate.
So what can you do about it? It looks like you're using these pages as product detail pages like you'd see on an ecommerce site. Product detail pages typically have descriptions providing details about the particular product. I'd recommend writing some descriptive content for each of these pages. This will increase the uniqueness of the pages and should differentiate them enough to not trigger the dupe content warning. Descriptions on these products will also help your users' experience on the site, so you're not doing this just to appease the bots.
Hope that helps clear things up!
Hi Rena,
You technically can do that, but it's not recommended - for the exact reason you state above. More often than not, 2 sites aren't going to have the same set of disallow rules.
Additionally, you should also be using robots.txt files to direct search engines to your XML sitemap, and if you're sharing a robots file, then you can't specify 2 different sitemaps on 2 different domains.
Do you have reviews on your homepage? On most sites, reviews are relative to specific products or services rather than the whole company. You would need to have an aggregate review score displayed on your homepage in order to be able to mark it up with schema. Schema is meant to identify data types that are visible on the site, it's not like meta data where it runs in the background for search engines to see and not people.
A site I manage has all their blog posts scraped and republished on a trash website. We've sent requests to the site a handful of times, and submitted a DMCA takedown request, but nothing has ever been resolved.
Your best bet is to disavow those domains scraping the original content. I'd also recommend more investigation into the sites publishing content and links that point to your site. It seems some sites are more prone to being scraped than others, haven't figured out any rhyme or reason to when it happens yet though. Definitely interested to see if anyone else chimes in with some insights on preventative methods.
I believe you could be at risk of duplicate content issues. If it were my client, I'd definitely consider this a code-red issue and attack it from all possible angles.
James,
They do have that threshold, confirmed by Tawny's comment on this thread from a while back.
Hi,
After reading your post, I'm wondering why the old domain hasn't already been 301 redirected to the new site? That would really be the ideal situation for not having your two sites compete against each other. You'd want to do this at the page level, so equivalent pages on the old site get redirected to the most relevant page on the new site.
For example: www.oldsite.com/calculator redirects to www.newsite.com/calculator
Hi,
As with most things in SEO, the answer depends. It depends on how many UNIQUE topics you can generate and how long you're willing to spread those out over. Unique being the keyword there.Don't just go out and write X amount of posts per week because someone recommended it or your read it somewhere - that's going to lead to crap or topically-identical content just so you can fulfill a quota.
Make a list of topics for posts, as many as you can that are unique and worth having their own URL. If you start to get too many posts that are basically the same, Google won't rank either as they'll view it as a spam tactic. Once you have that list, prioritize it in terms of average search volume for the keywords you've associated with each post. Then start with 1 or 2 posts per week. If you see good results from that, consider ramping it up. If you're not seeing good results, step back and shift your strategy.
As Andy said, there's no right answer to this question, but there are smart ways to plan your approach. Just remember to keep your posts unique (yes, I am saying that a lot, because it's important!), the best way to rank is to show Google something new that they haven't seen before.
Happy blogging!
You should definitely not let those URLs die. Having a ton of 404 errors is a sure fire way to have Google demote your site for poor quality. I recommend selecting your top 100 or so products, based on Google Analytics traffic numbers and importance to your business, and apply one-to-one 301 redirects for those. The rest of the old products can be redirected to the homepage, or maybe the main store page.
Isaac,
This is the same issue that I mentioned a fix for on this thread: https://moz.com/community/q/should-i-be-worried-about-our-duplicate-content
If you canonicalize product A to product B, product A will eventually drop out of search engines' index, not what you want to happen. Read through the recommendations I made on the other thread.