Welcome to the Q&A Forum

Hi, even if you use robots.txt file to block these pages, Google can still pick the references of these pages from third-party websites and can crawl from there. Such pages will not have the description snippet in the search results and instead will show text that reads:

A description of this result is not available because of this site's robots.txt.

So, to fully stop Google from crawling these pages, you can go in for the page-level meta robots tag along with the robots.txt method. The page-level robots meta tag complements robots.txt method.By the way, robots.txt file can definitely save you some crawl budget. I don't think you should be thinking much about crawl budget though, as long as your website is super-easy to crawl with simple text-based internal links and stuff like, super-fast servers etc.,

Those my my two cents my friend.

Best regards,

Devanur Rafi

Hi,

while the generation of sitelinks is fully automated as of now, I tried to look at the global search volumes for the terms,

stackstreet and stack street using Google keyword planner. It showed that the term, 'stack street' has an avg monthly search volume of 30 while the term, 'stackstreet' has none. Even the Google suggest feature does not show anything while trying to type, stackstreet in the search bar but it does suggest when you try to type, stack street.

Though it may sound weird, you can do as follows:

For 3-4 weeks, ask all your staff, family and friends to type, 'stackstreet' in Google and click on your homepage from the search results. You can ask them to do this few times a day. This will make Google learn about and recognize the search term stackstreet and possibly start showing the sitelinks for it just like it does for, 'stack street'.

Good Luck.

Best regards,

Devanur Rafi

Hi, any plugin like sh404SEF will work and accomplish your goal without hurting your rankings as long as it redirects, the index.php URLs to their corresponding without index.php URLs via 301. By the way, you don't need to list all your URLs in .htaccess file to implement this. You can go with pattern match redirection.

Here you go for more:

http://www.askapache.com/htaccess/301-redirect-with-mod_rewrite-or-redirectmatch.html

and

http://www.searchenginepeople.com/blog/htaccess-redirect-rewrite-rules.html

By the way, having index.php in URLs does not affect your SEO efforts directly but by stripping index.php from all the URLs will make them look pretty, clean and a bit user friendly.

Hope it helps. Good Luck to you.

Best regards,

Devanur Rafi

<colgroup><col width="182"></colgroup>
|   |

Hi, the suggestion was not to get the quality articles written that take an hour to write each but I meant to change the products descriptions that were copied and pasted with little variation so that they don't look like a copy, paste job.

Now, coming to the de-indexing part, let us look at a scenario:

Suppose I built a website to promote Amazon products through Amazon associates program. I populated its pages using Amazon API through a plugin like WProbot or Protozon. In this case, the content will be purely scraped from Amazon and other places. After a while, I realize that my site has not been performing well in the search engines because of the scraped content but haven't seen any penalty levied or manual action taken. As of now, I have about 3000 pages in Google's index. Now I want to tackle the duplicate content issue. This is what I would do to be on a safer side from a possible penalty in future like Panda:

1. First, will make the top pages unique.

2. Add, noindex to the rest of the duplicate content pages.

3. Keep on making the pages unique in phases, removing the noindex tag to the ones that were updated with unique content.

4. Would repeat the above step till I fix all the duplicate content pages on the website.

It greatly depends on the level of content duplication and few other things so, we will be able to suggest better if we can have a look at the website in question. You can send a private message if you want any of us to have a look at it.

Very true my friend. If you look at your top pages for last 30 days, there won't be more than 2000 approximately. So you can make the content unique on these over a period of six months or a bit more going at 300 per month. Trust me, this would be an effort well spent.

Hi,

While its not recommended to have duplicate content on your pages that is found else where, it is also not a good thing to de-index pages from Google. If I were you, I would have tried to beef-up these duplicate pages a little bit with unique content or at least rewritten the existing content so that it becomes unique.

Please go ahead and initiate the task of rewriting the product descriptions in phases starting with the ones that get the most traffic as per your web analytics data. Those were my two cents my friend.

Best regards,

Devanur Rafi

Hi,

You need to go in for server-side URL rewrites in this case. With URL rewrite in place, the URL in the address bar for the visitor will be your SEO and user-friendly, vanity URL and the same will be in search engines' indices there by accomplishing your mission. Then you might want to make sure that your old, ugly URL passes on all its SEO goodies to the vanity URL. For this, you can go in for a 301 redirection from the ugly URL to the vanity URL.

Here is a beautiful article from Moz: http://moz.com/blog/url-rewrites-and-301-redirects-how-does-it-all-work

Hope it helps. Good Luck.

Best regards,

Devanur Rafi

Hi Jane, thanks for stopping by and giving your inputs. You are right, no SEO tool is a one shop stop for carrying out an SEO audit. We use a myriad of tools to analyze the link profile, on-page elements with lot of manual scrutiny, Google penalty checks from GA and GWMT account perspective. The major tools that we use include but not limited to, Moz, MajesticSEO, ahrefs, SEMrush, Brightedge and Maven. Though there are many areas and features that these tools superimpose on each other, a consolidated audit data from these tools gives us a holistic picture of the issue at hand with lot of manual checks that go in to the process.

Recently, we saw a situation where page loading time was an issue with all the other stuff being superior to the competition. We use a set of tools we use for page loading performance test looking at Site speed data from GA. Sometimes, the historical indexing data can also have an impact on present ranking especially when the same website operated in a different niche (or even probably operated in an illegal niche) and now serving different content with no change in ownership (Whois data). In cases this these, we look at way back machine data and try to analyze things.The list of audit checkpoints goes on and on... :))

We try to leave no stone unturned when it comes to doing an SEO audit and as you know SEO forensics is a very time consuming tiresome process especially when we have a client with better SEO parameters over the competition and crying foul about Google.

Its a daily battle and the best thing about SEO is, we love it and Google is both our strength and weakness ; ))

Best regards,

Devanur Rafi

Hi, just in case, login to your Google analytics account and please visit this URL:

http://www.barracuda-digital.co.uk/panguin-tool/

Give read-only access to the application and check if this website has been hit by any of the Google updates like Panda or Penguin. Look from October 2012 till date. I see an issue between December 2012 to February 2013. Let us know what do you see and we shall take it from there.

By the way, no on-page SEO tool on this planet is comprehensive. A true SEO audit can be done by only a good SEO expert. Of course making use of best tools of the trade like Moz etc.,

Best regards,

Devanur

Forgot to include the disclaimer : ) ..With all things mentioned in that article from Google webmaster support, your chances to rank in .co.uk SERPs definitely become higher while there no guarantee though.

Hi Ben,

If coming up with a dedicated CCTLD (yourdomain.co.uk) is not possible in this case, you can go with a CC sub-domain like.. uk.yourdomain.com for the purpose. Personally, except for geo-targeting in a multi-regional setup, I stick to and prefer sub-directories to sub-domains.

Here you go for what Google has to say regarding this issue of yours:

https://support.google.com/webmasters/answer/182192?hl=en

If you look at the table, the sub-domain strategy has only one con compared to others.

Hope it helps. Good Luck my friend.

Best regards,

Devanur Rafi

Hi Somanathan,

Suppose:

Website = Car

Content  = Engine

DA/PA   = Turbocharger/Nitro booster

You need to have content that is unique, highly relevant, targeted, updated and useful to the visitors. With this in place, your website will get the boost from DA/PA and thereby can rank well in the search engines. You should do a thorough keyword research and analysis, come up with highly searched terms for your niche, add pages targeting each of these with highly relevant content targeting the main keyword/phrase. If you can let us know the website in question, we will be able to suggest you better. Good Luck.

Best regards,

Devanur Rafi

Hi,

Having multiple pages with similar or identical content confuses the search engines and the outcomes in SERPs will be undesirable. Here is the deal: No two unique URLs should serve substantially similar or identical content. If it is the case, you should decide the URL that you would like to rank in the search engines, make others point to it via rel=canonical attribute. In general, the page that targets the most search keyword/phrase can be made the canonical URL or the preferred URL.

If I were you, I would have added unique content to the existing pages targeting the main keyword for the page.

For example, if the page talks about, 'used cars', this would become my target term for the page:

abc.com/cars/used-cars.html

I would also go ahead with a thorough keyword research & analysis to find which keywords/phrases are being searched more in your geo-location or target market, add corresponding pages with highly targeted content for each of these keywords/phrases (if not added already).

The key here is, content that is unique, up-to-date, highly relevant and useful to the visitors. Such content would bring in dramatic improvements to your overall SEO ROI and search engines like Google love such content and these pages will be awarded good positions in the SERPs going forward. As you know, high quality content is a natural link magnet.

Here is an action plan, if I were you:

1. Make these pages unique by adding unique content

2. Do a thorough keyword research & analysis to find new content opportunities in your niche

3. Add new pages with unique content based on the outcome of step 2.

4. Update the Sitemap.xml file and submit it to webmaster tools

5. Repeat steps from 2 to 4 once in 6 months based on the results or as and when required.

Those were my two cents my friend. Good Luck.

Best regards,

Devanur Rafi

Hi Sam,

Here are my two cents..if I feel my visitors will benefit from the page that I linked to, I don't mind following the link. For example, when I link to a page on Wikipedia, I won't add the nofollow attribute. This is a good gesture from a webmaster. Coming to the affiliate links (no matter how important are they to my visitors) as most of the time, my intention is to make some extra money in the form of commissions, I would definitely have the links nofollowed.

I want to be very, very cautious while linking out to external web properties and even more cautious and conservative when it comes to sharing my link juice with other domains. Unless the other domains are very authoritative and extremely useful, I go with nofollow.

Best regards,

Devanur Rafi

Oh yes, the short answer would be, No as others posted here.

Here is an interesting discussion about this topic from Moz Q&A forum:

http://moz.com/community/q/how-google-adwords-can-impact-seo-ranking

And an interesting post on YouMoz blog:

http://moz.com/ugc/8-ways-ppc-and-seo-are-better-when-used-together

Hi, you can use Google Webmaster Tools account and see the total number of internal links for your website as follows:

1. On the Webmaster Tools Home page, click the site you want.
2. On the left-hand menu, click **Search **Traffic, and then click Internal Links.

While this can depend on the total number of pages from your website that are discovered by Google, this is the best way to achieve what you want.

The other tool that can show you internal links pointing to any page of your website, you can use Screamaing frog SEO Spider. This tool also gives you total number of out-bound links from a page of your website pointing to other internal pages of the same website as well as to the external websites. This tool is an indispensable weapon in any SEO's arsenal.

Hope it helps.

Best regards,

Devanur Rafi

Lesley is absolutely correct. I would never want to remove my pages from Google there by reducing the number of indexed pages (as the website has only about 600 pages), but would beef them up with unique and sizeable content of at least around 500 words each.

Hi,

You can look at your Google Webmaster Tools account to know what are the search queries that your visitors used to land on your website from Google. You can also look at your web analytics data (like Google Analytics, SiteCatalyst etc) to know this data but as Google started encrypting the search, you won't be able to see the complete picture of your keyword data that is drving traffic from Google. If you don't have an access to webmaster tools account or web analytics, you can try using tools like SEMrush to have a fair idea about the keywords that are driving organic traffic to a website.

Coming to the strategies, useful, relevant, updated content stands at the top followed by backlinks coming from other topical niche sites, good blogs and forums, authority websites like .gov, .edu, .org so on and so forth. If you can come up with top notch content, half the battle is won as good content is a natural link magnet. Those were my two cents my friend. Good Luck and please post back in case you have any other queries in this regard.

Best regards,

Devanur Rafi

Hi,

Pages blocked by robots.txt file will not be considered for ranking. You still see those pages in Google's index is because of their mentions on other websites. Google has been doing link discovery from websites other than these Web2.0 sites of yours. So to stop Google completely from indexing these pages, you should go in for page-levelrobots meta tag. Here you go for more: https://support.google.com/webmasters/answer/93710?hl=en

Once you implement the page-level robots meta tag to block those pages, Google will remove them from the index after the next crawl. You can expedite this process by using the link removal procedure in webmaster tools account.

Here you go for more: https://support.google.com/websearch/troubleshooter/3111061?rd=1

Good Luck to you my friend.

Best regards,

Devanur Rafi

Hi,

While no single backlink checker on this planet would show you same amount of backlinks that a website has, the only better way is to consolidate data from different tools like, ahrefs, OSE, Google webmaster tools account etc. No single tool can give you comprehensive backlink data.

Hope it helps my friend. Good Luck to you.

Best regards,

Devanur Rafi

Hi,

I agree with Dennis here.

One question that you can ask yourself:

"Why should Google have two websites with substantially similar content in their index?"

So, the answer would be to come up with substantially unique content on both the websites so that both of them can pass Google's radar safely  and justify their presence in their index.

In the past, I worked with a client who had multiple websites operating in the same niche but catering to different states in the US. With almost identical content on many of them, they were not at all ranking good in the search engines overall. We had the content rewritten in such a way that all of them had unique content (please note that all these domains still operated in the same niche but with slightly different keywords/phrases based on the differences in naming conventions in different states). This worked wonders. The results exceeded all the expectations of the client. There was a dramatic increase in the total number of pages from these domains in Google's index along with a whopping increase in overall ranking for many of the competitive terms in the search engines.

With substantially unique, updated, highly relevant and useful content, you will see a healthy and steady growth in your SEO ROI. Those were my two cents my friend. Good Luck to you.

Best regards,

Devanur Rafi

James got it spot on. Google automatically displays site links.

Here is what Google has to say in this regard:

"We only show sitelinks for results when we think they'll be useful to the user. If the structure of your site doesn't allow our algorithms to find good sitelinks, or we don't think that the sitelinks for your site are relevant for the user's query, we won't show them.

At the moment, sitelinks are automated. We're always working to improve our sitelinks algorithms, and we may incorporate webmaster input in the future."

Here you go for more:

https://support.google.com/webmasters/answer/47334?hl=en

Best regards,

Devanur Rafi

Pleasure is all mine my friend. You are most welcome. Moz SEO community is an indispensable asset and weapon in any SEO's inventory in my opinion. We learn a great deal here while helping others. I am really thankful to each and everyone here on Moz community. Long live Moz and Mozzers. YOU ROCK!!

Yes, of course my friend, Google has to crawl the page to see the page-level meta robots tag but till date I have not seen any page in Google's index that has been blocked using the robots.txt file and page-level meta robots tag. Password protecting your .htaccess file would be an overkill if you just want Google not to index a page. If you want Google to remove any particular page from its index, you can get it done from webmaster tools account. Here you go for more: https://support.google.com/webmasters/answer/1663419?hl=en

Good Luck to you my friend.

Best regards,

Devanur Rafi

Hi friend,

If a page has been blocked using Robots.txt file, then Google will not crawl and index the page from within the website but what if a reference of that page is found on a third-party website? In cases like this, link discovery will happen and the page will be indexed without a Description snippet and such pages will have the following text in the place of a description in the search results pages:

"A description for this result is not available because of this site's robots.txt – learn more"

So inorder to completely stop Google from crawling and indexing a page, you should should block the page by implementing, page-level meta robots tag.

Here you go for more: https://support.google.com/webmasters/answer/156449?hl=en

Please feel free to post back if you have any other queries in this regards.

Best regards,

Devanur Rafi

Hi, while it is definitely possible for Google to penalize a page or a folder for that matter any section of a website, unless we take a look at the page you are talking about, all that we write will be just shooting in the dark.

Best regards,

Devanur Rafi.

Hi, you can try using Gsitecrawler.

Download it from here : http://gsitecrawler.com/en/download/

Best regards,

Devanur Rafi

Hi William, thanks for sharing your experience here. My experience has been totally different from that of your's.

Here is what Dr.Pete has to say..

Taken from: http://moz.com/blog/rel-confused-answers-to-your-rel-canonical-questions

2) Can I Use Rel=Canonical Cross-domain?

Yes – in late 2009, Google announced support for cross-domain use of rel=canonical. This is typically for syndicated content, when you’re concerned about duplication and only want one version of the content to be eligible for ranking.

(3) Should I Use Rel=Canonical Cross-Domain?

That’s a tougher question. First off, Google may choose to ignore cross-domain use of rel=canonical if the pages seem too different or it appears manipulative. The ideal use of cross-domain rel=canonical would be a situation where multiple sites owned by the same entity share content, and that content is useful to the users of each individual site. In that case, you probably wouldn’t want to use 301-redirects (it could confuse users and harm the individual brands), but you may want to avoid duplicate content issues and control which property Google displays in search results. I would not typically use rel=canonical cross-domain just to consolidate PageRank.

Best regards,

Devanur Rafi

Except for very exceptional cases, Google considers and respects the rel=canonical implementation and its a strong hint for them.

Here you go for more:

http://www.mattcutts.com/blog/rel-canonical-html-head/

Best regards,

Devanur Rafi

Hi, with rel=canonical in place, there is no way that both the pages from the two sites appearing and ranking in the search results. Only the canonical or the preferred page will be indexed and can rank in Google.

You should be thinking along the lines to make the content on both the sites unique. Though these two sites operate and target the same niche, you can definitely make the content unique from each other.

Best regards,

Devanur Rafi

Hi,

Its not new that Google changes the page titles and descriptions.

Here you go for more:

http://searchenginewatch.com/article/2342232/Why-Google-Changes-Your-Titles-in-Search-Results

and here..

http://searchengineland.com/google-title-wrong-157819

Best regards,

Devanur Rafi.

Hi Mike, this is awesome but strangely, the only page on Wikipedia that gets returned for the following query,

"inurl:digitalization site:wikipedia.org" in Google is <cite class="_Zd">ar.wikipedia.org/wiki/Digitalization</cite> that too in Arabic language. So they do not have the exact match definition in English at this point in time as I write this. You can confirm this with the following query:

"inurl:digitalization site:en.wikipedia.org"

Out of the first 10 results in Google for the query, "inurl:digitalization", I tried to run the following query for each of these top 10 websites:

"inurl:digitalization site:<each 10="" of="" the="" top="" domain="" names="">",</each>

Only Gartner.com has 26 URLs (with "digitalization" in the URL) with only one exception, digitalization.nl which can be ruled out as the domain name itself contains, "digitalization". This case can be dug much,much deeper but let me stop here.

Without having dug deeper in to this case, at this point, the only thing I believe would replace Gartner's definition with the definition from Wikipedia is, if they come up with a page that contains the term, "digitalization" in the URL and in the title.

Best regards,

Devanur Rafi

Hi, its time to check your titles and descriptions. With all respect for Yoast plugin, I would still recommend to write the page titles and descriptions manually at least for the most important pages on your website. Google changing the titles and descriptions is not new and all that we can do is try and test different titles and descriptions.

Here you go for more on this..

1. https://yoast.com/google-page-title/

2. http://searchenginewatch.com/article/2342232/Why-Google-Changes-Your-Titles-in-Search-Results

3. http://wordpress.org/support/topic/plugin-wordpress-seo-by-yoast-title-changes-to-individual-pages-categories-tags-posts-is-not-working

and here..

4. https://productforums.google.com/forum/#!topic/webmasters/K6D14caef7U

Hope they help.

Best regards,

Devanur Rafi

Hi friend,

You can use SEMRush for the purpose. You can have the data that you were looking for under,

Organic Search > Positions

There you can see all the popular URLs (pages) from a domain sorted as per the total organic traffic that they each receive. You can sort using many parameters there.

Hope it helps. Good Luck to you.

Best regards,

Devanur Rafi.

Lesley is absolutely correct. Writing a unique copy or even thousands of unique variations of a single article is very much possible. The best examples are article directories that contain thousands if not lakhs of unique articles on a same topic. Writing unique copy does not always mean to re-invent the wheel but can also be rewriting it while retaining the meaning or essence of the article, including some phrases or sentences unique to the author's writing style or something of that sort.

The key here is to make your copy different from millions of others that are already written and in the search engines' indices. It is also highly important to come up with such a variation that really adds value to your website and the Internet on the whole. Your web copy should justify its presence in the search engines by providing real value to the visitors. If you can do this, chances of your copy out-ranking the competition become high (of course can depend on ton of other ranking factors) as such content will be a natural link magnet and within no time, it would gain lot of SEO goodies.

Being a very strong advocate of unique and quality content my self, I give clear instructions to my content team while pointing them to few sample articles that are already there on the Internet.

Best regards,

Devanur Rafi.

You are most welcome David. You can use Yoast for Wordpress for handling rel=canonial.

Here for more: https://yoast.com/wordpress/plugins/seo/#canonical

Here you go for implementing 301 permanent redirection using .htaccess file on Apache server(Linux hosting):

http://www.inmotionhosting.com/support/website/redirects/setting-up-a-301-permanent-redirect-via-htaccess

If you are on a Windows server, here are the steps for 301 redirection:

http://www.iis.net/configreference/system.webserver/httpredirect

301 redirection is definitely faster than rel=canonical and almost same when it comes to passing on the SEO goodies to the canonical page. With 301 in place, in this case with the URLs mentioned by you for example(suppose, the author page is redirected to the homepage via 301), no one will be able to see the author page as it would take you to the homepage but with rel=canonical in place, everyone will be able to see the author page and its just that the search engines like Google will not index the author page as the cononical or the preferred page would be the homepage. So, ideally, you should be going with the rel=canonical implementation here. Hope it helps.

Good Luck my friend.

Best regards,

Devanur Rafi

Hi David,

I will be very quick here. 'rel=canonial' can come to your rescue.

Here you go for more: https://support.google.com/webmasters/answer/139066?hl=en

Here is another article from Moz regarding duplicate content and how to go about it:

http://moz.com/learn/seo/duplicate-content

Best regards,

Devanur Rafi

Hi, ideally only one version of the URL should throw an HTTP header status code 200 and rest all can be redirected to the preferred version via 301.

Suppose, http://www.abc.com is your preferred version (the www version), all others like,

http://abc.com, http://abc.com/index.html, http://www.abc.com/index.php etc., can be redirected to the www version via 301. It is highly recommended to maintain this consistency through out your website. You can use a tool like the following to check if you have multiple versions of your homepage return status 200.

http://www.ragepank.com/redirect-check/

Hope it helps. Good Luck.

Best regards,

Devanur Rafi

Hi friend, you can display the disclaimer using a JavaScript overlay and this would be absolutely fine. The bots won't have any trouble crawling the website behind the JS overlay as they won't see it. This is a very common practice among the websites that display age gate verification page like porn sites and sites that talk or sell liquor etc..

This technique is not considered cloaking as the intention is not malicious or deceptive and Google handles these normally. Hope it helps and Good Luck.

I addressed a similar question here on Moz:

http://moz.com/community/q/different-user-experience-with-javascript-on-off

Best regards,

Devanur Rafi

Hi, there is no added advantage or disadvantage with www in the URL. You can go either way, with or without www but be consistent throughout the website. You also set your preferred version in Google Webmaster Tools account. Make sure only one version, either www or without www returns an HTTP header status code 200 and not both. If suppose, http://www.abc.com is your preferred version, then only http://www.abc.com should return 200 and not both (http://abc.com). In this case, http://abc.com should be redirected to http://www.abc.com via a server-side 301 permanent redirection. This way, you can take care of your URL canonicalization issues.

One more thing, while doing any link building/earning activities, please make sure to use the preferred version for backlinking purposes. Please feel free to post any queries that you may have in this regard.

Good luck.

Best regards,

Devanur Rafi

Thanks for the info. The page in discussion is as good or even better than many other pages trying to rank for the term, 'sage erp x3' from your website. I can clearly see an issue with too many pages from your website trying to rank for the term. My honest suggestion would be to please have the page titles and descriptions written optimally. Here is a general rule, one One page (internal page) should be optimized for one keyword/phrase.

Moreover if you look at the page titles of your website in Google results, Google actually modified the titles. This is a clear hint that its time to optimize your titles.

Here you go for more: http://searchenginewatch.com/article/2342232/Why-Google-Changes-Your-Titles-in-Search-Results

Its actually very confusing for a search engine line Google about which page from your website should rank high for the term, 'sage erp x3' as there are a many pages competing for the same term leading to keyword cannibalization.

Best regards,

Devanur Rafi

As mentioned earlier, you can create a comprehensive Sitemap.xml file and resubmit to Google. Give me your email address, I will create one for you and send it

Best regards,

Devanur Rafi

Nevermind, there are about 497 pages on your website. So, I would ask you to use a comprehensive sitemap.xml file and submit it to Google. You can use a tool like GSite crawler to generate one.

Thanks for the info Susanta. After a quick check, I see the sitemap.xml that you submitted has only 130 URLs in it and there are 230 pages (omitted after 182) indexed in Google from your website. So, can you tell me how many pages are there in total in your website?

Hi, have you seen a drop in the overall organic traffic from Google during the past year? However, if you are using Google Analytics for your site, you can try the following tool to check if there has been a hit due to the Panda updates:

http://www.barracuda-digital.co.uk/panguin-tool/

Without knowing the exact domain or the URLs, its hard to assess the exact issues. If you can share those details, we will be able to comment better.

Best regards,

Devanur Rafi