Hi Daniel,

The first thing you need to do is to find the definition of the container that “this” is built out of and then add an element called metaDescription.

This will create an empty string until you populate it elsewhere.

Now where you want metaDescription shown, you just use

<title><?= $this->metaDescription ?></title> (replace metaTag with metaDescription).

Next, you need to search your code for the place where metaTag is assigned and add an assignment for metaDescription which can be whatever you want it to be ie a substring etc.

Hope that helps,

Sha

Hi Daniel,

Whenever I hear "Nothing is redirecting when I add this to the .htaccess file" I have one thought:

Is the site using Joomla, Drupal or some other CMS that will overwrite .htaccess rules externally?

Sha

There are social sharing possibilities available with Tumblr that can help SEO wise, as well as domain authority benefits that come with links from the blog to your existing site.

Andrew Dumont just wrote a post about changing from Wordpress to Tumblr for his personal blog.

As with everything it is a personal decision, but if I were forced to have a separate blog anyway, I think I would most likely go with Tumblr, unless there is some compelling reason to build a new domain from scratch.

Hope that helps,

Sha

Glad to be of help

One of the things I worry about most in SEO is changing things for the sake of change ... I've learned the hard way to take a breath and think before reacting. It's a hard lesson to learn, but it can be really satisfying when you discover that you made the right decision.

Hi Bill,

I assume when you say "I do email my list with each new blog post" that you are bulk emailing your established readers, to let them know you have posted since I can't see an RSS subscribe button on the site.

The fact is that established readers for the most part don't need to read anything more than the post you are telling them about in your email. They have most likely already read any other material on your site that interests them.

It is well known that blogs with significant numbers of subscribers often have high Analytics bounce rates for this reason. The data to focus on is sources of your traffic. Do you have a large proportion of referral or direct (type-in) traffic?

The thing to remember is that the bounce rate reported in Analytics is not the same as "real bounce rate" determined independently by the engines. "Real bounce rate" takes into account time on page. I would think it most unlikely that you will see any negative impact in Google or Bing with the time on page numbers you are quoting.

My most constructive advice would be:

1. Check to see exactly what percentage of your traffic is coming from search as opposed to referrals/type-ins
2. Stop messing with things to try to fix what isn't really broken - sooner or later it will hurt you.

The other comments here about the "look" of the site are probably valid - the style does bear a resemblance to some "sales letter" type sites, but on the other hand, you do have a lot of content and it sounds like a good number of happy readers. I would think carefully about any changes you make.

Hope that helps,

Sha

Thanks Ryan,

It's the part of my former life that seems to follow me around, so it's a good way for me to satisfy my overwhelming urge to copy edit everything on the internet!

Sha

Hi Fredrik,

I took a look at the site you have listed in your profile - you might want to add an "s" to "cost" in the text beneath your logo

It looks as though your main problems are spelling, typos and the odd runaway train of thought!

I'm not big on shameless plugs, but will post this one anyway since the link actually includes a "first page or post free" offer for SEOmoz Pro Members.

The site is Little MisSpeller (apologies to anyone offended by the self-promotion). You'll find a short description of appropriate skills and experience there.

As Ryan mentioned, there are a lot of services out there and it is really a matter of finding the one that will work best for you.

Hope that helps,

Sha

Hi John,

http://www.ixl.com/ is returning a 302 Redirect Status from the server - ie there is a 302 redirect in place

Also, http://ixl.com returns a 301 Redirect - when opened in a browser it redirects to http://au.ixl.com

Hope that helps,

Sha

Hi Karen,

There are a long list of URLs in your site which, although they do (eventually) load in a browser, appear to be returning a Status of 0 - Connection Timeout.

It is well recognized that Bing crawlers are very sensitive to URLs in sitemaps that return significant numbers of errors. In this case it appears that the crawler may be encountering pages in the site that it sees as unfetchable.

The majority of these URLs are in the following format - http://www.showme.com/sh/?i=36999 (this is one of them), although this one also returns the same result http://www.showme.com/topic/Geography

If you download the Screaming Frog tool from screamingfrog.com and run it, you will see the list of files that are returning this status.

It appears that the amount of time these pages are taking to load is causing a problem for some crawlers.

Hope that helps,

Sha

Hi GaB,

Sorry, I missed your reference to the URL changes in the original question.

It will depend on exactly what the changes are as to how many Rules will be needed to create the 301 redirects.

If you are retaining file names, but moving whole directories to a new location, then this can be achieved with a single Rule.For example, to 301 Redirect all filenames in Useless Folder 1 to the identical filenames in Relevant Folder 2 can be achieved with a single Rule.

For URLs where the actual filenames will change, or where only some files are relocated you would need to implement individual Rules for each URL.

However, if your site is large, there is another alternative, provided that your URL structure supports it. That would be the use of Database Rewrites to implement the 301 redirects as URL's are requested from the server.

There are some requirements for this to be a viable option:

• There must be a Database behind your site infrastructure
• There must be some identifier in the URLs that will remain constant with the change in site structure. Most commonly this would be a unique product ID number or  product name

Basically what happens with Database Rewrites is that when the server receives a request for a URL, the identifier is matched against those in the Database and when the match is found the 301 Redirect is written and the new URL served.

For very large sites Database Rewrites would be the most suitable solution as very long lists of Rules in the .htaccess will eventually impact processing and load times.

Hope that helps,

Sha

Hi Dennis,

SEOmoz will track rankings for all of the pages crawled within the root domain or subdomain that you have entered for your campaign, so the answer is NO.

If you are seeing different results when you check rankings manually in Google, you need to keep in mind that the SEOmoz Ranking report provides non-personalized results. Also, the way in which embedded local results appear in the SERPs will affect the result. When they are normal results with enhanced local listing they are counted.  If they appear in a 6,8, or 12 pack they are not counted. While Google continues to experiment with local listings, the SEOmoz team continues to respond with updates.

Hope that helps,

Sha

PS - If you want to optimize different pages for your keywords using the On-page Analysis tool, you can click "Report Card" in the On-page tool. You will see a selector at the top where you can type in any URL and choose any keyword from your list.

Hi Volkinator,

The 5XX Error is a generic error message which means that the crawler encountered an Internal Server Error. You will need to check your server logs for information that will point you to the source of the error - generally this will be a server software error that needs fixing. If you don't have the expertise to do this you will need to ask your Techs or Server administrator to fix it for you.

The Overly Dynamic URL notice is a Warning, rather than an Error. This is intended to bring to your attention the fact that these URL's contain more than 2 parameters. While search engines can crawl dynamic URL's, representatives of some search engines have advised that the inclusion of more than 2 parameters may negatively impact their ability to crawl. The Warning is provided by the SEOmoz Pro App so that you have the opportunity to correct the situation if you wish, however this may not always be possible. It is essentially your choice whether you pursue a fix for this one or not. Generally, if you have a small number of URLs with this problem it is probably not going to have a huge effect on your site. However, if the URL's concerned happen to be those related to your most important products or services, then you may want to get them fixed.

The 302 Redirect notice is also a Warning rather than an error. The main reason for highlighting them is to make you aware that they are not passing link value from page to page. If you have redirected the URLs with the intention that they will always point to the new URL, you should consider changing them to 301 Redirects as this will allow link value to flow from page to page. One reason for a lot of 302 Redirects on an ecommerce site may be that Buy buttons are set up to 302 redirect instead of using standard links. If yours is an ecommerce site and you have a lot of 302's this could be the reason.

Also a Warning, the Too Many On-page Links notice is telling you that there are more than 100 links on the page.

The explanation at the top of the page when you click the button to see the URLs in question is probably the best way of explaining. Here it is:

"You should avoid having too many (roughly defined as more than 100) hyperlinks on any given page. When search engine spiders crawl the Internet they are limited by technology resources and are only able to crawl a certain number of links per webpage. In addition, search engine algorithms divide the value of some popularity metrics by the amount of links on a given page. This means that each of the pages being linked to from a given page are also affected by the number of links on the linking page. For these reasons, we recommend you include less than 100 links per page to ensure that they are all crawled, though if your pages have a high page authority, search engines will usually follow more links."

The most important part of this is the watering down of link value, but sometimes extensive navigation will mean that you are more prone to having more than 100 links on the page. Once again, it is your decision as to whether you accept the issue as sub-optimal, but unavoidable, or whether you pursue a fix.

Hope that helps

Sha

Hi Sonja,

It sounds like you are referring to issues with pagination within your site.

The rel=next and rel=previous tags were recently made available to address this situation.

Here is an explanation of how to implement the rel=next & rel=previous tags from the Google Webmaster Central Blog.

Hope that helps,

Sha

Hi William,

Of course your take on what is ideal (and Sparkplug's) is totally correct, but my guess, given that the recommendation is not to move the entire site to the LAMP setup, is that they are faced with a bit of a catch 22.

Obviously without knowing the exact situation it is only a guess, but I would be betting that the main site is inextricably tied to Windows technology which makes it either impossible or cost-prohibitive to switch platforms. If that is the situation, then the proposed solution would be the best option if you definitely require a Wordpress installation.

The obvious alternative, since the blog would be separate anyway, would be to set up a Tumblr blog with links back to the main site.

Sha

Hi Sonja,

Adding the meta noindex tag will instruct the search engine to remove the URL from the index.

In almost every case I would recommend using noindex, follow as this will allow link juice to continue to flow to other pages within your site.

Hope that helps,

Sha

Hi,

My first impression is that your existing site is under windows hosting and your developers' advice is aimed at ensuring that your Wordpress installation is on a LAMP system. This makes total sense.

While it is possible to install and run Wordpress on Windows, the preferred option would always be to use standard LAMP hosting as there are things in WP that will not work as they should with Windows hosting. Most notable of the things that will be a problem is URL rewriting. Running WP on Windows requires a high level of expertise and is likely to require ongoing work, hacks etc over time.

In contrast, Wordpress can be installed and self-managed with great success on a LAMP system.

On the whole I would say that the solution recommended by your developers would be less frustrating and way less expensive than installing WP in a Windows environment (for both you and your developers).

Hope that helps,

Sha

Hi Alexandre,

You will need to look at the code in the .htaccess file generated by All in One SEO to see whether the plugin is just using URL rewriting or creating 301 redirects. As far as I am aware, that particular plugin does not have an option to manually stipulate when you wish to create a 301, but the only sure way is to check the code.

You will need to go into your wordpress /blog directory and download the .htaccess file, then open it in a text editor (like notepad). This is a separate .htaccess file, specifically relating to what happens within your wordpress installation.The one in the root folder for your site will not tell you what you are wanting to know.

I don't use All In One SEO as I prefer the Yoast plugin, but typically, what you might expect to see is code that looks something like this:

BEGIN WordPress

<ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /blog/
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /blog/index.php [L]</ifmodule>

END WordPress

The key to identifying whether the plugin is using standard URL rewrites or creating a 301 Redirect is to look at the very last part of the Rule - the part that is enclosed in square brackets [ ]

A plain vanilla flavored standard URL Rewrite Rule will end with [L]

If the Rule is creating a 301 (Permanent) Redirect, it will end with [R=301,L]

and for a 302 (Temporary) Redirect, it will end with [R=302,L]

As far as your question, which I understand to be asking essentially, "what is the difference" between the two:

• A standard URL Rewrite is simply instructing the server that any request for a certain URL  should be served a different URL. As far as the search engine is concerned, nothing changes. It is simply used to change the ugly URL to a pretty URL (in your case one that contains the keywords you want).

• A 301 redirect serves the alternative URL, but also sends a signal to the search engine that the URL requested has been permanently replaced with the one that is served. This indicates to the search engine that the requested URL should be removed from the index and replaced with the URL that is served. A 301 redirect also signals to the search engine that most of the link value being passed to the requested URL should now be passed to the URL that replaces it in the index.

Hope that helps,

Sha

Hi Alexandre,

You will need to look at the code in the .htaccess file generated by All in One SEO to see whether the plugin is just using URL rewriting or creating 301 redirects. As far as I am aware, that particular plugin does not have an option to manually stipulate when you wish to create a 301, but the only sure way is to check the code.

You will need to go into your wordpress /blog directory and download the .htaccess file, then open it in a text editor (like notepad). This is a separate .htaccess file, specifically relating to what happens within your wordpress installation.The one in the root folder for your site will not tell you what you are wanting to know.

I don't use All In One SEO as I prefer the Yoast plugin, but typically, what you might expect to see is code that looks something like this:

BEGIN WordPress

<ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /blog/
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /blog/index.php [L]</ifmodule>

END WordPress

The key to identifying whether the plugin is using standard URL rewrites or creating a 301 Redirect is to look at the very last part of the Rule - the part that is enclosed in square brackets [ ]

A plain vanilla flavored standard URL Rewrite Rule will end with [L]

If the Rule is creating a 301 (Permanent) Redirect, it will end with [R=301,L]

and for a 302 (Temporary) Redirect, it will end with [R=302,L]

As far as your question, which I understand to be asking essentially, "what is the difference" between the two:

• A standard URL Rewrite is simply instructing the server that any request for a certain URL  should be served a different URL. As far as the search engine is concerned, nothing changes. It is simply used to change the ugly URL to a pretty URL (in your case one that contains the keywords you want).

• A 301 redirect serves the alternative URL, but also sends a signal to the search engine that the URL requested has been permanently replaced with the one that is served. This indicates to the search engine that the requested URL should be removed from the index and replaced with the URL that is served. A 301 redirect also signals to the search engine that most of the link value being passed to the requested URL should now be passed to the URL that replaces it in the index.

Hope that helps,

Sha

Hi again,

Currently you have that product number listed under two different URL's

http://www.vistastores.com/indoorlighting-patiolivingconcepts-20947.html is returning a 301

http://www.vistastores.com/floor-lamps-patio-living-concepts-20947.html is returning a 404.

There are those who would say that it is perfectly fine to return a 404 as long as the 404 page contains proper navigation links and access to other products etc.

I do not disagree with their opinion as far as what should be on the page that is returned (your custom 404 page does this), but in the case of an ecommerce site, I disagree that it is a good idea to just let all expired products return 404 errors.

The nature of ecommerce is such that this will eventually result in search engines seeing an ever-growing list of broken links on your site. With large ecommerce sites, this could result in thousands of 404 errors which the search engine will see no attempt is ever made to correct or reduce.

If you choose to allow searchers to be sent to your custom 404 page, I take a different approach:

• Name the custom 404 page something innocuous like /custom-introduction
• Remove the 404 references from the Title, description and page text - just use a simple explanation that the product is no longer available and invite the visitor to select from the categories available to search for another product, without mentioning the term "404 not found"
• add a catch-all 301 redirect which sends any request for a page that does not exist on the server to /custom-introduction.

By doing this you will avoid creating an ever growing list of 404's, but as I said, there are those who don't worry about that at all.

As to not using 301's because you have put all that work into the pages, if they are disabled in your store the content will no longer be visible to the search engines anyway. As much as we all hate having our hard work seemingly come to nothing like this, it is very much the nature of the business in ecommerce. Products come and go all the time. This is exactly why the 301 redirect is the best option - because it passes almost all of the link value to the new page. So your hard work has not been wasted, but helped to improve the value of another page on the site.

Hope that helps,

Sha

Hi there,

Unfortunately products that go out of stock are one of the biggest frustrations for ecommerce site managers. Where products are sourced from another company you can be very much at the mercy of the way that they manage their inventory. This means you need to decide which method of dealing with expired products will work best for you.

From what you have described it sounds like the product was marked temporary out of stock by the supplier, then more recently changed to "Do Not Reorder" status. So, the product has permanently expired and will not be restocked at any time in the future.

Your best option in this situation is a 301 redirect, which will pass link value from the page to whichever page you decide to redirect to. This is where you need to make decisions.

You have essentially 3 choices when choosing an appropriate page to set your redirect target.

1. If you have a product in your inventory that is very similar, you could redirect to that as the product may be of interest to people who were looking for the original product.
2. If you do not have a similar product, then you can redirect to the higher level category page for that item.
   For example, redirect to http://www.vistastores.com/lighting
3. Finally, since you have an internal search mechanism, you could choose to redirect to a search for a relevant term.
   For example, redirect to search ?q=lighting

The aim is to always send the visitor to the most relevant or helpful page to improve the chance of them converting.

Hope that helps,

Sha

Hi GaB,

I would suggest a completely different solution.

I would not use 301 Redirects for the Phase 1 change. Rather, I would use a Rewrite Rule to serve the file with the new extension. The Rewrite Rule is basically telling the server to serve OURLEX when a request is made for OURL. The nett result of this is that as far as the Search Engine is concerned, nothing changes.

Here is a Code example which will serve the .php when the .aspx is requested:

RewriteEngine on

RewriteBase /
RewriteRule ^(.*).aspx$ $1.php [L]

Then, when Phase 2 is complete, I would implement the 301 redirects using a single Rule once again to permanently redirect all OURL's to NURL's. So, the solution I would recommend is the last one on your list.

Option 6 - Something else I think is better:

• Phase 1: Use Rewrite Rule to satisfy all requests for the OURL by serving the OURLEX
• Phase 2: create one rule that will 301 redirect all the OURL to the NURL

It is best to keep in mind that a 301 redirect should only be used when you intend the redirect to be permanent. Also while Google's stated position is that googlebot will follow multiple 301 redirects, it has also said that "daisy chaining" them is not generally a good practice. There are two reasons for this:

1. Google's Matt Cutts has indicated that "at some point we will stop following them if there are a lot of them"
   Should you happen to have other domains that are already 301'd to the site, the daisy chain effect is amplified.
2. As Ryan mentioned, the small amount of link juice that is lost with a 301 Redirect does accumulate over multiple redirects and can have a negative impact.

Hope that helps,

Sha

Hi Augustos,

Yes, bounce rate is now acknowledged as a ranking factor by major search engines.

I would say you are probably doing exactly what I talked about in my response to Volkinator - are your conversion rates also improved?

Sha

Hi Volkinator,

I would actually offer a slightly different perspective on this.

I agree that there doesn't appear to be any real evidence that it helps rankings, but...

Think about what works for the visitor ... clever use of bold text can actually help conversions, but you have to think carefully about what you are doing because it can also hurt them terribly if misused.

First, highlighting just repetitive keyword terms or long sections of text is likely to hurt in 2 ways:

1. It is likely to annoy people and therefore increase the bounce rate for the page.
2. long passages of bold text reduce comprehension dramatically - if people don't understand the text on the page they are less likely to convert.

However, you can highlight just short phrases in the text to send a direct message to the reader. This will drive your message home more effectively and increase conversions.

This is an artificially constructed version to demonstrate what I am talking about, but in essence, the highlighted phrases when read alone create a shorter, more direct action statement from within the two longer sentences.

I have found this to be quite effective in improving conversions on client sites. One site in particular went from a conversion rate of 2.2% to a conversion rate of 26% after making only this change.

Hope that helps,

Sha

Hi Simon,

I would suggest starting with the basics - getting a good grounding in HTML and CSS will make a huge difference when you are not restricted to getting someone else to make every change for you. You will also find it easier to work with your tech people once you know the basics.

Once you get a taste for it I suspect you'll find yourself seeking out other languages to learn.

Hope that helps,

Sha

Hi Ian,

It's great to hear that Q&A is working for you.

I have to confess that a large share of the "warm fuzzies" in my life emanate from this forum. Seeing so many people sharing and working together here every day always puts a smile on my face.

Thank you too for posting this - as EGOL mentioned, it's great to know what has worked and what the outcome was for your site. Feedback like this is as valuable for the rest of us as the answers were to you.

I would most definitely agree - Q&A alone can make the subscription cost worthwhile

Sha

Hi again Greg,

Just one more option that is available to you if you happen to have a Wordpress blog on the site (or have the option of rebuilding the entire site using Wordpress).

You could install the Bad Behavior plugin for Wordpress. The plugin is part of Project Honeypot, which tracks millions of bad ip addresses and gathers information from the plugin and feeds it back to the honeypot. Bad Behavior also works against link spam, email and content harvesters and other malicious sites.

Sha

Hi,

You can certainly export all of the keywords in your campaign in a single csv file, select keywords using the usual "check the box" system to delete and also bulk add keywords to a campaign.

To export all of the keywords loaded in your campaign, go to the Rankings Screen, change the selector at the top right of the Rankings Report from pdf to csv and click export. You can now open the file in Excel and the first column of the report is your entire keyword list.

To bulk add keywords to your campaign, click Manage Keywords from the Campaign Overview OR Click the Add/Manage Keywords button on the Rankings screen.

You can paste a list into the input field. Keywords can be separated by comma, tab or new line. If you have the keywords in an excel list (maybe exported from Analytics or similar), just copy the keywords from the column and paste.

The one that will be most onerous if you have hundreds of terms is deleting, as terms are listed alphabetically, so you need to work through the list and check the boxes for each before clicking the red Delete Keywords button.

You will find the delete option in Manage Keywords also (below the input to add keywords). If by chance the keywords you want to delete happen to have labels assigned to them, you can use the selector at the top of the screen to choose only keywords with those labels, which would make choosing the checkboxes a little easier.

Hope that helps,

Sha

Sorry Alan,

Obviously misread what you were asking.

The answer is No. You cannot change the domain name. The following is from the FAQ on campaign setup:

Q.  Can I change the domain later?
A.  You cannot change the domain of a campaign once it is created. If you need to track content, rankings, etc. on a new subdomain or domain, you must create a new campaign.  Don’t panic – archiving and adding campaigns is easy.

It is also not possible to duplicate the campaign and change details, so you will need to archive the old campaign and create a new one, but that is not so difficult to do.

The most arduous part of re-creating a campaign is adding the keywords, but this is also quite simple. All you need to do is export your keyword list to csv from the old campaign, open it in Excel, select the keywords in the first column and copy to your clipboard.

When you create the new campaign, just add one "placeholder" keyword in the screen that asks for keywords, then continue setup. When the campaign has been created, click "Manage Keywords" link at the top of the Overview page and paste the keywords from your csv file into the input field. Click Add Keywords and you're done.

If you choose a "placeholder" keyword that was not in the original list then you're fine. If you used one that was already in the list, you will just need to delete the duplicate so it does'nt mess up your reports.

Hope that helps,

Sha

Hi Vjay,

The sitemaps protocol was adopted as a standard by the three major search engines back in 2007.

This is not to say that other methods of submitting a sitemap don't work, but if you follow the sitemap protocol as described at sitemaps.org then you know that you are working within the standard recognized by Google, Bing & Yahoo.

Hope that helps,

Sha

Hi Alan,

You can change the campaign name by clicking Settings in the campaign.

Just change the information in the Campaign name field and save.

Hope that helps,

Sha

hmmm...I like to pick my battles.

Scumbags are scumbags and will always find a way to win in the short term.

I like to live by two things my grandma taught me a long time ago...

"What goes around comes around" and "revenge is a dish best served cold"

As to there being an easy way out - you're an SEO Ryan! You know the deal.

Sha

Hi Ryan,

In this case Greg already knows the site has been scraped and duplicated. Blocking the scraper and serving the image via the bot-response php script is simply a "gift" to the duplicate site if they return to update their stolen content as they often do.

It is entirely possible to put the solution in place for well known scrapers such as Pagegrabber etc, but there are thousands of them, the people using them can easily change the name when they have been outed and anyone can write their own.

I understand that everyone wants a "list", but even if you Google "user agent blacklist" and find one, there will be problems. Adding thousands of rules to your .htaccess will eventually cause processing issues, the list will constantly be out of date etc.

As I explained at the outset, the key is to be aware of what is happening on your server and respond where necessary. Unfortunately, this is not a "set and forget" issue. In my experience though, bots will likely be visible in your logs long before they have scraped your entire site.

Sha

The reason is fairly obvious...gaming search engines with social mentions.

This will never go away because there will always be masses of people out there who will think "what have I got to lose?". People recognize that if it's a scam all they lost was a click and if it isn't they'll get something for nothing.

The general public has no idea that a share is "worth" anything for a website and values a click at zero.

Sucks to be them if someone sees it and decides to let the engines know what is going on though eh?

Sha

Hi Susanna,

If you have trouble getting someone to fix the .htaccess for you, feel free to private message the file and we'll be able to sort it out for you.

It is never advisable to treat rules in the .htaccess in isolation as the order in which they appear in the file will determine how things work as Ryan explained. There are also other things that will influence whether the redirects function the way you want them to such as rules added to the file or overwritten by standard installations such as Wordpress, Joomla etc.

Hope your provider can get it sorted out for you. If not, just let us know and we'll be happy to help.

Sha

I guess the use of bot-response.php and bot-response.gif is the gentle internet version of a public shaming campaign.

Sometimes it's a matter of picking your battles, but engineering enough of a win to make your client feel better without launching into an all-out war that could end up costing way more than you're willing to pay.:)

Sha

bot-response.gif

Authorship markup was recently introduced by Google because there was no reliable way to verify ownership. The markup can be added to any page on any site to show Google that you are the original author.

Sha

Hi Ryan,

The major problem is that any experienced programmer can easily write their own script to scrape a site. So there could be thousands of "bad bots" out there that have not been seen before.

There are a few recurring themes that appear amongst suspicious User Agents that are easy to spot - generally anything that has a name including words like grabber, siphon, leach, downloader, extractor, stripper, sucker or any name with a bad connotation like reaper, vampire, widow etc. Some of these guys just can't help themselves!

The most important thing though is to properly identify the ones that are giving you a problem by checking server logs and tracing where they originate from using Roundtrip DNS and WhoIs Lookups.

Matt Cutts wrote a post a long time ago on how to verify googlebot and of course the method applies to other search engines as well. The doublecheck is to then use WhoIs to verify that the IP address you have falls within the IP range assigned to Google (or whichever search engine you are checking).

If you are experienced at reading server logs it becomes fairly easy to spot spikes in hits, bandwidth etc which will alert you to bots. Depending which server stats package you are using, some or all of the bots may already be highlighted for you. Some packages do a much better job than others. Some provide only a limited list.

If you have access to a programmer who is easy to get along with, the best way to get your head around this is to sit down with them for an hour and walk through the process.

Hope that helps,

Sha

PS - I'm starting to think you sleep less than I do!

Hi Atul,

The most reliable way to show Google ownership of your content is to implement Authorship markup.

Matt Cutts and Othar Hansson explain the different methods in this Google Webmaster Help video on Authorship Markup and this alternative method Using URL Paramaters for Authorship Markup if you are unable to edit tags (if for example you are using a cms).

There is also a very good explanation for bloggers from Yoast that is worth reading.

Hope that helps,

Sha

Hi Greg,

Awesome information there from Ryan!

Implementing the authorship markup is important in that it basically "outs" anyone who has already stolen your content by telling Google that they are not the original author. With authorship markup properly implemented, it really doesn't matter how many duplicates there are out there, Google will always see those sites as imposters, since no-one else has the ability to verify their authorship with a link back from your Google profile

It is possible to block scrapers from your server (blacklist) using IP address or User Agent if you are able to identify them. Identification is not very difficult if you have access to server logs as there will be a number of clues in the log data. These include excessive hits, bandwidth used, requests for java and css files and high numbers of 401(unauthorized) and 403 (forbidden) HTTP error codes.

Some scrapers are also easily identifiable by User Agent (name). Once the IP address or user agent is known, instructions can be given to the server to block it and if you wish, to serve content which will identify the site as having been scraped.

If you are not able to specifically identify the bot(s) responsible, it is also possible to use alternatives like whitelisting bots that you know are OK. This needs to be handled carefully as ommissions from the whitelist could mean that you have actually banned bots that you want to crawl the site.

If using a LAMP setup (Apache server), then instructions are added to the .htaccess file using PHP. For a Windows server, you use a database or text file with filesystemobject to redirect them to a dead end page. Ours is a LAMP Shop, so I am much more familiar with the .htaccess method.

If using .htaccess you have the choice of returning a 403 FORBIDDEN HTTP error, or using the bot-response.php script to serve an image which identifies the site as scraped

If using bot-response.php, the gif image should be made large enough to break the layout in the scraped site if they serve the content somewhere else. Usually a very large gif that reads something like: "Content on this page has been scraped from yoursite.com. If you are the webmaster please stop trying to steal our content".will do the job.

There is one VERY BIG note of caution if you are thinking of blocking bots from your server. You really need to be an experienced tech to do this. It is NOT something that should be attempted if you don't understand exactly what you are doing and what precautions need to be taken beforehand. There are two major things to consider:

1. You can accidentally block the bots that you want to crawl your site. (Major search engines use many different crawlers to do different jobs. They do not always appear as googlebot, slurp etc)
2. It is possible for people to create fake bots that appear to be legitimate. If you don't identify these you will not solve the scraping problem.

The authenticity of bots can be verified using Roundtrip DNS Lookups and WhoIs Lookups to check the originating domain and IP address range.

It is possible to add a disallow statement for "bad bots" to your robots.txt file, but scrapers will generally ignore robots.txt by default, so this method is not recommended.

Phew! Think that's everything covered.

Hope it helps,

Sha

There has been quite a bit of experimentation which indicates that the "First link counts rule" does not hold in certain situations.

This Youmoz post gives a good explanation - Exceptions to the First Link Counts Rule

Hope that helps,

Sha

Hi again,

A 301 redirect is an instruction given to the server which tells it which page to send the request to.

If you were using an Apache server, you would create a .htaccess file with the lines of code in it to create the redirect.

With a Windows server you should have access to what is effectively a server management console that you can log into. In this console, within the Internet Services Manager, you basically choose the options which create the redirect within the server for you.

It does not matter whether your pages are .html or .php ...the process is the same according to which server setup you are dealing with.

I think perhaps you are confusing a 301 redirect with a redirect script which can be placed in the head of a document. This latter is not a great option for a number of reasons ...most notably that it does not allow link value to flow to the new page and also will not tell the search engine to de-index the old page and replace it with the new.

Hope that helps,

Sha

Hi MyNet,

The 301 redirect must be created on the originating server as it needs to intercept the request and send it to the new location.

Since you say that you are hosting on a microsoft server, I am thinking that you need the method for creating a 301 redirect on a IIS server.

You will find these instructions on this quick reference page for creating 301's.  This page actually provides instructions and/or code examples for about every situation you could think of...just choose the one you need. (you should check with your ISP to make sure that you know which suits your situation)

Hope that helps,

Sha

yep...what he said

The most obvious reason for using noodp and noydir is to stop the search engine from using information from those directory listings (which may be out of date), but Google is choosing to return what it thinks" most accurately represent the content on the page these days. Sometimes that might be what you provide in your Titles and tags, sometimes not.

Sha

Wow! I'll take that as a compliment

Just don't tell Dr Pete!

Always easier to switch on to something when you see it produce a real result. Glad I could help.

Have a great night,

Sha

Hey Ryan,

Just using Screaming Frog SEO Spider. It has proven very reliable and quickly identifies errors, server status for every page and much more.

The caveat is that I always check any Status code errors in the browser as there are quite often situations like this where the server is returning a Status error when the page renders fine in the browser.

You just have to be careful to ensure that if you want to scan the root domain you use the non-www URL as usual.

Hope it's useful,

Sha

Hi Carlton,

Awesome answers from everyone here!

I just wanted to add that it appears there is only the one 302 redirect that has been incorrectly placed, but of course it is an important one as 302 redirects do not pass link value, so any external site that happens to have linked to your non-www domain will not be adding value to your link profile.

Also, this page http://www.ietravel.com/central-south-america, although appearing to be fine when viewed in the browser, is returning a 500 Error for crawlers (Internal Server Error). You will need to get your tech people to sort this out for you.

Hope that helps,

Sha

Hi Marc,

Great answer from Ross!

What you need to do is be very strategic and learn to pick your battles so you have a better chance of winning

I believe the best way to do this is to stick with SEOmoz and soak up all the knowledge you can from the Blog, Webinars, Whiteboard Friday videos and of course, make use of Q&A.

Also, use the SERPs Analysis Tool to generate advanced reports for your "money" keyword terms and see if there are easy wins in on-page and/or social media that you may not have thought of. Rand gave a detailed explanation of the tool and how you can use it to compare metrics for the top 10 sites returned for your chosen keyword term. There is also a link to the tool at the beginning of the post.

Hope that helps,

Sha

Hi Anthony,

No...what I was talking about would give you two almost identical URL's.

For example:

http://outdoortrailcams.com/busebo.html and http://www.outdoortrailcams.com/busebo.html

Since the page you have a problem with is a blank page, this help page is more likely what you need How to Delete a Page in Yahoo Store. Do be careful to ensure that it is not a "section" page as described there, since pages related to those would also be deleted.

Hope that helps,

Sha

Hi Anthony,

If the issue you have is related to the fact that you have both www and non www versions of the pages, then you can implement a 301 redirect within the Yahoo Stores system to take care of this.

If you go to this Yahoo Stores 301 Help Page you will find a detailed explanation of the various ways that you can implement redirects.

The part that applies to this situation is in the following paragraph:

You can also choose to redirect all hostnames and domain to your selected domain so your non-www address (acme.com) points for example to your www domain (www.acme.com). This setting also redirects your index page (for example www.acme.com/index.html) to your selected redirect setting (for example www.acme.com). This allows you to have one url (also known as a canonical url) rather than multiple so search engine relevancy factors will accrue to one url rather than being spread out across multiple urls.

The instructions are directly below that paragraph and the diagram shows you where to choose the redirect you want to use A.

Hope that helps,

Sha

Hi Steven,

There are two different robot related warnings in your SEOmoz PRO App report. The second one refers to pages or directories blocked by a disallow statement in a robots.txt file.

Next step is to check whether you have a robots.txt file in place for your site and if so, then check to see if there are files or directories disallowed in the file.

If the robots.txt file is not an issue, then the your best course of action is to email the SEOmoz Help team direct - help [at] seomoz.org. Give them the details of your campaign and explain the problem and they will take a look at it for you.

Hope that helps,

Sha