Hi Zippy-Bungle,

To understand first why the 803 error was reported:

When a page is called, the web server sends header details of what's to be displayed. You can see a complete list of these HTTP header fields here.

One of the headers sent by the web server is Content-length, which indicates how many bytes the rest of the page is going to send. So let's say for example that content length is 100 bytes but the server only sends 74 bytes (it may be valid HTML, but the length does not match the content length indicated)

Since the web server only sent 74 bytes and the crawler expected 100 bytes the crawler sees a TCP close port error because it is trying to read the number of bytes that the webserver said it was going to send. So you get an 803 error.

Now browsers don't care when a mismatch like this happens because Content-length is an outdated component for modern browsers, but Roger Mozbot (the Moz crawler, identified in your logs as RogerBot) is on a mission to show you any errors that might be occurring. So Roger is configured to detect and report such errors.

The degree to which an 803 error will adversely affect crawl efficiency for search engine bots such as Googlebot, Bingbot and others will vary, but the fundamental problem with all 8xx errors is that they result from violations of the underlying HTTP or HTTPS protocol. The crawler expects all responses it receives to conform to the HTTP protocol and will typically throw an exception when encountering a protocol-violating response.

In the same way that 1xx and 2xx  errors generally indicate a badly-misconfigured site, fixing them should be a priority to ensure that the site can be crawled effectively. It is worth noting here that bingbot is well known for being highly sensitive to technical errors.

So what makes the mismatch happen?

The problem could be originating from the website itself (page code), the server, or the web server. There are two broad sources:

1. Crappy code
2. Buggy server

I'm afraid you will need to get a tech who understands this type of problem to work through each of these possibilities to isolate and resolve the root cause.

The Moz Resource Guide on HTTP Errors in Crawl Reports is also worth a read in case Roger encounters any other infrequently seen errors.

Hope that helps,

Sha

Hi Eric,

This should give you the basics:

• Add Keywords to Product Titles and Descriptions
• Add UPC Codes (required)
• Add Shipping cost info (required)
• Add Seller Reviews
• Add Rich Snippets
• Optimize your Product_Type column
• Only use Google's product taxonomy for google_product_category

Remember not to confuse product_type and google_product_category. Both should be used, but the latter must use the correct taxonomy. Taxonomy is downloadable from the Google Merchant Center.

Do some careful reading on that page and throughout the Merchant Center. This is one area where crossing t's and dotting i's is VERY important!

Even though google_product-category is not required for all countries and products, recommendation is to use it for ALL products. Taxonomy is available for download as both .txt and excel spreadsheet.

Make sure your feeds are clean and push them every day.

Best of Luck!

Hi Ryan,

All great advice here (wearing out my thumb up guys!).

Just wanted to suggest that you try running the keyword terms it help and computer help. I just ran them in the Google Adwords Keyword Tool (exact match) and it returned some related terms with decent search and low/medium competition that could work for you.

Also, any time you are struggling to come up with the right terms for a site, try just asking people you know what they would search for if they needed ".................". Sometimes the bleedingly obvious can be the last thing we think of and other times people come up with what I think are the weirdest terms that are frighteningly common!

Once you have chosen some terms that might be suitable, you can also use the Keyword Difficulty & SERP Analysis Tool to help identify the strengths & weaknesses of sites already ranking in the Top 10 for that term. If you have not used the tool before, Rand's post The Best Kept Secret in the SEOmoz Toolset gives a good explanation of how you can use it.

Hope that helps,

Sha

Hi Charles,

I would say

1. Keep an eye on #Mozcon and connect with some people before you get to Seattle
2. Stay over and take a Tour of the Mozplex
3. Spend some real time checking out the Speaker photos on this page so you know who they are and what they look like...there is a real chance you will find yourself standing next to them in the lunch queue, sitting down to breakfast or lining up to sing Karaoke with them!
4. Be at the conference early every day and try to sit with new people as well as catch up with friends you have already made.
5. Read this post from Zeph , hit him up on Twitterand get him to hook you up for a spot at the informal Meetup on the night before the conference
6. Make a list of awesome people that you would like to meet while there and keep it close handy
7. Make a donation to the Myrtle Point Library in honor of Michael Cottam's parents at the conference and help send love to Michael and his family from everyone in the Moz "family"
8. Get some tips on where to find the Best coffee in Seattle from Jonathon Coleman
9. Arm yourself with Rand's Restaurant and Bar Guide to Seattle and the The Not-so-Short Shortlist of Moz's Top Seattle Restaurants, Bars, and Activities for MozCon 2013
10. Bring business cards and make sure you make notes on the cards people give you so you can easily put names to faces and shared experiences when you get home.

Oh...and you might consider joining a PosSEO, becoming a PosSEM or something like that, but whatever you do, say G'day!

Sha

Hi Eric,

A quick look in Majestic SEO shows nearly 34,895 external backlinks from 260 linking root domains with large numbers of links being added during March and April.

As mentioned by Bradley and Chris, there is a lot of suspect anchor text (including cialis, adipex, tramadol etc) in there and many of the referring sites are not places would really want to be getting your links from regardless of the anchors.

It seems your site has attracted some unwanted attention which has dramatically lowered Google's trust. While the obvious conclusion is that Penguin might be responsible, it is clear from the screenshot you provided that the dropoff actually happened around 5/6 May (Dr Pete has talked about "something" happening with the algorithm around 5 May and that would seem to correlate with your problems). Of course, it is possible that this was a test rollout of Penguin and your site was captured as a result, or it may be something completely different (which Google has given no information about).

Either way, it is easy to see that the linking behavior is harming your site's trustworthiness and unfortunately, this is likely to also affect your visibility in Bing and other Search engines as well.

Dealing with the backlink problem is the most critical place to start ... site updates will come later I'm afraid.

Hope that helps,

Sha

Hi Rob,

First of all ...this is not a domain redirect.

What they are actually doing is pulling the content of your site into their own using an iframe.

They are not able to do anything through your site by doing this as the content is rendered by the browser and not their server.  So, the question is why they would do it.

Best guess: This could be someone who is planning to set up some kind of low quality site (possibly full of ads), but wants to build up backlinks for the domain. They can go to blogs, forums etc and leave comments & posts with their URL. The webmaster checks the URL and sees your site, so approves the comment or post...after a few months of doing this, BAM! they remove the iframe and let loose their real content.

hmmm...

Sha

Hi Croozie,

Awesome work once again from Ryan!

Since your question feels like a request for suggestions on "how" to create a solution, just wanted to add the following.

When you say "classified listings" I hear "once off, here for a while, gone in 45 days content".

If that is the case, then no individual expired listing will ever be matched identically with another (unless it happens to be a complete duplicate of the original listing).

This would mean that it would certainly be relevant to send any expired listing to a higher order category page. If your site structure is such that you have a clear heirarchy, then this is very easy to do.

For example:

If your listing URL were something like http://www.mysite.com/listings/home/furniture/couches/couch-i-hate.php, then you can use URL rewrites to strip out the file name and 301 the listing to http://www.mysite.com/listings/home/furniture/couches/, which in most cases will offer a perfectly suitable alternative for the user.

There is another alternative you could consider if you have a search program built in - you could send the traffic to a relevant search. In the above example, mysite.com/search.php?s=couch.

Hope that helps,

Sha

Hi waspman,

No, you are absolutely not doing anything wrong

The simple answer is that you are acquiring links primarily to the sub-domains and not the root domains.

To explain:

"mozRank is SEOmoz's global link popularity score. It compares the relative link value (ranking power) between URLs on the Internet. It is similar in premise to Google's original PageRank metric but is updated more frequently and offers greater precision."

You can read more about all of the SEOmoz Metrics at the Open Site Explorer explanatory page.

So, if you go to Open Site Explorer and run a query for your sub-domain you will see the number of external links to that URL. If you then remove the www and run the query for the root domain you will see that there are 0 links to that URL.

Since mozRank = link popularity - it should be higher for the sub-domain because that is where the links are pointing, and of course you want to ensure that your links are always consolidated to just one domain. So, things are exactly as they should be.

Hope that helps,

Sha

Hi Rick,

If you wish to use the robots.txt method to disallow all or part of your site's https protocol, you simply need to load two separate robots.txt files.

The http and https protocols are basically viewed by bots as if they were two completely separate root domains (which I guess you already know as you have mentioned the fact that port 443 is used for the secure protocol).

Google's advice is that to use this method, you should have a separate robots.txt file for each protocol with code as follows:

For your http protocol (http://www.startuploans.org/robots.txt

User-agent: *
Allow: /

For the https protocol (https://www.startuploans.org/robots.txt

User-agent: *
Disallow: /

However, blocking crawlers with robots.txt is not the most reliable method for excluding pages from Search engines. The reason for this is that the page will continue to be indexed if it happens to be found via a link from another page. Basically, the robots.txt is the sign on the front door that says "Please stay out of our house", but it is never seen by the people who enter via the rear exit or climb in a window!

The most reliable method of excluding pages is to add the noindex meta tag as suggested by MagentoWebDeveloper and Alan.When a bot encounters the noindex meta tag it will send a signal to the search engine to de-index the page and there is no further problem.

I would generally use noindex, follow rather than noindex, nofollow as the nofollow tag will stop the flow of link value through your site. In most cases, as long as the noindex is in place, there is no reason to be worried about the links on the pages being followed.

You should NEVER use both methods at the same time.

Hope that helps,

Sha

Hi Paul,

Great answers from Ryan and EGOL.

The other thing to keep in mind is that traffic doesn't automatically equal business.

While it may seem attractive to be "free from Google" and other search engines, there is an important consideration that must not be forgotten.

If the Lion's share of your traffic is perhaps from RSS subscribers or referrals from more general sites, how many of those visits are actually delivering you people who need a lawyer?

When I need a lawyer for myself or a family member (when my need is immediate and I am ready to contact the one that seems best able to help me), what am I going to do?

Will I:

• Look for a blog to subscribe to?
• Surf around and read other websites looking for links to lawyers that might help me?
• Go to my favorite search engine and search on a term that will take me straight to a list of local law firms in that practice area?

While diversity of traffic sources is important, the reality is that if they were not the most useful means of finding what you want on the web, search engines would not still hold the place they do.

I have a number of clients who are lawyers and would agree that the approach Ryan has described for you is the best way to go. In addition to that I would say that the key in developing content for them has always been remembering the mindset of the client. A person who needs a lawyer is a person who needs help. The practice area and seriousness of the situation will dictate what type of help.

Hope that helps,

Sha

Hi Rafael,

On the subject of updating content, there are a couple of basic things you could think about that will help send a signal to the search engines that tells them the content on your site is kept current and therefore more likely to be relevant.

• Take a look at your existing pages and see if any of the content can be updated. There is an old blog post from Rand which talks about this and is still very relevant

Headsmacking Tip #11 - Refresh Legacy Content for a Rankings Boost

Paddy Moogan also talks about this strategy in a post from earlier this year

Actionable Link Building Strategies

• The second option is to look for opportunities to build new content (which can serve the dual purpose of helping to build more of those back links you want)

On the subject of building more links, yes, that will help improve your PA & DA, but be clever and thoughtful about the way that you do this. Don't just go for every link that crosses your path - work to get better quality links rather than just going for quantity.

Hope that helps,

Sha

Hi Volkinator,

I would actually offer a slightly different perspective on this.

I agree that there doesn't appear to be any real evidence that it helps rankings, but...

Think about what works for the visitor ... clever use of bold text can actually help conversions, but you have to think carefully about what you are doing because it can also hurt them terribly if misused.

First, highlighting just repetitive keyword terms or long sections of text is likely to hurt in 2 ways:

1. It is likely to annoy people and therefore increase the bounce rate for the page.
2. long passages of bold text reduce comprehension dramatically - if people don't understand the text on the page they are less likely to convert.

However, you can highlight just short phrases in the text to send a direct message to the reader. This will drive your message home more effectively and increase conversions.

This is an artificially constructed version to demonstrate what I am talking about, but in essence, the highlighted phrases when read alone create a shorter, more direct action statement from within the two longer sentences.

I have found this to be quite effective in improving conversions on client sites. One site in particular went from a conversion rate of 2.2% to a conversion rate of 26% after making only this change.

Hope that helps,

Sha

Hmmm...

Hi echo 1,

My view of Press Releases is quite different. The sole aim of a press release is to provide writers/bloggers/journalists with the key information they need to research and write stories or articles of their own which will promote your site (or your business, products & services).

A Press Release that is likely to be successful in achieving its aim is by necessity quite structured, following a pre-determined format and containing particular facts and information ordered quite specifically to make the prospective writer's job easier. So, by its very nature, a good Press Release is not what I would consider suitable page content for my site. For this reason I would never consider adding releases to my on site content.

Of course, the other natural conclusion from this is that if my Press Release reads like great page content for my site, then it does not follow the basic principles of a well constructed release and should be looked at closely before it is submitted.

If you are submitting to the major online Release sites, they all have resources which will basically teach you the format that you should use for your releases.

Hope that helps,

Sha

Hi HashtagHustler,

Great advice from WilliamKammer

No single tool will give you a complete look at your link profile - especially not Google Webmaster Tools!

If you are serious about auditing and dealing with unnatural and/or undesirable links in your backlink profile you should be gathering link data from as many sources as possible. While the data returned from different tools will overlap, there will always be links surfaced in every tool that you do not see in others.

Every tool that provides link data is crawling with different primary aims - some crawl more broadly, some more deeply, each with different preferences and limitations - the result being that the more tools you use to gather link data the more complete look you get at the profile.

As for Google Webmaster tools data, there are some specific limitations to be aware of:

1. Google provides only a "sample" of the links they are aware of in downloaded link data
2. Each link data download from your Google Webmaster Tools account is a unique sample. This means that you can actually get different data in the list if you download at different times!

In addition to the tool William mentioned above, these are some of the most commonly used tools for gathering backlink data:

Bing Webmaster Tools (they actually give you all of the link data they have for your site, although they generally see less than others.)
Majestic SEOahrefs
SEMrush

To get better data from Google Webmaster Tools you can use Tony McCreath's SWAT Google Webmaster Tools Exporter tool.

If you are doing pre-emptive disavow work and likely to need to update your disavow list from time to time, you can use our free tool disdit. This enables you to upload your first disavow list and then just update new lists to have them appended to the existing list. Download the updated list ready to upload to Google.
Why? Because the Disavow Links Tool upload is an overwrite, not an update. This means that if you just upload a new list to Google without appending to the existing list, you will have reavowed all of the domains/links that were previously disavowed!

Hope that helps,

Sha

Hi Doug,

301's are the right way to go as they pass link juice from the old domain and will preserve the traffic from existing external links.

The key here is "301's" (plural). Redirect each page on the old domain to the corresponding or relevant page on the new domain and also notify in Webmaster Tools that you have moved the site to the new domain.

There is some discussion of moving sites and using 301's in this SEOmoz Q&A post where I added a link to a Google Webmasters video that explains the process.

Hi Robert,

I think I've picked up on all of the questions here (there's a lot going on!) and have borrowed some awesomeness from my Tech Wizard (Boss) to fill in the exciting bits, so here goes:

I'll start with the easy one first... well actually, none of them are that hard

As a part of our ongoing SEO we check page load speed for our clients. A few days ago a client who has their site hosted by the same company was running very slow (about 8 seconds to load without cache). We ran every check we could and could not find a reason on our end. The client called the host and were told they needed to be on some other type of server (with the host) at a fee increase of roughly $10 per month.

OK, basically the answer to this one would be that your client's site was being throttled back by the host because it was using more bandwidth than was allowed under their existing plan. By moving them to the next plan (the extra $10 per month) the problem is resolved and the site speed returns to normal. Throttling it back gets the client to call... 8(

OK, 1 down and 2 to go...

About 4 months ago we realized we had a group of sites down thanks to monitoring alerts and checked it out. All were on the same IP address and the sites on the other IP address were still up and functioning well. When we contacted the support at first we were stonewalled, but eventually they said there was a problem and it was resolved within about 2 hours. Up until recently we had no problems.

and also

Yesterday, we noticed one group of sites on our server was down and, again, it was one IP address with about 8 sites on it.

OK, you know already that there can be up to 8 IPs on a box and at times something in the network will go bad. There are some variables here as to what is wrong. If you are on a Class C Network and one IP goes down then it means that the Switch or Router has gone bad (whether it is a Switch or a Router is determined by how the host has their hardware set up). If you are on a Class D Network and one IP goes down, then the problem is one of 3 things, the Card, the port, or the cable connecting the two, related to that IP.

The trick is that the person on the phone needs to realise what they are dealing with and escalate it to get the hardware issue resolved (A recent interaction with that particular host for one of our clients indicated to me that the realization part might be a little hit and miss, so good to have an understanding of what might be happening if it happens again)

Phew! Nearly there, last of all...

**On chat with support, they kept saying it was our ISP. (We speed tested on multiple computers and were 22MB down and 9MB up +/-2MB). We ran a trace on the IP address and it went through without a problem on three occassions over about ten minutes. After about 30 minutes the sites were back up. **

Here's the twist: we had a couple of people in the building who were on other ISP's try and the sites came up and loaded on their machines. Does anyone have any idea as to what the issue is?

OK this one is all about DNS caching. That particular host (the one that likes lady racing drivers) has a fail-over system in place. This means that if an IP goes down, the domains on that IP will automatically fail-over to another box.

So, if you have looked at those domains on your machine, it will be cached. When you go back to check the site you are still looking at the cached version. The other people in the building are coming to the domain fresh and through a different ISP, so they see those domains because they are back up on the new box. 

When the host reps were telling you that it was your ISP, what they really meant was that it had failed-over to a new box and you were still seeing the cached DNS location.

OK, think I covered it all so....that's all Folks!

Have a great holiday weekend!

Sha

Hi Jampaper,

Just to preface, I spend my days wading through the unnatural links sewer looking at the mess people have gotten themselves into because they thought they were smarter than Google or had that "how would Google ever know" thought in their heads.

EGOL is spot on with his response.

The criteria for undesirable links is not "how would Google ever know it's unnatural?", but "is it unnatural?"

On the "How", here are some things to consider:

1. Google's reach and ability to mine and interpret data (accurately or not) is so far outside our comprehension that it is probably better we don't even think about it.

2. Reviewers have a habit of unitentionally sharing information or creating patterns in the way they do things that are a clear red flag for orchestrated reviews

3. "These reviews always point to inner pages" ...Ooops! There's a pattern

4. "We're obviously targeting authoritative sites which do do reviews" ...Ooops! another pattern

5. Unnatural links on "Authoritative sites" would be more likely to enrage me if I were a member of the Webspam team than those on less influential sites. Let's face it, nobody ever sent me an email suggesting they could sell me links on a crap site

6. (and this you should take as very tongue in cheek, but perhaps give some thought to implications)
   This site has upwards of 400,000 community members. One of them is a guy who is currently on leave from his job at G, but occasionally comments on Moz blog posts that interest him (that's the tongue in cheek part as while it is possible, I seriously doubt he or any of the other Googlers who might be members spend time combing through this site looking for extra work!)

   However, it doesn't take much imagination to think there may be other people out there who could be made aware and if they were a certain kind of person might be likely to look into a backlink profile and perhaps lodge a report. Once the manual review process comes into play, the cleverness of the algorithm is irrelevant.

When you have a great product your customers will always be your best sales force! Do things that make THEM want to tell people how THEY feel about you. If you do that enough, even those Authoritative sites will be checking you out for themselves and gifting you natural links

Hope that helps,

Sha

Hello again,

First, I have to say that I think part of the problem here may be expectation - as Dunamis already mentioned, 2000 uniques a day is not bad for a site of this age.

After taking a look at your site there are a few things I see that may be negatively impacting your search traffic and visitor behavior:

• The first thing I see that would be of concern to me is the "Link Exchange" page. Using this term along with the text on the page that describes the trading of reciprocal links is sending a signal to search engines that this site is openly engaging in link trading.

  Search engines (Google in particular) can see this as a manipulative practice aimed at artificially improving rankings. The ongoing message from search engines is that they want to see linking based upon the real worth of a site (the value and relevance of its content). I would remove the "Link Exchange" page and work on finding ways to provide reasons for other sites to naturally want to link to your site. The added bonus here is that a natural, relevant, one-way link is much more valuable to your site than a list of unnatural looking irrelevant links. Russ Virante wrote a nice blog post on the link profile and what would be expected from a good quality site that is achieving links based on its value and relevance The Wikipedia Model.

  There are a ton of great posts and Q&A answers here on SEOmoz that will give you a never ending source of ideas and tips that can help you to improve links to your site. I ran a quick site search for link building and came up with an awesome reading list. Some of the ideas I've found in these are just plain FUN too!

• When I looked at your home page I felt that the amount of information there has actually tipped over the line into being a little overwhelming. While I was blown away by the recycled tire sculptures, I'm kinda glad they were there to grab my attention, because I was starting to feel a little like I was drowning in a sea of text, ads and well, empty ad spaces

  I would look carefully at how many articles you keep on the home page all at once, try to ensure that the images loaded for the posts are as eye catching as possible (like the tire sculptures) and make sure that your home page does not become one big category page rather than the home of great content that provides a "hook" to draw your visitor further into the site.

• Although you said that the intent of the site is to "be a leader in Tech Gadget niche" the message your pages are sending (both to a human like me and to search engines) is that your real intent is to earn money from advertising. So, there are a few things to consider:

  If you really don't want to earn from advertising - Do you really need all those ad spaces on your site?

  If in fact you are about earning money from the advertising - what is the best way to make that happen?

  Given that you have 2000 unique visitors a day - Why are you not already earning enough dollars?

  Since the last answer would be that they are not converting - How can you convert those daily visitors to sales?

  Personally, all those blank "Advertise Here" banners seem like a waste of space on your page and as a user seem to say to me that selling advertising is more of a priority than providing me with good information. I would definitely reduce the number of those. After all, you can place one or two of them and when you have sold those spots, add a couple in other positions.

• Lastly, when I went to the Tire Sculpture page (god I want one of those!) the floating cell phone ads seemed to be really slowing the page down (Latest version of Firefox). There are two issues here. As a user, if it had been anything else I was trying to look at other than those awesome sculptures, I would have left the page quickly. Second, Search Engines know this and use page load speed as an important signal for rankings. Google has made it clear that they are very sensitive to this, so anything that is negatively impacting your load speed will be doing the same to your ranking opportunities.

  The thing to keep in mind is that "cool" ads are not selling anything if nobody sticks around long enough to actually look at them.

OK, that's a fairly long list, but I think some things you can definitely take a look at and work on. Overall, if you are wanting to earn from the advertising on your site, you should be really happy that you have 2000 Unique visitors a day and focusing all of your energy on converting as many as possible from visitors to customers

As to "What am I doing Wrong?"

You've done an awesome job getting this far in such a short time! One of the things to keep in mind is that the world of search and online marketing is constantly changing. That in essence means that as long as you follow ethical practices, there is no wrong answer - just the solution that is acknowledged as best practice for today.

Pay attention to what is best practice, listen to the important people in the search industry that work hard to help us discover and follow it, then just take it one step at a time and enjoy the ride!

Hope this helps,

Sha

Hi Jason,

There is obviously something going on with this that is affecting what some crawlers are seeing on your pages.

I ran the Screaming Frog Tool and it shows that the majority of your pages have empty Titles even though I can see that there are Titles loading in the browser.

On checking your code I see that you are using the pragma directive meta element , but it actually appears below the Title element in the code.

Example from your code:

<head>
<title>Are You Socially Awkward? | Branding Blog | The Bullettitle>
**<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />**

So I ran the page through the W3C Markup Validation Service  and it also indicates that it sees no character encoding declaration:

No Character encoding declared at document level

No character encoding information was found within the document, either in an HTML meta element or an XML declaration.

So, I believe the issue here may be related to the fact that the pragma directive should appear as close as possible to the top of the head element ie before the Title element.

The following is from the W3.org documentation on declaring character encoding. You will see that there is specific reference to the fact that the use of the pragma directive is required in the case of XHTML 1.x documents as yours is:

For XHTML syntax, you should, of course, have " />" after the content attribute, rather than just ">".

The encoding of the document is specified just after charset=. In this case the specified encoding is the Unicode encoding, UTF-8.

The pragma directive should be used for pages written in HTML 4.01. It should also be used for XHTML 1.x documents served as HTML, since the HTML parser will not pick up encoding information from the XML declaration.

In HTML5 you can either use this approach for declaring the encoding, or the newly specified meta charset attribute, but not both in the same page. The encoding declaration should also fit within the first 1024 bytes of the document, so you should generally put it immediately after the opening tag of the head element.

Hope that helps,

Sha

Hi Niners52,

Rather than rely on an external tool to have found all of the URLs in a crawl and report them accurately, I would run a direct crawl using Screaming Frog.

There is a post on their blog which provides step by step instructions on how to use the tool to crawl a specific list of URLs.

You can only do this with the licensed version of Screaming Frog, but if you are not already using it, the cost is only 99 British pounds per year, and it is crazy useful for a thousand other things. A little more value for money than just subscribing to a more specific tool for just one month.

This way you will actually have a crawl result, not just a read from another tool.

Hope that helps,

Sha

OK Robert,

First I'm going to tip my hat to Ryan, who has perfectly explained the fact that some of what you see in your site: search can be because the 301's have not yet been recognized by the search engine.

Second, an apology to Alan as I went right to the LAMP solution because of prior knowledge from a previous thread or two that you were going to be talking about .htaccess

Now...I will spell out a couple of things because I have a feeling that you are likely to come across them again in the future and quick recognition can often mean a lot of time saved.

So here goes.

When I first read your question, my little web developer antennae suddenly started twitching! When I hear that there are multiple versions of a file with different file names deployed on a server I generally suspect one of two things:

• The site has been developed from a standard Template package, or
• There has just been a little "untidiness" taking place in the development process.

In your example, the /contact.php was the original file deployed live to the server, then the /contact-us.php file was created to replace it (presumably for SEO purposes - debatable, but that is a whole other conversation). As I'm sure you can imagine, /contact is pretty common in template packages, although the biggest template producer out there is much easier to spot, as the pages in their templates are always in the format /index-1.htm etc. It may just be that the developer creates their own standard template from an original design and rather than pre-planning and creating the file names to maximize SEO, they create standard page names and change them later.

While there is nothing really wrong with either of these things (unless you are charging the client for an original design and buying a pre-designed template at a fraction of the cost), both methods do open up the way for mistakes and errors to occur. As a result, there are a few things to keep in mind if you are working this way -

1. It is a much better idea to build on a development server so that none of the files that will become obsolete during the process will be indexed by search engines in the meantime. Tidy architecture, remove the obsolete files, test, then push to production.
2. When changing file names it is ALWAYS better to re-name the existing file and do a global update of links rather than create a duplicate with a different name. As soon as you create two files, you open up the possibility of accidentally linking both files within the site. You could have /contact.php linked from the home page and contact-us.php linked from the footer for example. There is a danger here that should you decide to delete the unwanted file, you create broken links without knowing it, or you have duplicate content. Either way, you have to recognize the problem and either fix it, or put a 301 in place to catch it.
3. NEVER hard code your links, because as soon as you change the name of the directory you placed your files in, you create a broken link! If you use relative links, the change of directory name will not matter.

I can see from Screaming Frog that some of the URL's for the pdf files have 301's in place, but it appears that the Redirect URL may also be hard coded to the /pdfs directory.  The fact that they all return a 404 when the directory name is changed to match that section makes it purely a guess as to what is happening here. It seems both www and non www pdf's are returning 404's in the browser.

The picture is muddied a little by the fact that there appear to be internal URL rewrites in the mix as well (to produce those pretty URL's with trailing slashes). So, there are a few options as to why the pdf's are not accessible:

1. They are not actually on the server at all (unlikely)
2. The names of the pdf's themselves have been changed, so even if the URL rewrite is sending the request to the new directory, the file requested does not exist.
3. The /pdfs directory has been named something completely different and the hard coding is the problem
4. The /pdfs directory has been moved to another location within the site architecture

I tried guessing a couple dozen of the obvious options, but no luck I'm afraid

There is one other possibility, in that the internal URL rewrites and 301 redirects could be creating a problem for each other. I am not clever enough to identify whether this is the case without a hint from the code, but will ask the God of All Things Code (my Boss) if he can answer that for me when daytime arrives 8D

OK....this is now so long that I really need to read the whole thread back to see if I have forgotten anything! If I find something I have missed, or can find anything else when help arrives, I'll be back!

Hope it makes some sort of sense and ultimately helps,

Sha

Hi Zsolt,

Did you actually look at the link I gave you with the code that you needed?.

The answer is to stop trying to use Redirectmatch and use the code I gave you in that example:

RewriteEngine on

RewriteBase /score

RewriteCond %{QUERY_STRING} score=
RewriteRule ^/*$ newlocation.html? [R=301,L]

This is the simplest, cleanest, and most reliable solution to your problem.

Sha

Hi Matthew,

When you are the originator of the content, the most important thing is to be able to show the search engines that you are the author.

There are some people who are producing great content and actually being outranked with it by sites who have "borrowed" it.

If you implement the rel=author meta tag for all of your content, then you are claiming it as your own. Do this when you create it and even if people use it they will be "outed" with the engines.

John Doherty posted recently in the SEOmoz Blog on Social Network Spam and Author/Agent Rank

Hope that helps,

Sha

Hi Greg,

Awesome information there from Ryan!

Implementing the authorship markup is important in that it basically "outs" anyone who has already stolen your content by telling Google that they are not the original author. With authorship markup properly implemented, it really doesn't matter how many duplicates there are out there, Google will always see those sites as imposters, since no-one else has the ability to verify their authorship with a link back from your Google profile

It is possible to block scrapers from your server (blacklist) using IP address or User Agent if you are able to identify them. Identification is not very difficult if you have access to server logs as there will be a number of clues in the log data. These include excessive hits, bandwidth used, requests for java and css files and high numbers of 401(unauthorized) and 403 (forbidden) HTTP error codes.

Some scrapers are also easily identifiable by User Agent (name). Once the IP address or user agent is known, instructions can be given to the server to block it and if you wish, to serve content which will identify the site as having been scraped.

If you are not able to specifically identify the bot(s) responsible, it is also possible to use alternatives like whitelisting bots that you know are OK. This needs to be handled carefully as ommissions from the whitelist could mean that you have actually banned bots that you want to crawl the site.

If using a LAMP setup (Apache server), then instructions are added to the .htaccess file using PHP. For a Windows server, you use a database or text file with filesystemobject to redirect them to a dead end page. Ours is a LAMP Shop, so I am much more familiar with the .htaccess method.

If using .htaccess you have the choice of returning a 403 FORBIDDEN HTTP error, or using the bot-response.php script to serve an image which identifies the site as scraped

If using bot-response.php, the gif image should be made large enough to break the layout in the scraped site if they serve the content somewhere else. Usually a very large gif that reads something like: "Content on this page has been scraped from yoursite.com. If you are the webmaster please stop trying to steal our content".will do the job.

There is one VERY BIG note of caution if you are thinking of blocking bots from your server. You really need to be an experienced tech to do this. It is NOT something that should be attempted if you don't understand exactly what you are doing and what precautions need to be taken beforehand. There are two major things to consider:

1. You can accidentally block the bots that you want to crawl your site. (Major search engines use many different crawlers to do different jobs. They do not always appear as googlebot, slurp etc)
2. It is possible for people to create fake bots that appear to be legitimate. If you don't identify these you will not solve the scraping problem.

The authenticity of bots can be verified using Roundtrip DNS Lookups and WhoIs Lookups to check the originating domain and IP address range.

It is possible to add a disallow statement for "bad bots" to your robots.txt file, but scrapers will generally ignore robots.txt by default, so this method is not recommended.

Phew! Think that's everything covered.

Hope it helps,

Sha

Hi Chammy,

With no other information on how he came to "lose" it, I would say his only hope is if he still has access to the Google Web Master Tools account, but he would have to act NOW, in the hope that the new domain owner is slow in getting things organized.

The process:

1. Purchase a new domain
2. Upload all content and ensure that site is ready to go as soon as new domain resolves
3. In Google Webmaster Tools, create a Site account for the new domain & verify it
4. In your old Webmaster Tools Site account, notify Google that the site has moved to new domain
5. Add a few pages of completely new content to the site on the new domain
6. In the Webmaster Tools Site account for the new domain, use "Fetch as Googlebot" to fetch each of the new pages and up to the maximum 50 most important pages in the site. (If a page is successfully fetched with this feature, you can manually add it to the index.)
7. Find all existing backlinks possible (using Open Site Explorer, Majestic SEO, link: search operator in Google, Bing & Yahoo) and contact as many of those webmasters as possible, advise of the domain change and ask them to correct their links.

As Alan pointed out, the only way that this will work for him is if the new domain owner is slow in setting up a GWMT account and he still has access to his existing account, so basically, he needs to act immediately and pray a lot

Incidentally, if he has already missed out on the opportunity to tell Google about the change of domain he should still follow the steps above for the new domain and try to get the links corrected if he can to get some quick traction where possible.

Hope that helps,

Sha

Hi Julian,

Since it is competitive analysis you are looking for, you can use the SEOmoz Keyword Difficulty Tool to check your keywords in Google Brasil.

When you add a keyword (up to 5 at a time) there are two selectors to the right of the input screen. The first allows you to choose a search engine. It is set to Google by default. The second selector enables you to choose your preferred Country. Just choose Brazil from the list and then run your analysis.

Rand wrote a comprehensive blog post on the recent update to the tool - The Best Kept Secret in the SEOmoz Toolset.

So far as I know it is not possible to choose a region specific search engine for Keyword Rankings in your campaigns, but if you want to check this for sure you could email the SEOmoz Help Team at help [at] seomoz.org.

Hope that helps,

Sha

Hi Dominic,

If you want pages to be removed from the index and ensure that they are not crawled again, you do exactly what you did with the others EXCEPT for one detail - the robots meta tag that you use should be "noindex, follow.

If you use "nofollow" you are trapping all incoming link value on that page and preventing it from flowing internally through your site.

Yes, you can add robots meta tags using the Yoast SEO plugin. You will find the features you are looking for in the "Indexation" section. Read the support documentation and on-page instructions carefully and you should be fine.

Hope that helps,

Sha

Hi threebiz,

First, welcome to the community! Look forward to seeing you around in the future

Since there is a small loss of "link juice" when you apply a 301 redirect, you should pay attention to which domain you choose to redirect and which you keep as the primary domain.

Your choice may be influenced by suitability for branding, but if there is no real branding advantage in using one or the other, then I would make sure that I redirect the site with the lesser Domain Authority to the one with the greater.

You can use Open Site Explorer to check Domain Authority (DA) for both domains. You can learn more about DA under Metrics, Link Counts and Data Points.

If for example, you have Site A with a DA of 30 and Site B with a DA of 20, then I would redirect Site B to Site A, since this allows you to maintain the full domain strength of the site with the greater prospect of ranking.

Hope that helps,

Sha

Hi Christopher,

Yes, I'm having the same issue with emailed comments from the blog.

Three guesses as to what might be causing it:

1. Images
2. javascript objects
3. the email tracking software being used

The best thing to do is go to the Help hub and lodge a ticket with as much explanatory detail as you can give. I know Mozzers are already working on it, but the more specific information they have, the easier it will be to fix it.

In the meantime, I hope it helps to know you're definitely not alone

Sha

Since the intention of alt text is to help the user understand what the image is about, it should use natural explanations which utilize a range of variations on your keyphrase(s).

Keep it natural and keep the user in mind.

Hope that helps,

Sha

Hi Mike,

The only way to limit the content included in a campaign is to choose "subdomain" when you set up the campaign. So, the only way to limit the campaign to just one page, would be to create a separate subdomain with just that one page in it, but I am curious as to why you would want to do this anyway.

The errors highlighted by crawl diagnostics are largely easy to identify (or irrelevant) in a single page.

If the page you have been working on is live, then you can check it using the On-page Report Card for the specific page.

To do this:

Once inside your Campaign, choose On-page and then the blue Report Card link at the top of the page.

Select the primary keyword from your list (any loaded in your campaign) and type in the URL of the page, then generate your report.

While the Report Card is primarily about optimization for the chosen keyword, it will also highlight things like URL & Title length, number of internal and external links, meta description etc

Hope that helps,

Sha

Hi James,

First, you don't need to delete your campaigns.

You can archive a campaign that you are not wanting to actively check. If you should wish to make it active again, you will need an empty campaign slot, so can archive something else that you don't need and activate the old campaign again.

To Archive, just go to the campaign in question, click the blue Campaign Settings link at the top. Scroll right to the bottom of the page and click the Archive button (next to delete). To make an archived campaign active again, go to Campaigns, choose Archived Campaigns, then click Activate for the campaign you wish to restore.

On your other question

I think the Social analysis you are referring to is that included in the Open Site Explorer tool. When you are logged into your SEOmoz Pro Account and use Open Site Explorer, you can see social data for your site. There is a button in the Open Site Explorer interface (right below where you type your URL) that allows you to load additional sites for comparison. The information that appears in the Competitive Analysis section of your campaign is drawn from Open Site Explorer, but the tool itself provides much more powerful options for analysis.

Check out all of the different Filters and the Advanced Reports section while you are there.

Hope that helps,

Sha

Hi Stephanie,

Sounds like a situation where you really need to go straight to the Help Team and get them to take a look at the specific domain to diagnose the issue.

I would start at the Help Hub and click the big pink button.

Hope that helps,

Sha

Hi,

The SEOmoz Term Extractor Tool will pull the top 10 optimized one, two and three word terms from any requested page. If you put the extracted terms into the On-page Keyword Optimization Tool, you will see recommendations about keyword frequency on the page.

I should point out that this is NOT a "keyword density tool" as such. Rather, it tells you what could be improved about a range of on-page elements, including how many times the term appears on the page. Most in the industry have moved away from using specific keyword density values for optimization these days

If you think it would be good to have a tool which combines the two, you could always make a suggestion to SEOmoz by lodging a feature request.

The Feature Update List  details changes that have been made as well as those that are on the board for the future. If you scroll down to where you  see Roger Mozbot on the page, click Request a Feature and suggest away!

Hope that helps,

Sha

Locate and correct 83 outdated and incorrect or mismatched citation sources, ensuring that all of them now use the same exact Business Name as used on the Google Places profile and all profiles are claimed and completed.

Hi Tom,

I would never throw away traffic unless it came from a "questionable" source.

If the reason for the domain switch is because the domain more accurately reflects the client's business, then the existing traffic is also likely to be relevant. This gets you "one foot in the door" in terms of conversion.

If you still have the existing site loaded, then you are in a good position, because you already know the URL and content of the existing pages.

My recommendation to the client would be to do the work to salvage the existing traffic for the domain. How well you do the job will depend on how large the existing site is, how much traffic it has and how much work the client is willing to pay for,

The process would be to add 301 redirects for all existing URL's on the new domain (that will not exist once the new content is added) in addition to 301's for all of the pages on the domain that you are transferring from. The best way to do this is to ensure that each is redirected specifically to a page containing content that is relevant. Obviously, if the old site is large and/or the budget is small, this will be more work than you are getting paid for. In that case, you have the option of redirecting groups of pages to a relevant category page, or in the easiest, but least advisable, just redirect them all to the home page.

If the existing traffic to the new domain is insignificant, then you shouldn't lose too much sleep over just redirecting them all to the home page, but if there are some visitors there, I would explain to the client that sending them to a relevant page gives him a chance of converting them.

If you choose not to redirect any of the pages you are just setting up a situation where you constantly play "catch up' on the 404's - you cannot know how long it will really take for every link in existence to be clicked by someone. Don't forget that the biggest link source will be indexed pages in search engines. These don't always get de-indexed in the time frame we expect, so there is no correct time period within which links will just go away.

I wrote a blog post on this recently after doing a "salvage' job for a client whose designer installed a new site version and killed off all his old pages without redirecting anything. While your situation is different, It talks about redirecting for relevance, so might be worth a read.

Hope this helps

Sha

Hi there,

Unfortunately products that go out of stock are one of the biggest frustrations for ecommerce site managers. Where products are sourced from another company you can be very much at the mercy of the way that they manage their inventory. This means you need to decide which method of dealing with expired products will work best for you.

From what you have described it sounds like the product was marked temporary out of stock by the supplier, then more recently changed to "Do Not Reorder" status. So, the product has permanently expired and will not be restocked at any time in the future.

Your best option in this situation is a 301 redirect, which will pass link value from the page to whichever page you decide to redirect to. This is where you need to make decisions.

You have essentially 3 choices when choosing an appropriate page to set your redirect target.

1. If you have a product in your inventory that is very similar, you could redirect to that as the product may be of interest to people who were looking for the original product.
2. If you do not have a similar product, then you can redirect to the higher level category page for that item.
   For example, redirect to http://www.vistastores.com/lighting
3. Finally, since you have an internal search mechanism, you could choose to redirect to a search for a relevant term.
   For example, redirect to search ?q=lighting

The aim is to always send the visitor to the most relevant or helpful page to improve the chance of them converting.

Hope that helps,

Sha

Hi Martin,

You might find it useful to take a look at the Linkscape Update Schedule in case timing is a factor.

I believe Rand outlined the recent changes to the indexing rationale in this webinar

Using Open Site Explorer to Uncover New Marketing Opportunities , but if you still have questions then as Brian suggested, it may be a good idea to lodge a ticket or email the Help Team. help [at] seomoz.org.

Hope that helps,

Sha

Hi Lee,

There was a file handling issue with the changeover of SEOmoz crawlers month before last. I know that the SEOmoz Engineering team are aware of the problem and had thought that a fix was on the way.

This particular site seems to be one that has slipped through the net.

The best thing to do is to email the SEOmoz Help Team help [at] seomoz.org and give them as much detail about the particular OSE search you are looking up.

Hope that helps

Sha

Hi Jesse,

Basically, the only significant search volume is for the shorter two word term "minneapolis massage", so there is no benefit in going for the longer term (which also has a negative correlation for rankings).

So, since Ryan advised that the .net version of minneapolismassage is available, if you are wanting an exact match domain that might be a better option.

Since the return you are wanting is quite modest, then the traffic afforded by that term and the little extra you might attract from other terms should work for you if you can get some reasonable rankings.

I would suggest that you put some good effort into local search optimization if you decide to go with that domain.

Hope that is clearer,

Sha

Hi Dylan,

All evidence is that if the URL is not wrapped in an <a href="">tag, then it won't be crawled.</a>

<a href=""></a>

<a href="">Matt Cutts has discussed this and not surprisingly suggested that it would not be a good thing if these URL's were crawlable. If that happened it would be open slather on adding your link to any user generated content such as forums, comments etc. You can view the video here</a> Will Google use non-link references?

There is a post from SEOmofo that suggests in certain circumstances, google may accidentally make links crawlable if it strips out <text area="">code, but once again, the URL's in his demonstration are actually wrapped in the a href tag. You can find the post here</text>

So, as far as I am aware, there is no evidence that URL's wrapped in a SPAN tag would be crawled - but as you will see, Matt emphasises that Google doesn't like to ever take anything "off the table".

Hope that helps,

Sha

Hi Will,

The advice already given here is solid.

In addition to that, a little personal experience I will share.

I had a client who accepted the offer you have received when the service first came out a couple of years or so ago. They told me that they saw a significant loss of indexed pages and traffic to their site within a short time.

I cannot say that what happened to their site was a direct result of using the service as I was not in control of the site at that time. However, I do know that once they decided they wanted to remove the service it took me almost 8 months to get the company to do so!

Hope that helps,

Sha

Hi Z,

There are basically 5 things that make PRO worthwhile for me

• Outstanding Tools that save me time and money, allowing me to give better service to my clients
• Online Webinars from industry experts who can quite literally rock my world with a single idea at times
• This Q&A, which provides access to other members, but also the ability to ask a private question and have SEOmoz staff and Associates help me nail a problem.
• Top notch customer service if I ever need help with something.
• The opportunity to be included in the development of new tools, analysis of new hypotheses on search etc combined with the knowledge that the people pursuing them are authentic thought leaders in this industry.

That about says it I guess.

Hope that helps,

Sha

Hi Terrie,

Don't feel bad about asking the question at all - this is a tough topic to get your head around!

In fact, rather than butcher an explanation and confuse you more, I would recommend you read this excellent post from Rand How to Solve Keyword Cannibalization. The post is from 2007, but still the best and simplest explanation of how to fix the problem that I have seen so far. I would suggest you read the comments as well - Rand has dropped a whole lot more excellent information in his answers.

The key to seeing the difference here is not just about the keywords, but the anchor text and which pages the links are pointed to.

Once you follow the structure illustrated in Rand's post, it should become much easier to understand, and also much easier to spot the problem when you look at the pages on your site.

Hope that helps,

Sha

Hi getzen,

The primary problem here is that these links are resulting in 404's which can hurt your site.

If they are just not terribly relevant, but may produce traffic that is useful to you, then you can create a nice "Oops!" page to give visitors options to go to other pages on your site (we call this our "Bounce Page").  This is preferable to a "catch-all" redirect to your home page when you know that the likelihood of a high bounce rate is a concern, since Bounce is now known to be a ranking factor for major Search Engines.

If you are clever with the design of this page you can create ways to keep these non-relevant visitors on the page (rotating banners with funny or clever images, or maybe a video embed could be used).

The primary goal is to eliminate the 404 Errors, but to also be mindful of the impact of Bounce.

Hope that helps,

Sha

Hi John,

Excellent response here from Ryan as always, so not much more for me to say except that there are some other excellent resources here at SEOmoz which can help you to focus on the broad range of things that might be influencing your site visibility in the Search Engines.

Essentially, as Ryan indicated, it is about taking a "future proofing" approach to your site. If you haven't already found the Pro Webinars section of the site, there is an excellent Webinar from Doctor Pete on Future Proofing Your SEO: 2012 Edition. One extra thing to note here is that all SEOmoz Pro Webinars uploaded include the presenter's Powerpoint Slide Deck. Since SEOmoz presenters make a point of including links to useful resources etc in their decks, this is a huge extra help, so don't forget to grab it.

Also, one of my favorite tools for seeing exactly what is contributing to the top 10 Rankings in a niche is the Keyword Difficulty SERP Analysis Tool where you can run Advanced Reports to see how each of the Top 10 Ranked sites for a particular keyword term are doing against a whole range of key metrics. The bonus is that even if your site isn't in the Top 10, you can enter the URL to have it included in the comparison 8D.

Rand also provided a great post that explains how to use the tool: The Best Kept Secret in the SEOmoz Toolset.

Hope that helps,

Sha

Hi Peter,

I don't really see how this can be related to images as I have always had outlook set up not to download images.

After your response that it was related to images I edited the settings to ensure that there is absolutely no way outlook will ever try to download images, but emails from the blogs and Q&A still freeze Outlook for between 1 and 2 minutes before they resolve to show the actual text of the message...it is as if the actual text of the post itself is being called and queued.

As mentioned by Christopher above the problem does not occur with ranking reports, new crawl notifications, the Moz Top 10, Monthly Newsletter or general email from Mozzers.

At this point I can do nothing but avoid allowing those emails to open and can't even delete them without the same issue unless I am lucky enough to have them sandwiched between two other emails that I want to delete

I know it probably seems like a trivial issue if it isn't happening to everyone, but honestly, for those of us who are fairly active on the blogs and in Q&A, 1 - 2 minutes of having your mail program disabled for every email that comes from there gets out of control very quickly!

I'm sure I'm not alone when I say, Please HELP!

Sha

P.S. A heartfelt apology to all of you who just had your mailbox freeze up again because I added this post   hugs

Hi Ryan,

A lot of people have gotten much more worried about dates since they heard about the "Freshness Update" late last year. Unfortunately a lot of people assume that it is a factor for all keyword terms & niches, but that is not the case. It is quite easy to find out whether it is a factor for your keywords. I gave a detailed explanation of this in this Q&A thread in November.

As is mentioned in the Quora thread you quoted, there are much more reliable ways for search engines to determine freshness (timestamps & previous crawl data).

I would agree with Brent and EGOL that the significance of year to your user base makes it reasonable (more likely expected) to include the year. However, I would take it a step further and suggest that you consider leveraging the intelligence of the bots a little.

We know that  bots are now smart enough to help assess relevance. In fact it has become the centerpiece of their day to day work.  For me, that should mean that using words like "wine" or "vintage" would signal to the search engine that this URL and its content might reasonably include date references in the form of 4 digit and/or 2 digit year information

That decided, I would build my site infrastructure accordingly, placing individual pages within directories using a reasonable and natural naming structure that includes the appropriate words. Depending how you prefer to approach it, a couple of possible examples might be  grapesinyourtoesexample.com/07-vintage/2007-cellar-pod-viognier-adelaide-hills/ or grapesinyourtoesexample.com/red-wines/2007-cellar-pod-viognier-adelaide-hills/.

Hope that helps,

Sha

Hi Hanson,

I would recommend checking out the recent Whiteboard Friday from Cyrus on anchor text matching - Beyond Exact Match Anchor Text to Next Generation Link Signals

Hope that helps,

Sha