Welcome to the Q&A Forum

Hi Steven,

This message refers to a meta tag which appears in the head statement for the page.

the code looks like this:

OR

This tag tells crawlers not to index the page (or to remove the page from the index if it already appears there) and to either follow or not follow links on the page.

In most cases I would use noindex, follow as this allows any link juice being passed to the page to flow on through the site. The meta noindex tag is the most reliable way to ensure that pages are not indexed since it occurs at the page level. Other methods such as disallowing pages in a robots.txt file can be unreliable as crawlers that come to a page via an external link do not see the disallow statement, so will index the page.

The noindex tags on your pages may have been placed in order to exclude pages that were producing errors or are duplicates of other pages.

Hope that helps,

Sha

Hi Shradda,

I agree with Ryan that the use of a meta noindex tag is the preferable way to block the pages, but obviously there may be difficulties with applying the tag, depending upon how your pages are generated and whether you are able to alter the code or not.

You can also use ?option=com_fireboard etc to create 301 redirects back to a higher order category page or search.

You should be able to use a single line of code to 301 all pages within each directory.

Using 301 redirects will also send a signal to search engines to de-index those pages.

Very clever 404 page too! Had to watch him go all the way across the page and back just so I knew I wasn't missing anything!

Sha

Hi Alan,

Since the subdomains are captured within the root domain, all subdomains will always be seen within the root domain unless they are successfully blocked.

As I understand it, Google's view of the world is that every specific account created in Webmaster Tools is a separate site.

Hope that helps,

Sha

Wow! Awesome work from Ryan on this one

Your duplicate content problem is caused by the fact that the blog software generates a duplicate URL for each tag, archive, rss feed etc that is related to an individual post. This is a standard problem with common blogging software such as Wordpress and normally (if you were running a standard Wordpress Blog), the fix would be to install an SEO plugin which allows you to either turn off these things, or add code which will tell the search engine that they relate to the original post.

In this case though, your site is using the blog offered through your IDX/MLS account. In order to address the duplicate issues you will need to contact your IDX/MLS provider and ask them to deal with the problem. We have clients who use the IDX, but we have installed standard wordpress blogs for those as there are no self management options with the blog supplied by that provider.

The following pages are also returning 404 not found errors on your site:

http://www.tricitiesrealestateagent.com/agents/brandon-patton/

http://www.tricitiesrealestateagent.com/buying.php

http://www.tricitiesrealestateagent.com/Homes-for-Sale-in-Richland-Washington.php

http://www.tricitiesrealestateagent.com/Richland-Real-Estate.php

http://www.tricitiesrealestateagent.com/selling.php

http://www.tricitiesrealestateagent.com/thumbs/280x186/www.tricitiesrealestateagent.com%252Fimages%252Ffeatured%252F1298323189.jpg (this is an image URL)

If you have deleted them by accident, then you will need to go back to your last backup and restore them, otherwise you will need to create 301 redirects to send any traffic to relevant pages on the site.

In addition to following Ryan's suggestion of reading the Beginner's Guide to SEO, I would suggest taking some time to read through as many of the posts in the SEOmoz Blog as you can, especially watching the Whiteboard Friday videos. Most of those provide quick, easy to understand explanations of many basic concepts of SEO.

Hope that helps,

Sha

"There are none so blind as those who do not wish to see!"

I'm sure you know already that you dodged a bullet by losing that client Lee

What you currently see in the keyword difficulty tool is just a representation of how competitive the term is. The tool previously also showed local and global broad and exact match search volume for your term, but Google's recent decision to withdraw developer access for the Adwords API means they are no longer visible.

As mentioned by others here, the report generated below the competition result, which shows results on specific metrics for the Top 10 Domains ranking for the term (and the advanced Report you can generate if you want to) is by far the most valuable output from this tool.

There is a detailed blog post from Rand Best Kept Secret in the SEOmoz Toolset, which explains in detail how to use the information to see what you can improve to have a shot at matching it with those sites.

Hope that helps,

Sha

Hi Rick,

Your first thought was correct. If you apply the noindex meta tag to every page in the secure part of the site, then all of those pages will be de-indexed and you will have no duplicate content problem.

For Wordpress, you just need to install a plugin that allows you to edit and apply page elements and meta tags. My preference is Yoast SEO. If you do a plugin search from your dashboard you will find it.

Hope that helps,

Sha

Hi Rick,

If you wish to use the robots.txt method to disallow all or part of your site's https protocol, you simply need to load two separate robots.txt files.

The http and https protocols are basically viewed by bots as if they were two completely separate root domains (which I guess you already know as you have mentioned the fact that port 443 is used for the secure protocol).

Google's advice is that to use this method, you should have a separate robots.txt file for each protocol with code as follows:

For your http protocol (http://www.startuploans.org/robots.txt

User-agent: *
Allow: /

For the https protocol (https://www.startuploans.org/robots.txt

User-agent: *
Disallow: /

However, blocking crawlers with robots.txt is not the most reliable method for excluding pages from Search engines. The reason for this is that the page will continue to be indexed if it happens to be found via a link from another page. Basically, the robots.txt is the sign on the front door that says "Please stay out of our house", but it is never seen by the people who enter via the rear exit or climb in a window!

The most reliable method of excluding pages is to add the noindex meta tag as suggested by MagentoWebDeveloper and Alan.When a bot encounters the noindex meta tag it will send a signal to the search engine to de-index the page and there is no further problem.

I would generally use noindex, follow rather than noindex, nofollow as the nofollow tag will stop the flow of link value through your site. In most cases, as long as the noindex is in place, there is no reason to be worried about the links on the pages being followed.

You should NEVER use both methods at the same time.

Hope that helps,

Sha

Hi Micelleh,

Ryan has given you a great answer here.

There is also a really detailed post about getting into Google News from Cyrus Shepard in the SEOmoz Blog

Sha

Hi Robert,

Sounds like time to hit SEOmoz Feature Requests and suggest something

Sha

Hi Z,

There are basically 5 things that make PRO worthwhile for me

• Outstanding Tools that save me time and money, allowing me to give better service to my clients
• Online Webinars from industry experts who can quite literally rock my world with a single idea at times
• This Q&A, which provides access to other members, but also the ability to ask a private question and have SEOmoz staff and Associates help me nail a problem.
• Top notch customer service if I ever need help with something.
• The opportunity to be included in the development of new tools, analysis of new hypotheses on search etc combined with the knowledge that the people pursuing them are authentic thought leaders in this industry.

That about says it I guess.

Hope that helps,

Sha

Hi,

The accepted best practice is no more that 100 links on a page, however there are some things to consider.

For a more detailed explanation, you can take a look at my answer in this thread

Hope that helps,

Sha

Hi NYC,

I am guessing you are referring to the "Too many on-page Links" warning in the crawl report?

The accepted best practice is no more than 100 links on a page, but keep in mind that what you are seeing is a best practice recommendation, not an "error". So, if there is no reasonable way to reduce the number of links without lessening the user experience, then you may need to just accept that you can't comply with best practice in that case.

If, on the other hand, you see that there are unnecessary links on the page, you can reduce the number and this may even improve the experience for the user.

In general, if the links are predominantly from menus and necessary elements like buy buttons etc, then I would not be too worried.

If the page is full of internal text links trying to pass link juice with anchor text, then this would be something to fix. Two reasons:

1. It is not a good experience for the user as it makes the whole page look and feel really "spammy".
2. The amount of link juice passed by a link which is one of more than a hundred on the page is so miniscule that it is pointless anyway.

Hope that helps,

Sha

Awesome Carin!

Thanks for responding so quickly with a great explanation of what's happening.

Sha

Hi  Eric,

The best thing to do would be to email the SEOmoz Help Team direct so that they can take a look at the issue. The email is help [at] seomoz.org

Make sure that you give them as much information as possible, including the username that you log in with and the specific OSE search that you are trying to retrieve a report for.

Hope that helps,

Sha

Hi Lee,

As I understand it the problem is just a handling issue - the crawler is seeing downloadable files and recognizing them as regular links.

Sha

Hi Lee,

There was a file handling issue with the changeover of SEOmoz crawlers month before last. I know that the SEOmoz Engineering team are aware of the problem and had thought that a fix was on the way.

This particular site seems to be one that has slipped through the net.

The best thing to do is to email the SEOmoz Help Team help [at] seomoz.org and give them as much detail about the particular OSE search you are looking up.

Hope that helps

Sha

Hi Jay,

Australia is a little too far away to make the trip, but since Duane Forrester is tweeting about how nice North Carolina is, I'm guessing you already know where he's been the last couple of days

Sha

Hi Chris,

Do you own the site which generates the images?

In order to verify a site you need to create a separate Webmaster Tools account for it and then go through the normal WMT verification process. This requires you to have sufficient control of the site to upload a file or add a meta tag to the index page.

If you cannot do this then you will not be able to verify the site.

Sha

Wow! Thanks Ryan.

I'm sure it won't surprise you to know that I'm always reading eagerly when I see you respond to a question as well.

Hi Croozie,

Awesome work once again from Ryan!

Since your question feels like a request for suggestions on "how" to create a solution, just wanted to add the following.

When you say "classified listings" I hear "once off, here for a while, gone in 45 days content".

If that is the case, then no individual expired listing will ever be matched identically with another (unless it happens to be a complete duplicate of the original listing).

This would mean that it would certainly be relevant to send any expired listing to a higher order category page. If your site structure is such that you have a clear heirarchy, then this is very easy to do.

For example:

If your listing URL were something like http://www.mysite.com/listings/home/furniture/couches/couch-i-hate.php, then you can use URL rewrites to strip out the file name and 301 the listing to http://www.mysite.com/listings/home/furniture/couches/, which in most cases will offer a perfectly suitable alternative for the user.

There is another alternative you could consider if you have a search program built in - you could send the traffic to a relevant search. In the above example, mysite.com/search.php?s=couch.

Hope that helps,

Sha

Hi Ade,

My apologies on this one. My brain was a little addled after a long day of driving I think!

You are correct in that the "set and forget" solution we gave you to start with will be overwritten by the Joomla error handling after the .htaccess file has been read.

It is only possible to make standard 301's work using the .htaccess file if you were to manually create an individual rule for every deleted file using its specific joomla generated URL like this:

RewriteEngine On

RewriteRule  ^courses/highways/6-NRSWA/27-nrswa-operative-sept-11.html$ /courses/highways/6-NRSWA?course=not-available [R=301,L]

The problem with this is that if your courses are regularly deleted, it will not be long before you have a very large .htaccess file, which could foreseeably lead to processing issues.

So in fact, the only "set and forget" solution is the one that you have already put in place and from a practical standpoint, it is the best solution.

Well done!

Sha

Hi John,

Since your campaign is set up using the root domain, all subdomains will automatically be included.The only possible way that you may be able to remove the subdomain from the crawl would be to use robots.txt to specifically block mozbot from crawling it. However, if you do this, then it will not be possible to access the information for the subdomain separately.

So, what you are really asking is whether the subdomain can be split into a separate campaign without you having to start from scratch. I suspect the answer is no (certain things, like keywords may not be relevant anyway).

The only way to get a definitive answer on whether it can be done would be to email the SEOmoz help team direct - help [at] seomoz.org.

Hope that helps,

Sha

Hi Ade,

We have used .htaccess to create 301 redirects for Joomla sites in the past.

Can you email me your .htaccess file and the URL of your site and I will get our Chief Programmer to take a look for you.

My direct email is on my profile page (or you can private message me from your profile).

Sha

Hi Ade,

So sorry I wasn't around to follow this up for you. I have been away for the day and had wireless connection issues, so could not check Q&A until now.

Oops! Yes. Joomla does have its own error handling which does make a big difference, but it should be simple to fix once you understand what happens when you put the .htacces file in place.

When a request is received by the server, the .htaccess file is read from top to bottom, checking each rule in the file for a match. Once a match is found, the specific action assigned to that rule is executed. This means that no rules thereafter are read.

So, if you ensure that your code appears at the beginning of the .htaccess file, then whenever the conditions described by the rule are matched, the redirect will occur. However, if no other rule in the .htaccess is matched, then Joomla error handling will come into play should any other error be present.

This of course means that any specific rule you wish to add in the future should also appear before the Joomla code. As long as you always make sure it is last to be read, everything should work just as you intended.

Hope this helps,

Sha

Hi Ade,

Since learning this one thing will be something that you are likely to use over and over, I figure it is much better if you see how it actually works. So, we wrote a little resource to show you how to do a basic 301 redirect as well as one that goes back one level to your category page.

If you take a look at this simple 301 Redirect course for managing 404 errors, you can see three working pages and also download the code.

Let me know if you have any questions.

Hope that helps,

Sha

Hi again Robert,

God of All Things Code is away from the office for a while today, so will need to wait a little longer for his input.

A couple of things that happened since my last post though:

Those twitching antennae just wouldn't stop nudging me to look a little further as everything I see with this site is saying "template" to me. Add to that the URL rewrites which hide the actual URL's and the broken pdf files...so i went digging a little further and ... Aha!

Not a template, but a "Theme". The entire site is built in Wordpress!

Now, I am pretty sure that the broken pdf's are the result of the Wordpress URL rewrites changing the directory name in combination with the hard coded links. If this is the case, then it ought to be just a matter of adding a rule to the .htaccess file to deal specifically with the pdf's. The order in which the rules appear will determine whether the issue is resolved or not.

I'll let you know as soon as I've confirmed the specifics with my Boss.

Hope that helps,

Sha

OK Robert,

First I'm going to tip my hat to Ryan, who has perfectly explained the fact that some of what you see in your site: search can be because the 301's have not yet been recognized by the search engine.

Second, an apology to Alan as I went right to the LAMP solution because of prior knowledge from a previous thread or two that you were going to be talking about .htaccess

Now...I will spell out a couple of things because I have a feeling that you are likely to come across them again in the future and quick recognition can often mean a lot of time saved.

So here goes.

When I first read your question, my little web developer antennae suddenly started twitching! When I hear that there are multiple versions of a file with different file names deployed on a server I generally suspect one of two things:

• The site has been developed from a standard Template package, or
• There has just been a little "untidiness" taking place in the development process.

In your example, the /contact.php was the original file deployed live to the server, then the /contact-us.php file was created to replace it (presumably for SEO purposes - debatable, but that is a whole other conversation). As I'm sure you can imagine, /contact is pretty common in template packages, although the biggest template producer out there is much easier to spot, as the pages in their templates are always in the format /index-1.htm etc. It may just be that the developer creates their own standard template from an original design and rather than pre-planning and creating the file names to maximize SEO, they create standard page names and change them later.

While there is nothing really wrong with either of these things (unless you are charging the client for an original design and buying a pre-designed template at a fraction of the cost), both methods do open up the way for mistakes and errors to occur. As a result, there are a few things to keep in mind if you are working this way -

1. It is a much better idea to build on a development server so that none of the files that will become obsolete during the process will be indexed by search engines in the meantime. Tidy architecture, remove the obsolete files, test, then push to production.
2. When changing file names it is ALWAYS better to re-name the existing file and do a global update of links rather than create a duplicate with a different name. As soon as you create two files, you open up the possibility of accidentally linking both files within the site. You could have /contact.php linked from the home page and contact-us.php linked from the footer for example. There is a danger here that should you decide to delete the unwanted file, you create broken links without knowing it, or you have duplicate content. Either way, you have to recognize the problem and either fix it, or put a 301 in place to catch it.
3. NEVER hard code your links, because as soon as you change the name of the directory you placed your files in, you create a broken link! If you use relative links, the change of directory name will not matter.

I can see from Screaming Frog that some of the URL's for the pdf files have 301's in place, but it appears that the Redirect URL may also be hard coded to the /pdfs directory.  The fact that they all return a 404 when the directory name is changed to match that section makes it purely a guess as to what is happening here. It seems both www and non www pdf's are returning 404's in the browser.

The picture is muddied a little by the fact that there appear to be internal URL rewrites in the mix as well (to produce those pretty URL's with trailing slashes). So, there are a few options as to why the pdf's are not accessible:

1. They are not actually on the server at all (unlikely)
2. The names of the pdf's themselves have been changed, so even if the URL rewrite is sending the request to the new directory, the file requested does not exist.
3. The /pdfs directory has been named something completely different and the hard coding is the problem
4. The /pdfs directory has been moved to another location within the site architecture

I tried guessing a couple dozen of the obvious options, but no luck I'm afraid

There is one other possibility, in that the internal URL rewrites and 301 redirects could be creating a problem for each other. I am not clever enough to identify whether this is the case without a hint from the code, but will ask the God of All Things Code (my Boss) if he can answer that for me when daytime arrives 8D

OK....this is now so long that I really need to read the whole thread back to see if I have forgotten anything! If I find something I have missed, or can find anything else when help arrives, I'll be back!

Hope it makes some sort of sense and ultimately helps,

Sha

Hi Robert,

OK, just to clarify...

1. You want to check for sure that newclient.com is 301 redirected to www.newclient.com?
2. You want to check for sure that ALL URL's which have been individually 301'd are redirecting to www.newclient.com/filename?
3. You want to understand why the non www version of pdf files works and the other doesnt?

Right off the top, the definitive way to check whether there is a properly functioning redirect in place is to type the URL into a browser and see whether it resolves to the redirect target :). You can also run Screaming Frog and see what status the pages return, but be aware that this does not always reflect the real situation in the browser (pages can return status that does not match what you see).

On the other questions, I think perhaps what you really want is to first determine what is happening and then, WHY?

So, first things first:

• do you have access to the .htaccess file?
• Can you provide the URL (and .htaccess if you have it)? You can PM this info if you don't want to share it publicly.

Sha

Hi,

I would host on site using a service that will create & submit video xml sitemaps and give you access to awesome statistics, while passing link credit back to your own site, no matter where it gets shared.

Wistia would be my preferred option as I know that they fit the bill on all counts, but there are a few good services out there, so you could check out what is on offer.

This post from the SEOmoz Blog will give you an idea of the kind of data you have access to with the Wistia service, so you can make clear comparisons.

Of course, you can also add content to youtube, but I believe it isn't possible to create video xml sitemaps for youtube content.

Hope that helps,

Sha

Hi,

The reason that the Crawl Diagnostics report is divided into three different sections is because the impact of the things highlighted within is completely different.

Warnings are there because it is a known fact that "Pages with crawl warnings are often penalized by search engines." It is up to you whether you take notice of them, and obviously if you have specific conditions on those pages, such as the presence of a noindex meta tag, then obviously you may choose not to bother adding them.

That doesn't actually make the function spammy. It just means that the warning isn't relevant for your particular situation. Requiring the App to check pages with the noindex meta against pages that have missing meta descriptions and remove them from the reporting would add a whole other layer of complexity to the tool.

In short, the tool is meant to identify pages with missing meta descriptions. It does that very well. If you don't need to use the information then you are free to ignore it.

Sha

Hi Catriona,

Rand provided an explanation in a recent webinar as to why social shares that I know exist are not appearing in OSE.

The way that social metrics are currently returned in OSE is limited to the URL that you are checking with the tool. So, if you check your home page URL in OSE, then you will only see social shares for the home page. If, for example there are social shares out there for other pages on your site, they cannot be rolled up into the report.

The upcoming Social Analytics feature looks set to give us much improved data right in the PRO App in future. The word is that the Private Beta will get underway within the next couple of weeks and public launch is likely around Nov/Dec.

If you haven't caught up on that yet, you can read the post from Rand to find out more.

Hope that helps,

Sha

Hi William,

As indicated from the help page that Keri provided, the problem is that the page is entirely rendered in javascript and SEOmoz crawlers do not follow javascript links or redirects.

Of course, the reason why the SEOmoz crawlers do not do this is most likely because Google's (and other search engines) stated position is that they are "getting better" at handling javascript, but the likelihood of trouble free crawling for googlebot is likely low or at the very least unknown.

Bing now has an option in its Webmaster Central that lets you indicate that javascript crawling is required for a site. I have not seen any information on the effectiveness of this as yet, but you could investigate that by hitting their help forum.

Even if search engines manage to crawl the javascript without issue, there are other significant problems with the content on the site. It appears that the site is a multi affiliate whitelabel? All of the text is actually being pulled in from an external page and that page contains content that is duplicated across many other websites. This is the case with every "page".

Unfortunately, all of these things add up to a fairly bad SEO situation. Your best option for generating traffic would be to become massively popular through social channels and use them to feed traffic to the site. That is assuming that this whitelabel platform does not give you the option to create your own content (which would be much better).

Another alternative would be to create a site on a new domain with awesome, unique, shareable content with links to feed traffic to this site, but if you are going that route, making people take an extra click through a second domain on the way to the retailer's site would not be optimal for conversions. So it would be better to add direct affiliate links within the pages.

So, on the whole, I would say that ramping up your social activity is your best approach.

Hope this helps,

Sha

Hi Charles,

Sorry to see that your question hasn't been answered yet!

I just stumbled upon it way down in the list, so am assuming that you still need an answer.

First, you are absolutely correct in deducing that the default Traffic Data report is based on weekly data from Google Analytics.

You can also see Monthly data by going from the report you see in the campaign overview to the Traffic Data page (either click See Full Traffic Summary, or click Traffic Data on the Pro App menu bar).

Once on the summary page, you will see that there is a selector at the top which allows you to change the filter to monthly instead of weekly.

It is important to remember that these Analytics results are for the preceding period (ie the past 30 days, not month to date).

Hope that helps,

Sha

Hi Robert,

Great answers here already, but I just wanted to perhaps give a little different perspective on your problem...

I know it is very frustrating to see what is happening, but I would be focusing on traffic and conversions rather than rankings. In the end, if you have a home page link embedded in body copy on a high authority site, it is quite possible that you will see an increase in traffic from referrals. If this were me I could care less about a couple of ranking positions!

With the effects of localization and personalization on today's SERPs, it is impossible to know exactly what result each person sees anyway.

Also, have you checked whether the link is followed? If it is not, then the effect upon your site won't be as great as you expected, but the potential for referrals is still enormous

Hope this helps,

Sha

Hi trickyzoo,

Awesome information from Ryan on this one!

Given your industry, and the fact that you do have some budget, one other thing that could make a big impact for traffic, links, brand visibility and in the long run, overall authority of the site is the development of a phone app that people can't bear to be without

Hope that helps,

Sha

Hi Vishal,

I am not aware of there being a separate RSS Feed for the Q&A Section, but the best way to verify that would be to email the SEOmoz Help team  help@seomoz.org.

Hope that helps,

Sha

Hi blitzna,

It is highly unlikely that any of the comments he is submitting will ever be posted anyway, since most bloggers either hold them for moderation and would delete without approving, or they constantly monitor comments and immediately delete anything spammy.

The very first thing that you should do is share your SEOmoz referral link with your guy so that he can sign up for a PRO Trial. At the very least, he will get exposure to all of the content on SEOmoz (including Q&A) and learn some good things about link building. Even better if he sees the value of the membership and decides to sign up - and earn you $100 Referral!

With regard to your question:

The only time that blog commenting is really effective in link building is when you are able to make real contributions to build your authority and reputation. The prerequisite is always that you have something of value or interest to say and that it is totally relevant to the content of the post. Of course, for this to be effective, the blogs (or forums) you choose to comment on must be high visibility and seen as authoritative.

If you think that the problem is really just his inability to write well in english, then you could try another way of approaching the problem:

1. Get him to send you his responses for editing before he posts them (if you feel he has the knowledge to make a valuable contribution on your behalf)
2. Get him to identify posts that you might be interested in contributing to.
   He could either find the posts and send you the links so that you can drop by the blog and add the comment yourself, or if it is difficult for you to do that - perhaps you are likely to be interrupted, don't want to spend time reading other comments etc - he could send you the content of the post and you could type your comments at a convenient time and return to him for posting.

The value of the links themselves will not have a huge impact for your site, but the fact that you become more visible as an authority in your industry is quite likely to result in people becoming interested in you and your business. The overall outcome is that you will quite possibly gain links naturally through increased visibility of your content, social sharing etc.

In addition, I would point him to the SEOmoz SEO Web Directory List and this post from Cyrus about using SEO Link Directory best practices.

If Local search is important for your business, you could also suggest that he try the free version of Whitespark Local Citation Finder to identify any other local citation sources that might be relevant.

With some good exposure to the resources available here on SEOmoz and a good team approach, it is quite likely that you can improve things for your site as well as for your SEO guy

Hope that helps,

Sha

Hi Diane,

I like the way Ryan thinks! ...but I am hoping we won't have to go to the length of having to resort to a content bomb.

The lesson in a situation like this is to realize that being a good white hat SEO unfortunately means needing an understanding of some of the tactics used by black hats or maybe just a little help from some friends

Since there is obviously an issue with your content being copied, the first thing I would do is to implement Authorship markup across your entire site. By doing this you ensure that any content that is "borrowed" is immediately "outed" to the search engines because it doesn't have an external link from your Google profile page, which acts as verification that you are in fact the author of the content. Matt Cutts and Othar Hansson have given a really easy rundown on implementation in this Google Webmaster Help video

For the moment though, it would be better if you can avoid making any changes to the site until we can identify all of the issues in play.

BTW ... I think I have an inkling of what might be going on here ... is there a programmer or designer who is or has been involved in the development of the site besides yourself?

In the meantime, I'm continuing with a diagnostic based on the information we have and will let you know as soon as I have confirmed my suspicions or otherwise.

Hang in there,

Sha

Antonio,

Ryan has explained this perfectly.

For a more detailed explanation of methods for controlling page indexing, you could read this post on Restricting Robot Access for Improved SEO

It seems from your comments and questions about 301 redirects, that there is some confusion on how they work and why we use them.

A 301 redirect is an instruction to the server which is most commonly done by adding a .htaccess file (if you are using an Apache server).

The .htaccess file is read by the server when it receives a request to serve any page on the site. The server reads each rule in the file and checks to see if the rule matches the existing situation. When a rule matches, the server carries out the action required. If no rule matches, then the server proceeds to serve the reqested page.

So, in Ryan's first example above, there would be a line of code in the .htaccess file that basically says to the server IF the page requested is /apples, send the request to /granny-smith-apples using a 301 (Permanent) Redirect.

The intent of using a 301 Redirect is to achieve two things:

1. To prevent loss of traffic and offer the visitor an alternative landing page.
2. To send a signal to Search Engines that the old page should be removed from the index and replaced with the new page.

The 301 Redirect is referred to as Permanent for this reason. Once the 301 Redirect is recognized and acted upon by the search engine, the page will be permanently removed from the index.

In contrast, the request to remove a page via Google WMT is a "moment in time" option. The page can possibly be re-indexed because it is accessible to crawlers via an external link from another site (unless you use the noindex meta tag instead of robots.txt). Then you would need to resubmit a removal request.

I hope this makes clearer the reasons for my response - basically, the methods you have used are not "closing the door" on the issue, but leaving the possibility open for it to occur again.

Sha

Hi Diane,

Ryan's response is spot on and his suggestions are excellent.

If you can provide the URL(s), then we can take a look and see exactly what is going on with the referring page(s).

If you don't want to share the information publicly in the Q&A, you can private message each of us through your SEOmoz profile page.

If you ever think that someone has access to edit pages on your site without your permission, the first thing to do is to check with your service provider whether there are any active ftp accounts that you are unaware of. I have seen situations before where people have managed to get a "back door" set up and then it is as simple as logging in and changing pages without your knowledge.

Given that this involves a legal dispute, if we can do a proper diagnosis and trace the source of the errors (or if there happens to be a back door in place), then you would be able to:

1. issue a cease and desist
2. better secure the server against unauthorized access
3. Ban any ip addresses identified as malicious

Hope that helps,

Sha

Hi Greenman,

I would guess that your problem is most likely caused by the fact that you have used the robots.txt method to block the pages you removed.

robots.txt is not the most reliable method of ensuring that pages are not indexed. Even though robots.txt tells bots not to crawl a page, Google has openly stated that if a page is found through an external link from another site, they can be crawled and indexed.

The most effective way to block pages is to use the noindex meta tag.

So, the simple answer is that there are links out there which still point to your old pages. Just because links are not highlighted in OSE or even Google WMT, does not mean that they don't exist. WMT should provide you with the most accurate link information, but even that is not necessarily complete according to Google.

Don't forget that there may also be "links" out there in the form of bookmarks or favorites that people keep in their browsers. When clicked these will also generate a 404 response from your server.

If you know that you have removed pages you should add 301 redirects to send any traffic to another relevant page. If you do not know the URL's of the pages that have been removed the best way to stop them from returning 404's is to add a catch-all 301 redirect so that any request for a page that does not exist is redirected to a single page. Some people send all of this traffic to the home page, but my preference would be to send it to a custom designed 404 or a relevant category page.

Hope that helps,

Sha

Hi,

I don't use Blogger myself, but I'm thinking that this answer to Blogger question on noindexing archives might give you a viable workaround?

Hope it helps,

Sha

Hi Matthew,

When you are the originator of the content, the most important thing is to be able to show the search engines that you are the author.

There are some people who are producing great content and actually being outranked with it by sites who have "borrowed" it.

If you implement the rel=author meta tag for all of your content, then you are claiming it as your own. Do this when you create it and even if people use it they will be "outed" with the engines.

John Doherty posted recently in the SEOmoz Blog on Social Network Spam and Author/Agent Rank

Hope that helps,

Sha

Hi Brenan,

Social is worthless if it is not, well, Social!

The most important facet of using social media is building authority and/or recognition in your space. This kind of authority brings a raft of benefits, not the least of which is the fact that people will want to share what you share...

At a minimum I would pick the two that "work" best for the individual (or company they represent), and start participating in those in a meaningful way. If that works out, then try adding another and see if it also works for you. How many you can realistically manage to make a real contribution to will become clear over time.

My personal view is that a profile that looks like it has been deserted is more likely to do harm to your reputation than good.

Cyrus talked about managing profiles as part of a link building strategy in a blog post just last night. There is also some discussion relevant to your question in the comments on that post. Of course, there are some directories in the SEOmoz resource list that fill the bill for your co-worker too

In short, give yourself a big thumbs up on this one and send your co-worker a link to the post as well!

Sha

Happy to help

If you strike any issues let me know.

Sha

Hi Jaycie,

Google's view of the issue is that you should have a robots.txt file in order to eliminate the risk of your web host dealing with requests in an unexpected way and returning something strange.

Matt Cutts talked about robots.txt in this Webmaster Help Video last month.

Hope that helps,

Sha

You're welcome.

Happy to help

Sha

Hi Sandeep,

Keyword Rankings are updated every Wednesday and On-page reports are available on demand for any keyword that is loaded in your campaign and any page that you choose to enter.

Crawls Diagnostics are usually updated weekly on whichever day of the week you originally set up the crawl, but crawling is currently disabled due to a major outage on the weekend. There is a post in the Blog about the outage, which SEOmoz staff are updating from time to time to keep us  all in the loop.

Once crawls are re-enabled things will go back to the normal weekly crawl schedule.

Hope that helps,

Sha

Hi again Chris,

OK, well that makes it interesting!

First, if the list of 301's in your .htaccess file numbers in the hundreds, then there is definitely cause for concern about the effect it might be having on your load times.

The .htaccess file is read from top to bottom until a rule is matched. The first one that is matched will be used and no rule after that will be checked. Obviously, if there are hundreds of rules to check, this can cause processing bottlenecks.

Now there are some things you might be able to do to alleviate the problem IF you have the right conditions.

1. If your rebuild involved the relocation of an entire directory or directories which still contain the same pages as before, you can write a 301 Redirect for all pages in a directory with a single line of code

2. If there is a database behind your site and you have some means of matching pages from the old URL to the new (for example, is there a unique product ID for each page?) then you can use database lookups to write the redirects on the fly. This will eliminate the processing bottlenecks, but can only work if you have a reliable means of matching the pages.

3. If you have only one competition or offer at a time, then you could use database lookups to match any page with say, "competition" in the URL and 301 it to the current competition page (and the same with "offer").

4. For unknown links from external sites you can use a "catch-all" 301 to catch any other page that returns a 404 and send it to a single page (you put this rule at the very end of your .htaccess so it is the last to be matched). You could send these all to home, a generally relevant landing or category page, or a specially designed 404 page. The most important thing is that the page you redirect them to is as relevant as possible, or provides options that may keep the visitor on your site (search, menus etc). I prefer not to send these "random" 404's to home as they are likely to bring an  increase in bounce rate. Since bounce is now acknowledged as a ranking factor, I prefer to keep any traffic with a higher probability of bouncing away from pages with the highest Page Authority (PA).

Hope that helps,

Sha