Welcome to the Q&A Forum

Hi Sonja,

Adding the meta noindex tag will instruct the search engine to remove the URL from the index.

In almost every case I would recommend using noindex, follow as this will allow link juice to continue to flow to other pages within your site.

Hope that helps,

Sha

Hi again,

Currently you have that product number listed under two different URL's

http://www.vistastores.com/indoorlighting-patiolivingconcepts-20947.html is returning a 301

http://www.vistastores.com/floor-lamps-patio-living-concepts-20947.html is returning a 404.

There are those who would say that it is perfectly fine to return a 404 as long as the 404 page contains proper navigation links and access to other products etc.

I do not disagree with their opinion as far as what should be on the page that is returned (your custom 404 page does this), but in the case of an ecommerce site, I disagree that it is a good idea to just let all expired products return 404 errors.

The nature of ecommerce is such that this will eventually result in search engines seeing an ever-growing list of broken links on your site. With large ecommerce sites, this could result in thousands of 404 errors which the search engine will see no attempt is ever made to correct or reduce.

If you choose to allow searchers to be sent to your custom 404 page, I take a different approach:

• Name the custom 404 page something innocuous like /custom-introduction
• Remove the 404 references from the Title, description and page text - just use a simple explanation that the product is no longer available and invite the visitor to select from the categories available to search for another product, without mentioning the term "404 not found"
• add a catch-all 301 redirect which sends any request for a page that does not exist on the server to /custom-introduction.

By doing this you will avoid creating an ever growing list of 404's, but as I said, there are those who don't worry about that at all.

As to not using 301's because you have put all that work into the pages, if they are disabled in your store the content will no longer be visible to the search engines anyway. As much as we all hate having our hard work seemingly come to nothing like this, it is very much the nature of the business in ecommerce. Products come and go all the time. This is exactly why the 301 redirect is the best option - because it passes almost all of the link value to the new page. So your hard work has not been wasted, but helped to improve the value of another page on the site.

Hope that helps,

Sha

Hi Aaron,

Your 301 redirect is working fine and sending all non www traffic to your www subdomain, so there is no harm to your SEO. The potential for that was explained by Ed in his reponse.

Google has simply indexed your non www root domain before you put the 301 redirect in place and has not picked up the redirect and adjusted the index as yet.

If you have not already done as Nico suggested and advised Google in your Webmaster Tools account which is your preferred domain, then you can do this. It is likely to enable Google to pick up the 301 more quickly, although I have heard Googlers refer to it as a "hint" which means that you should not hold your breath, or be overly concerned if you do not see a change in the results page straight away.

Bing also has the same function in Webmaster Central.

Sha

Hi Wardy,

The .htaccess file is read from top to bottom and the first Rule which matches is the Rule that will be used. This means that currently, your Wordpress Rule is a "catch-all" and the 301 will never be executed.

So, the 301 needs to be placed above the Wordpress Rule in the .htaccess

Hope this helps,

Sha

Hi Gary,

If the two pages are identical (or close to identical), you could add a canonical link element to one of them as an alternative. Just choose the one that you wish to be seen as the preferred page and add the code in the HEAD of the other to identify it. Here is an example of the code you might use to identify your new page as the preferred page:

Hope that helps,

Sha

Hi,

Without seeing the site it is a little hard to say exactly what is happening and how things could be improved. Perhaps if you could tell us these few things:

1. Is that 2000 Uniques per month or per day?

2. What is the intent of the site? Are you trying to sell products, earn from advertising, be a leader in your space? etc

3. What is the URL for your site (if you are happy to share it)

4. Do you know how many backlinks you have?

With a little extra information we can better understand the problem and suggest some solutions.

Sha

Hi Dan,

There are a couple of simple ways you can deal with this problem. Basically all you need is the ability to edit a .htaccess file. So simple even I can do it!

Our head of programming wrote a blog post to explain both methods for you. Take a look here Don't Steal My Images!

Hope that helps,

Sha

Hi Derek,

I think kjay, Harald & Will have all pretty much covered removal (or correction) of links that are being called from within the pages on your site and using 301 redirects to capture traffic from external links.

I would also make sure that you have a well designed custom 404 page set up for your site so that any link that isn't covered by the work you have done on the above will still provide an opportunity for the visitor to get to the content they are looking for.

You should make sure that your custom 404 page contains some well written text that invites the visitor to explore the site to find other content that will help them and a menu that will allow them to click through to other parts of your site.

Hope that helps,

Sha

Hi Jesse,

Without a little help from a googler in the forums there is only one sure way to determine whether you have a manual penalty or not - a reconsideration request.

BUT with the knowledge that you have of some issues and the absence of an obvious effect, I would not recommend the reconsideration route, as it is likely it may actually GET you a manual penalty which at this stage I am doubtful you have.

I would continue to do good work for your site and at the same time, put some effort into eliminating the most troublesome of the links you know about. Keep a close eye out for any effect (especially when news of the next penguin refresh appears).

also...good on you for taking a proactive approach to this and having the guts to make it an issue that gets dealt with.

Sha

Hi Tim,

My point really was that with the right expertise you should be able to improve both usability and seo usefulness of the pages you have without needing to create alternative pages. Even unfriendly URL's can be dealt with by using URL rewrites.

If you have strong page authority for your product pages now, a change of structure would generally be a last consideration for me.

The MozBar is really useful for this kind of page-level analysis. If you work through the page attributes and elements it is easy to identify both tech and optimization issues that may otherwise go unnoticed.

Sha

Hi Wardy,

Check your Private Messages please.

Also, a couple of questions.

1. Is the file called .htaccess?
2. I the page magazine-freebies a static URL or a Wordpress Blog Category Page?

Please make your .htaccess file look like this

RewriteEngine On

RewriteRule  ^/magazines.htm$ /magazine-freebies

Just these two lines, and test to see if it works.

Sha

Hi Patrick,

Your first link is going to your custom 404 as well, but I get what you are describing, so yes, if it were me I would make sure that the pages are noindexed (using the noindex meta tag, which is the most reliable method) by default when they are created. When you decide that a page is to remain on the site permanently you can then remove the noindex tag and make it visible.

Your custom 404 has everything you need to make sure there is a good chance the visitor will click through to another page in your site, so you are doing all you can to make sure that those people following a broken link don't bounce right back to the search engine.

I would say your logic is correct in that the only way to further improve the odds is to reduce the potential for broken links before they happen.

Hope that helps,

Sha

Hi Matthew,

As Mark suggested, create a spreadsheet in Google Drive and list all of the domains/URLs in the spreadsheet.

Add a single URL in the first field of the Spam Report form (the required field), then in the third field, provide a link to the spreadsheet (300 character max)

Something like: "I believe the sites listed in the googledoc at URL are using manipulative linking practices to influence search engine rankings"

Do be confident though, as Mark warned, that your client's site is able to withstand any scrutiny that might come with a review of sites in the niche.

Hope that helps,

Sha

Hi BJ,

From the information you included here it seems your programmer's approach is sound.

We made a demo for you with a working test and example code for the .htaccess so that you can check that it will achieve your aim in any likely scenario.

Take a look at 301 Redirect and URL Rewrite Example

Hope that helps,

Sha

Hi Rob,

If you can edit your template, you could add the piece of javascript below to the head of each page on your site.

As long as the visitor has javascript turned on in their browser, this will detect that the page is rendered on the wrong URL and send it back to the correct page, outside the Iframe.

You can find more about this on this page (thanks to Ryan for catching the broken link)

There is a nice bonus in that you will then see traffic from referring sites in your server logs. You can very easily follow up with webmasters who have been duped into providing the links - show them how the other site has fraudulently acquired a link from them and suggest that they correct the link to point to your site.

I love it when I find a way to eliminate those guys with the other hats AND get the benefit of all their hard work!

Have a great day,

Sha

Hi Andarilho,

Welcome to Q&A!

Good advice from both Andrews above. There are so many ways to learn how to improve your site here at SEOmoz.

A while ago I created a post with a list of resources for a bunch of new people who were starting out here, so I thought it might be helpful to share with you. You can read it here

Look forward to hearing more from you around the community

Hope it helps,

Sha

Hi,

Google and Bing can see how much time your users spend on the page, and since they can also see that there is a large amount of information accessible through that page, I don't think you need to be as worried about the "single page" factor as normal.

That said, just because your main user interface lives within a single page, there is no reason that you cannot have other pages linked to it. In fact there are a number of other pages which should be included in your site. For example: Contact, About, Terms, Privacy Policy and (if relevant) Disclosure and/or Disclaimer. They do not have to be right up front or included in your main UI, but they should at least be available for users as text links at the bottom of the page, in a sidebar or somewhere. If you don’t include them you are reducing the appearance of transparency for the site. This works against trust and will make people less confident about doing business through your site. Given that you are in real estate, these things should be a major consideration.

Also, if you do not have an About page, you are reducing your opportunity to grow your customer base and add more clients.

Hope that helps,

Sha

Not having seen the code your programmer intends to use, that is not possible to say definitively. That is why we built the test for you, so that you can see the code.

If you take a look at the code you will see that the Rule is writing 301 redirects if the conditions are met.

A .htaccess file is read from top to bottom. The first condition matched will be the one used.

Basically, you have one set of files on your server, all of which have the extension .html.

When a request is made, you are asking the server to load  the .html file in the browser, but rewrite a "pretty URL" for the user to see in the Address Bar.

So, there are three possible scenarios for a request:

1. the user requests /page.html (server loads.html and rewrites the URL to /page)
2. the user requests /page (server 301 redirects request to /page.html, loads it in the browser and rewrites the URL to /page)
3. the user requests a page that does not exist on the server (server 301 redirects the page to /404.html, loads it and rewrites the URL to /404)

Personally, I would not be going out of my way to rewrite simple URL's just to remove the .html extension, but that is your choice, and I am assuming there is more to it and you had some good reason for doing this in the first place.

The one thing you need to be cautious of in making these types of decisions is that chopping and changing from one thing to another and back again is not a great idea. A 301 redirect should really only be used when you are sure that you want to make a permanent change.

Hope that helps,

Sha

Thanks for catching that Ryan! Having a bad copy/paste day

Hi sansonj

If you are asking about how long it will take for search engines to recognize the changes you have made, then you can speed up the process for Google by using the "Fetch as Googlebot" feature in your Google Webmaster Tools account. This allows you to fetch individual URLs (and pages linked from them if you choose). Once a page has been successfully fetched you have the option of adding it to the index.

There are limits on the number of pages per month that you can fetch, but this is a good option for new or changed pages on your site.

Bing Webmaster Tools has a feature which allows you to add pages to the Bing index in the same way.

Hope that helps,

Sha

Thanks Alan,

Pointed me in the right direction - only focusing on the external links.

Turns out a forum was shut down recently which was the source of more than 1200 links (and 1 linking c block, of which there don't seem to be a whole lot). Seems the client has not employed a very diversified linking strategy.

Thanks again,

Have a great night

Hi CP,

What you are seeing is the internal links between pages on your site.

When I checked your home page using the Mozbar I see more than 200 links (most are from your sidebar menu). I also checked one of your product pages and see almost 190 links on that page too.

Your other sites may have less links per page, the menus on them may be built in a different way, you may have more external inks than internal, or Webmaster tools may just not be showing them to you (Google says that we will see more links in Webmaster Tools than using a link: search in a browser, but still may not see all of the links that they are aware of).

Hope that helps,

Sha

Hi Tim,

hmmm..that is a quick response from Google!

I suspect your problem is that you have unintentionally broken all of the indexed links in the google search results- Ouch! ...and any external links to posts from other sites - Ouch again!!

Basically, Google has the URL of your pages indexed, but you have changed all of the URL's without creating a 301 (permanent) redirect. So, when a user clicks that link in the Search Results page it returns a 404 Not found error.

Ah..I see that Dan has explained what you need to do to fix the problem.

Sha

Hi Naomi,

Since the problem is happening when you are using the On-page Tool, I would suggest you send an email direct to the SEOmoz Help Team providing the URL, keyword phrase and campaign number (if there is one).  Any time that you are seeing results from the tools which seem strange, it is a good idea to ask the Help Team

The address to send your request to is help [at] seomoz.org

Hope that helps,

Sha

Hi Don,

There are some things you can do to optimize your product feeds.

There is some specific information about optimizing feeds for Google in this Q&A thread Does anyone have any tips for optimizing your Google product feeds? 

CPC Strategy are quite generous with information to help site owners who wish to manage their own feeds in their blog. There is some information about Bing feed optimization on their Blog in this Tip of the Day post.

Hope that helps,

Sha

Hi CP,

If you wish to use robots.txt to block crawlers, then your two robots.txt files should be as follows:

For your http protocol (http://vistastores.com/robots.txt

User-agent: *
Allow: /

For the https protocol (https://vistastores.com/robots.txt

User-agent: *
Disallow: /

Personally, I prefer to use the noindex meta tag for page blocking because it is a more reliable way of ensuring that the pages are not indexed.
(Never try to use both at once)

This link explains the difference between the two:
[Google Webmaster Tools Help.](http://www.google.com/support/webmasters/bin/answer.py?answer=35302 "Robots blocking crawlers")  

Hope that helps,

Sha

```You can use a robots.txt file to request that search engines remove your site and prevent robots from crawling it in the future. (It's important to note that if a robot discovers your site by other means - for example, by following a link to your URL from another site - your content may still appear in our index and our search results. To entirely prevent a page from being added to the Google index even if other sites link to it, use a [noindex meta tag](http://www.google.com/support/webmasters/bin/answer.py?answer=61050).)

Hi CP,

This is an issue for all stores that use the information supplied by manufacturers without any change. Unfortunately the sheer number of people out there with the same content duplicated across hundreds or even thousands of sites, makes it a serious problem.

Search engines will see this as low quality content and it will certainly make it more difficult to rank above your competitors. After all, by copying everyone else, you are not providing anything of extra value for the user.

Katie Morris wrote a blog post recently on this subject - eCommerce Syndicated Content: How to Win. In it she explains how you can improve your content very easily by doing some clever, but simple editing to improve rankings.

Hope that helps,

Sha

Hi,

The quickest, easiest and most accurate way is to crawl your site with the Screaming Frog SEO Spider tool.

This will give you the information you are looking for and a whole lot more besides and depending on the size of your site, you could have it in seconds.

If you have your site loaded as a campaign in Moz then you would be able to see the information current to the time of the last crawl.

Hope that helps,

Sha

Hi Jorge,

I have not noticed this problem in my own campaigns.

If you click the F from the first screen it should show you all of the icons, but only the F icon should have reports showing next to it. There is a manual selector at top of this page too so you can change the letter to get a different set of reports.

There is a great explanation of how the On-page feature works if you click the Help link at top of the page and go to this help page.

If yours appears to be malfunctioning, best thing to do is click the link at top of that Help page to Submit a Request and ask Moz staff to check out your campaigns. Be sure to tell them which specific campaigns it is happening with to make it easier to track down.

Hope this helps,

Sha

Hi Ade,

Since learning this one thing will be something that you are likely to use over and over, I figure it is much better if you see how it actually works. So, we wrote a little resource to show you how to do a basic 301 redirect as well as one that goes back one level to your category page.

If you take a look at this simple 301 Redirect course for managing 404 errors, you can see three working pages and also download the code.

Let me know if you have any questions.

Hope that helps,

Sha

Hi Jon,

While it is difficult to pinpoint exact reasons for traffic fluctuations without detailed information on the site, there are some things that come to mind.

First, since one of the pages you mentioned relates to the Rugby World Cup, I would check the originating country(ies) for the traffic. It is quite possible that the visitors were interested in betting on matches involving a team or teams who have been eliminated, or whose playing schedule has been less frequent in recent days.

Also, when I took a look at the Metrics for your site in Open Site Explorer, I noticed a few things that could certainly be negatively affecting your rankings.

1. The anchor text distribution for the small number of external inks that you have is quite narrow. This Whiteboard Friday Video gives a good explanation of using a diverse range of anchor text.

2.All of your links are followed. Cyrus gave a good explanation of the negative correlation for high percentage of followed links in this video.

3. When checking the load speed of your home page with the Mozbar, I see that it took 13.5 seconds to load the entire page. The impact of page load speed is also mentioned in the video on negative ranking factors above.

4. All of your external linking root domains are from dot com domains. This means that your link profile does not really have any diversity. Justin Briggs wrote a great blog post which explains the signals that search engines take into account when looking for low quality sites. In the explanation he talks about the importance of a diverse natural link profile.

My guess would be that a combination of all these factors is responsible for your drop in traffic. While the playing schedule is not something that you can control, all of the other things can be improved with a little work.

Hope that helps,

Sha

Thinking about it in terms of percentages might be causing you a problem...

One thing that can be helpful if you are unsure whether your text reads naturally, is to get someone else to read it aloud while you listen. It's a good excuse for a break and a cup of coffee

Sha

Hi Deanna,

I'm happy to take a look and help you get set up.

Normally I would suggest you give the SEOmoz staff a call, but HOPE with a holiday weekend happening they will be out and about enjoying some relaxation til Tuesday.

No need to worry about hiring - I'm happy to spread a little TAGFEE and help get you started.

If you go to your SEOmoz profile and check your Private Messages I left one for you.

Catch up soon,

Sha

Hi Ade,

So sorry I wasn't around to follow this up for you. I have been away for the day and had wireless connection issues, so could not check Q&A until now.

Oops! Yes. Joomla does have its own error handling which does make a big difference, but it should be simple to fix once you understand what happens when you put the .htacces file in place.

When a request is received by the server, the .htaccess file is read from top to bottom, checking each rule in the file for a match. Once a match is found, the specific action assigned to that rule is executed. This means that no rules thereafter are read.

So, if you ensure that your code appears at the beginning of the .htaccess file, then whenever the conditions described by the rule are matched, the redirect will occur. However, if no other rule in the .htaccess is matched, then Joomla error handling will come into play should any other error be present.

This of course means that any specific rule you wish to add in the future should also appear before the Joomla code. As long as you always make sure it is last to be read, everything should work just as you intended.

Hope this helps,

Sha

Hi Jaycie,

Google's view of the issue is that you should have a robots.txt file in order to eliminate the risk of your web host dealing with requests in an unexpected way and returning something strange.

Matt Cutts talked about robots.txt in this Webmaster Help Video last month.

Hope that helps,

Sha

Hi Catriona,

Rand provided an explanation in a recent webinar as to why social shares that I know exist are not appearing in OSE.

The way that social metrics are currently returned in OSE is limited to the URL that you are checking with the tool. So, if you check your home page URL in OSE, then you will only see social shares for the home page. If, for example there are social shares out there for other pages on your site, they cannot be rolled up into the report.

The upcoming Social Analytics feature looks set to give us much improved data right in the PRO App in future. The word is that the Private Beta will get underway within the next couple of weeks and public launch is likely around Nov/Dec.

If you haven't caught up on that yet, you can read the post from Rand to find out more.

Hope that helps,

Sha

Thanks Keri,

Happy to help where I can.

Actually, I think a little taste of TAGFEE could lead to addiction!

Have a great weekend

Wow! Thanks Ryan.

I'm sure it won't surprise you to know that I'm always reading eagerly when I see you respond to a question as well.

Hi BaseKit,

What is the reason for there being so many 404's?

Did you move your site or rebuild your site structure? Do you have a lot of pages that are removed after a short time?

The answer to these questions will help to know what is the best approach for your situation.

Sha

Hi Neil,

I would say that probably the perfect place to seek a highly technical answer like that - and have the opportunity to explore the subject if you need to - would be the SEOmoz Engineeering team.

My suggestion would be that you use your Private Question here in Q&A (you get one private question a month as a Pro member) to ask for some help.

An administrator will make sure that your private question is directed to someone on the SEOmoz staff with the specific expertise to help you out.

Of course, with Thanksgiving it may take a little longer than usual, but I would ask now anyway - I'm constantly surprised by Mozzers responding out of hours etc.

Hope that helps,

Sha

Hi John,

Since your campaign is set up using the root domain, all subdomains will automatically be included.The only possible way that you may be able to remove the subdomain from the crawl would be to use robots.txt to specifically block mozbot from crawling it. However, if you do this, then it will not be possible to access the information for the subdomain separately.

So, what you are really asking is whether the subdomain can be split into a separate campaign without you having to start from scratch. I suspect the answer is no (certain things, like keywords may not be relevant anyway).

The only way to get a definitive answer on whether it can be done would be to email the SEOmoz help team direct - help [at] seomoz.org.

Hope that helps,

Sha

Hi Ryan,

The major problem is that any experienced programmer can easily write their own script to scrape a site. So there could be thousands of "bad bots" out there that have not been seen before.

There are a few recurring themes that appear amongst suspicious User Agents that are easy to spot - generally anything that has a name including words like grabber, siphon, leach, downloader, extractor, stripper, sucker or any name with a bad connotation like reaper, vampire, widow etc. Some of these guys just can't help themselves!

The most important thing though is to properly identify the ones that are giving you a problem by checking server logs and tracing where they originate from using Roundtrip DNS and WhoIs Lookups.

Matt Cutts wrote a post a long time ago on how to verify googlebot and of course the method applies to other search engines as well. The doublecheck is to then use WhoIs to verify that the IP address you have falls within the IP range assigned to Google (or whichever search engine you are checking).

If you are experienced at reading server logs it becomes fairly easy to spot spikes in hits, bandwidth etc which will alert you to bots. Depending which server stats package you are using, some or all of the bots may already be highlighted for you. Some packages do a much better job than others. Some provide only a limited list.

If you have access to a programmer who is easy to get along with, the best way to get your head around this is to sit down with them for an hour and walk through the process.

Hope that helps,

Sha

PS - I'm starting to think you sleep less than I do!

Hi Diane,

Ryan's response is spot on and his suggestions are excellent.

If you can provide the URL(s), then we can take a look and see exactly what is going on with the referring page(s).

If you don't want to share the information publicly in the Q&A, you can private message each of us through your SEOmoz profile page.

If you ever think that someone has access to edit pages on your site without your permission, the first thing to do is to check with your service provider whether there are any active ftp accounts that you are unaware of. I have seen situations before where people have managed to get a "back door" set up and then it is as simple as logging in and changing pages without your knowledge.

Given that this involves a legal dispute, if we can do a proper diagnosis and trace the source of the errors (or if there happens to be a back door in place), then you would be able to:

1. issue a cease and desist
2. better secure the server against unauthorized access
3. Ban any ip addresses identified as malicious

Hope that helps,

Sha

Hi Gary,

A 301 Redirect should only be used if the change of URL is unlikely to ever be changed. One of the reasons for this is that a 301 Redirect also signals to Search engines that the originating URL should be de-indexed and replaced with the target URL.

Since you could conceivably wish to use the same advertised URL at some time in the future, but send it to a different product, this is likely to be problematic.

My suggestion would be to use a simple URL Rewrite Rule instead of a Redirect. URL Rewites basically tell the server to load the target URL when a request is received for the originating URL.

A URL Rewrite has no effect on Search engines and can be removed or changed at any time with no effect at all.

Hope that helps,

Sha

Hi Robert,

Sounds like time to hit SEOmoz Feature Requests and suggest something

Sha

Hi,

My first impression is that your existing site is under windows hosting and your developers' advice is aimed at ensuring that your Wordpress installation is on a LAMP system. This makes total sense.

While it is possible to install and run Wordpress on Windows, the preferred option would always be to use standard LAMP hosting as there are things in WP that will not work as they should with Windows hosting. Most notable of the things that will be a problem is URL rewriting. Running WP on Windows requires a high level of expertise and is likely to require ongoing work, hacks etc over time.

In contrast, Wordpress can be installed and self-managed with great success on a LAMP system.

On the whole I would say that the solution recommended by your developers would be less frustrating and way less expensive than installing WP in a Windows environment (for both you and your developers).

Hope that helps,

Sha

Hi Shradda,

I agree with Ryan that the use of a meta noindex tag is the preferable way to block the pages, but obviously there may be difficulties with applying the tag, depending upon how your pages are generated and whether you are able to alter the code or not.

You can also use ?option=com_fireboard etc to create 301 redirects back to a higher order category page or search.

You should be able to use a single line of code to 301 all pages within each directory.

Using 301 redirects will also send a signal to search engines to de-index those pages.

Very clever 404 page too! Had to watch him go all the way across the page and back just so I knew I wasn't missing anything!

Sha

Good point Keri,

Perhaps most important of all would be a nice big visible link to the product from the home page

Sha

Hi callassist,

You are correct in assuming that the SEOmoz crawler does not distinguish pages that are noindexed.

There is a simple explanation of the reason for this in a response to this thread by Daniel Deceuster.

As you will see there, a feature request is already in play for a feature to allow users to "turn off" the pages they wish to ignore in their reports (and this would be an example of where that will be very useful). If you follow the link in the previous thread, you will be able to keep updated on progress with the feature request.

Hope that helps,

Sha