Welcome to the Q&A Forum

Melissa.

If anyone knew the answer, they would be going around selling the ability to fix panda and other problems, but I don't think there is anyone outside of google who knows the answer. We are on our own. The same thing that happened to you happened to us over 12 months ago and I'm still struggling to find a cure.

Working on a drastic change next week, but that change will not help you - your site is much too small for this change.

Not a lot of this makes any sense.

Linking to other sites - so is everyone else and they are not affected.

short content - so are many others and we're not hearing from them

some racy content - maybe, but look at all the others.

You have many incoming links from big sites. Maybe their algorithm is so paranoid about new links that it has hit you.

Here are just about my final ideas for you, after this, I can't think of anything.

1. Do you have any copyright content that someone could have reported you for?

2. Are you linking out to any bad sites - check all of your links - including the ones in your footers and sidebars.

3. check your pages with "fetch as googlebot" in case they can't find what they want

4. Check to see if you've been linkbombed with a ton of bad links, by someone who doesn't like you.

5. Register your writers with a google profile and link from their profile on their site to their google profile

You said you checked WMT many times, so you didn't get an unnatural links message

Two possible reasons nobody else has answered - they don't have an answer - or maybe they are too frightened to connect with you, because of the takedown.

Melissa, I doubt you'd get anything in webmaster tools.

That is because it would be a manual penalty, and I have no idea whether they would admit to it or not.

Keith,I think the only way to stop Roger and google from indexing those pages is to put them in the robots.txt file

I made some things global, but Roger seemed to ignore those, so I gave him his own section.

Just modify these to suit your setup.

User-agent: *
Disallow: /tag/*
Disallow: /wp-login.php*

User-agent: rogerbot
Disallow: /tag/*
Disallow: /wp-login.php*

It is possible that one or more people, or a group, has reported you for something.

Some time ago, someone emailed us to say they had reported us to google for something. I don't know what it was, but about a week later, our ranking dropped.

When you have two million pages, its a bit hard to work out what might have caused it.

That is part of the problem - other sites ranking above you.

But thats not the reason it is happening.

The reason is something else - this is just the fallout from that.

I was just searching for one of our stories that I wrote myself.

Everyone that links to us - and even sites that copied the story, with my byline on it - they are all ranking, but we're not.

Its killing us.

Melissa,

That is a dramatic drop, much worse than ours was.

The issue of google indexing index pages and showing them and tag pages in SERPS, but not the original, is typical.

For some reason I don't see yet, google hates your guts.

Your stories are showing up in Australian results, but back a page or two.

If you search for

Drake Announces Second Leg of Club Paradise Tour

you won't find it until you get to the last result and then click the link that says they will show you the results they hid.

This means they hate something on your site and until you fix it, you are out of luck.

I know, because I've been trying to fix this same thing for more than 12 months.

From looking at just a few of your pages, it appears that you have nothing that is 100% original and unique to your site. That probably means they see you as an aggregator or content farm.

I could be wrong, but thats what it looks like.

Show me something that is 100% unique to you.

Some of your content is very thin, too, 20-30 words and a video.

That probably makes your content look like it is mostly duplicate, with just a few words changed.

Your sitemap looks OK.

I don't think having a subdomain is a problem in itself, but the problem is that the internal URLs are relative and so is the canonical link, and if someone gets a shop URL and removes shop or replaces it with www or a browser mishandles it, then you will end up with people getting 404 errors.

At the very least, whatever is handling the www side, if it gets a shop URL, it should 301 to the shop.

Whatever it was that Ben and I drank, it isn't good for us.

Favours it is.

Your H1 has spurious whitespace:

  <h1> Wedding Favoursh1>

Brendan,

yes, what Ben said, plus, if you want to rank for wedding favours, then not having it in the H1 isn't helping you.

You are #15 for Wedding Flowers - which is most important to you?

Your Canonical tag is relative, which is allowed, but an absolute path leaves no room for misinterpretation.

Plus, you have a big problem:

You have shop.website and www.website and if you try the www version of that URL you get a page not found, so you are throwing away traffic.

Your system needs to be smart enough to realise that is a shop URL and do a 301 redirect.

• and this is the reason you really need the canonical URl to be absolute.

Yes, I agree.

Unfortunately, google helped to create almost all of the problems in SEO.

Hopefully, some of this stuff can be rolled back without destroying everything and everyone.

Good answer Ben.

My main site is my own CMS, that I built 10 years ago, so after I added a lot of things to the .htaccess file and it became too large, I just moved the handling inside the control program, that only looks up filed URLs when they are broken. This processing is fast, but if there was any degradation, it only affects the broken URLs.

Speaking of broken URLs, I was getting a few 400 return codes and it seems the webserver handles those, so you have no chance to handle it in .htaccess. So the wat to handle that is with a 400 handler - that on cpanel sites just needs a 400.shtml file, that you can customize.

• you get a 400 response if you request a URL with a % symbol on the end, and some other site did that, thanks very much, and then google decided it would be a great thing to index.

Another option, without needing extra software is to calculate how many rows you can show in google docs, given the number of columns, and then use the text editor to split the CSV into two or more files.

Excel for Mac is brilliant, by the way. I love it.

I agree with Click2Rank and EGOL

• Now that we know what we think we know about google.

It used to make perfect sense to use the keywords that match what you want to be known for, but the game has changed a lot in the past year.

For the case of providing something that others will use, hopefully in large numbers, the domain name seems like a better option.

Some webmasters may have a standard they use, and they will stick to it whatever you do, but you are probably better off using the domain name, with or without the www. - whichever way you do it on your site, and any variations from that will be out of your control.

If you have an exact match domain name, here is where it will help you.

Another option could be the name of your site. For example a shoe store whose domain name is easyfitshoes.com could use "Easyfit Shoes." I don't see how google could have a problem with that. Anyone else got ideas about this?

Thank you Ben.

Sometimes, life is tough! It could be that not all responses are read, so your post might not have been seen. It could also be that another post is easier to read, or the way it was written strikes a chord with the reader, more than other responses.

We are all learning to communicate here, and this forum is a great learning tool, because you can see how some people really connect with you, in their writing, while others struggle to connect.

As my parents used to tell me, practice makes perfect.

Good advice from Andrea and Brent.

To use multiple sitemaps, do something like this:

The main sitemap points to the other sitemap files.

You can have up to 50,000 URLs in those files.

• mine are gzipped

This one is sitemap_index.xml

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>http://yourdomain.com/writermap.xml.gz</loc>
 <lastmod>2012-03-15</lastmod></sitemap>
<sitemap><loc>http://yourdomain.com/mainmap.xml.gz</loc>
 <lastmod>2012-03-15</lastmod></sitemap>
<sitemap><loc>http://yourdomain.com/201201.xml.gz</loc>
 <lastmod>2012-03-15</lastmod></sitemap></sitemapindex>

<sitemap><loc>http://yourdomain.com/201202.xml.gz</loc>
 <lastmod>2012-03-15</lastmod></sitemap>

Here is a tip:

Google will index some of those pages and some it will not index.

If you have 5,000 urls in one sitemap and they only index 4957

you probably can't work out which 43 URLs they didn't index,

so if you make the numbers smaller, it can be easier to discover the pages they don't like.

• not easy, but easier

I'm just going to sleep as you guys wake up, not sure I can sleep after seeing your page, though!

For a start, you have multiple H1 tags.

There should only be one H1

your sitemap looks OK

This appears in results:

<cite>sexylime.com/cu3er/CU3ER-config.xml</cite>

There isn't a lot of text.

Your front page seems to have no power at all.

It almost looks like it has been penalized - maybe people have complained about it. Maybe a lot of people did a search, found you, visited, went back to search and hit the "hide all results from sexylime.com"  There are activists who do that.

I did the "redo the search with the hidden pages included" but that didn't show your page either.

Sorry, I am out of ideas - and I need to sleep.

As EGOL says, it isn't black hat.

It may be duplicate

If you think google cares about duplicate content on different domains, find an Associated Press story, then search for that headline.

You will find it on 100 sites. If google cared about duplicate content on multiple sites, none of those would show up in the serps.

However, and there is always a however... if they don't like your site for some other reason, they may bury your site in the results if you have duplicate content that appears on other sites.

another question:

Do you have a sitemap?

If so, what does it say about the home page?

Deciph22

There are many things to check.
If you search for exact content that appears on your home page and it isn't in the results, then there is a problem.
Did you do an exact search, in quotes?
What happens if you go to the end of the results pages?
Is there a link that says redo the search with the hidden pages included?
If you click that, does your page show up then?
You said it shows up if you do a site: search, so it is in the index,
Have you inadvertently added no index, no follow or no cache tags to it?
Or inadvertently added it to the robots file?
Does it have any bad links in it?
Is the text you mentioned hidden?
Have you crawled it as google bot?
Do you deliver different text to crawlers?
Has your site been hacked?
Have you registered with Webmaster Tools?

• are there any messages about it?
  What was the date of the most recent cache copy of your site?
• have you checked to see if that is ok?

That should keep you busy for a few minutes.

Hello deciph22 Also check that you have no links that still point to the old structure urls Ad check that you only refer to your home page in one way - don't refer to it as / and also as index.html or any other default page. I see a lot of sites that do it and it will hurt your ranking

The downside, of course, is that if something happens, your whole business depends on one traffic source. That is a very bad thing.

Louis, I think it is just a standard message, so don't worry, I doubt it will take that long.

I just started one on a 30 page site, so I expect it will show results tomorrow.

The ease of grabbing an expired domain depends on many things.

If nobody else wants it, then it is easy.

The first time I did one, several years ago, I failed.

You pay the money and it gets you a backorder credit, that lasts for a whole year. If you fail, you try another one, because you don't lose the credit until one of two things happens - you get a domain or your credit expires.

On my third try, I did get a domain.

I also tried another method before using backorder - I wrote to the existing owner, - he didn't even know it had expired. He wanted $1500 for it, and I said it was too much for me. He renewed it. He still has it, and has never done anything with it. That was a bad move on my part, because its not likely anyone else was interested in it. That is what made me try the backorder method for other domains.

Godaddy backorder is now $20.99 - I do them for $18.95

It is about $8.99 for private backorder.

In my system, I think you can also request a refund if you fail to get the domain you wanted and you don't want any others. Godaddy is probably the same.

Thank you David,

I didn't know that!

I believe that the wider you cast your net, the more fish you catch.

I would rather have one link from each of 10 different sites, than 10 links from one site. Or even 100 links from one site, assuming the sites were of equal power.

But I also think it is nice to have a few different links, pointing to different content you produce, even from one site.

For example, a big site links to my newspaper, with several hundred thousand links. It is because they put the link in their sidebar. I am very grateful for that, because we get visitors from there. I much prefer the links they have directly to stories that relate to their audience, because not only do we get repeat visitors, but they are more targeted and more likely to read more than one story on the same subject.

I also notice we have a single link from thousands of small websites, linking directly to a particular story. That definitely seems to help those individual stories.

Most links that you don't solicit are good, but I think links from more new sites are better.

Welcome to SEO - not always as interesting as photography.

Yes, I agree you can't always tell.

Quite often, the posts are not advice on how to fix your problem. Quite often, the posts are more like asking you questions, trying to qualify the question you are asking, or a poster may take the time to tell you what their experience was, with something they believe is similar to the question you are posing.

If a response makes you think about your question, so you can go back and update it, so following readers can more easily understand your question, then that poster helped you, by giving up their time, so you could improve.

As humans (I think most SEOMoz contributors are) we often see interactions that we don't contribute to, but we can quite often learn something from the posts - and that can be worth a click. Or, we may agree with what someone else posts, and rather than writing an entry that says, Yes, I agree with EGOL (he writes as lot of responses) you can simply give his post a click.

I've never been one for touchy feely stuff, but I know that feedback is important to all of us.  ( I just got a new Personal Development writer in my newspaper, and he is making me rethink a lot of things. )

Hello.

I think google doesn't like some things about your site.

You rank #14 for motor cycle center.

I see you have a hidden div that contains a login box, right up in the top left corner, so the first 50 words on your page relate to your menu and that login text.

The rest of the text on your page consists of a mass of motorcycle-related words in links, but nothing in the way of text to tell the crawler what your page is really about.

Surprisingly, the on-page report gives you an A, but this report isn't smart enough to understand that you don't have any human-readable descriptive text,

I suggest you do a Crawl as Googlebot and see what it tells you.

Thank you Miriam, this is excellent advice, thank you for taking the time to do it. I wish I could give you 10 thumbs up!

Sorry Dru

CDN is a Content Delivery Network.

It helps make your site faster by serving your images, CSS and javascript from a network across the world.

If you misconfigure it or configure it lazily (as I did) then it is possible for the search engines to access your content from the CDN and that can create duplicate content, accidentally, for which you may be penalized.

• and that might be a reason for a ranking drop.

Thank you Edward.

I don't have quite that problem, but I think you are right too.

My CDN is set up to be Origin Pull.

That means there is no need to FTP - the system just fetches content as requested.

• you should check that out if you have to ftp everything.

But what you said that helped me is this - that I should have had one CNAME for images and anotehr CNAME for content and the content should be limited to a folder called content, so I can put the CSS files and the JS files in it and that way, the plain HTML pages at teh root level will never be affected.

I also realized, while checking the system, that I wasn't using a canonical tag in the intermediate pages, as I was in the story pages. So I just added code to add canonical tags for all the intermediate pages and the front page.

I do have a few other types of pages, so I will handle the code for them next.

I think adding the canonical tag might fix the problem, but I will also work on reconfiguring the CDN and change over when the action is not too busy, in case it takes a while to propagate.

I see a lot of people asking questions and getting some pretty good responses, but the people who are responding, often do not get a thumbs up for their answer.

If you are new and you are reading this, then you maybe do not understand that a thumbs up actually helps the person who gave their time to answer your question.

There are several ways it helps, including giving feedback that the answer or the attempt to answer was actually useful, that you learned something or you appreciate the time it took to check your questions and give you feedback that you don't get in other forums, where you might be ignored for weeks or months.

This is a collaborative forum and we are all here to learn something and to pass some of our knowledge to others who need it.

Not every answer we give needs a thumbs up, but if you got something out of the answer, then, surely it is worth a second of your time to say "Thanks, that helped me" or "Yes, I agree with this"

Do you have a CDN?

They started crawling some of my pages through the CDN, which I have never seen before, and they are dropping my real pages and replacing them with CDN pages.

What you just said is only true if you have nothing to write about.

I should have made it clear that I only advocate doing that if you have something to say that is meaningful.

If you don't, then don't do it - If you have something relevant and useful to say, it is better than repeating the same information, whether you rewrite it or not.

Spinning the 750 word story into 10 different versions is a really bad idea, in my opinion.

From what I can see, there is no way to change the color of the G+ button.

They wanted it that way, most people seem to hate it, but we're stuck with it until enough people complain - and maybe not even then.

Remember when everyone hated Microsoft because they knew what was best for us, and they never listened?

google is the new Microsoft.

I have never liked www. it is a waste waste waste of time and effort.

Be sure to tell webmaster tools about your preferred version,

in Site Configuration -> Settings

I know, it is horrible.

I don't mind them testing stuff, but I wish they wouldn't do it on my sites.

It would be nice if they had an opt-in to be their beta tester.

I had this happen to me a few times, but the last time was about a year ago, from memory, but I have never used any automated tools to do searches, and I rarely do more than a few dozen searches per day (actually, I try not to use google now) If all of your PCs are on the same IP address, it will look like automated testing, so probably your only way to stop that is to use proxies with different or changing IPs.

Only if you want them to be indexed.

Alternatively, you can write 150 to 200 words that apply specifically to the location and link off to the original page with 750 words of stunning content.

For example, here is an idea:

_ Window Cleaning and Carpet Cleaning in Murfreesboro _

Murfreesboro and surrounding areas sometimes presents a problem for carpet cleaners, because of the high incidence of termites. These termites..... blah blah blah.

....

Our Murfreesboro carpet cleaning crews are all locals, so they have an intimate knowledge of ... blah blah blah.

....

Read about how our carpet cleaning service fits your unique needs.

Just when I thought I'd seen it all, google hits me with another nasty surprise!

I have a CDN to deliver images, js and css  to visitors around the world. I have no links to static HTML pages on the site, as far as I can tell, but someone else may have - perhaps a scraper site?

Google has decided the static pages they were able to access through the CDN have more value than my real pages, and they seem to be slowly replacing my pages in the index with the static pages.

Anyone got an idea on how to stop that?

Obviously, I have no access to the static area, because it is in the CDN, so there is no way I know of that I can have a robots file there.

It could be that I have to trash the CDN and change it to only allow the image directory, and maybe set up a separate CDN subdomain for content that only contains the JS and CSS?

Have you seen this problem and beat it?

(Of course the next thing is Roger might look at google results and start crawling them too, LOL)

P.S. The reason I am not asking this question in the google forums is that others have asked this question many times and nobody at google has bothered to answer, over the past 5 months, and nobody who did try, gave an answer that was remotely useful. So I'm not really hopeful of anyone here having a solution either, but I expect this is my best bet because you guys are always willing to try.

Hello Cindy Another thing you could consider is to remove the www redirector and see if that then does the correct redirect immediately That would stop the doubles and as the domain isn't going to be in the index eventually, you don't care about having both the www and non-www live.

The 'small' one is Chester, a red Maine Coon. He weighs 20 pounds and he eats 4 packets of cat food per day.

The big one is Lester, a blue Maine Coon. He weighs 22 pounds and he eats mainly biscuits, about 1.5 cups per day, and he drinks a lot of water.

They talk a lot - but they know nothing about SEO!

25139_1401622207763_1448552004_31048632_297990_n.jpg 40021_1531415972526_1448552004_31364194_6540711_n.jpg

Well spotted Stubby.

Warrantywise, stubby has your answer.

Either you or your SEO company have been massively keyword stuffing.

You set up a big red flag and told google to slap you down.

Remove that garbage and great good useful, well structured content for your readers.

After you do that, then go do a reconsideration request and in your request, say what you found, and that you fired the SEO company and removed the keyword-stufft text and rewrote the pages - and anything else you found and changed.

Then wait, because it can take up to a month before you see a response back.

You are lucky you didn't drop further back.

Another option is to fix it and just wait, but if they put on a manual penalty, not an algorithmic one, waiting probably won't help.

Hello Robert.

Why aren't you asleep?

Hello Zeal Digital,

I use a CDN (Content Delivery Network) for images, CSS and javascript.

Doing that adds only about $10 to cost per month for a site that had around 800,000 pageviews per month.

You have complete control over the images. If there is a problem, you can force the CDN to flush a file and reload it from the source. You add code to your .htaccess file that tells the CDN how long to store images before fluching them and getting a new copy. It is all automated, there is generally no work for you to do. I host with softlayer.com and this is part of their service.

The change from self-sourced images, css and scripts had a massive improvement on the server.

• it is a 16-processor linux box with twin 15,000rpm SCSI drives and 12Gb RAM - it is quite fast!

Page delivery times improved by 1-2 seconds.

The server now is so lightly loaded that it could be downgraded to save more money.

It has zero effect on SEO. The CDN is accessed using a CNAME.

-   static.domain.com  - so don't worry about it looking like components are from other places.

The CDN has servers all over the world, so no matter where the visitors are, it is only a few hops for them to get most of the content, making it much faster for someone in Australia who would normally pull images from a server in the USA.

Your only problem with Amazon S3 is that they have crashed it a few times, but other than that, it is a good thing to do.

I wouldn't advise them to self-host, unless you want to increase their costs, server loading and page delivery times.

Hello.

http://dev.visualwebsiteoptimizer.com

seems to be hanging all the pages.

Most pages will not load for me.

Even your contact page will not load.

The forum seems OK.

Alan

We all know nothing lasts forever.

A code change can do all kinds of things.

Things that were important are sometimes less important, or not important at all.

Sometimes yesterdays advice no longer is true.

If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.

While working on this other thread:

http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287

I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.

That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)

Martins - what Steve says is true.

the nofollow tag doesn't stop even googlebot following down the tree.

• I checked my logs.

The nofollow probably tells their system that no pagerank is passed through that link. - and any links on that page do not receive any pagerank.  I say "probably" because we don't know for sure.

Hello Martins.

nofollowing a highlevel folder doesn't necessarily stop robots reaching the lower levels.

You may have links that point to the lower levels or there may be external links pointing to those lower levels.

Unless you test it and prove something, I wouldn't believe that nofollowing an upper page will prevent crawlers from accessing, indexing and following lower pages.

The nofollow only tells the crawler to not follow any links it finds on that page.

If you had the same links on another page that did allow following, the crawler can follow them

It is because Roger ignores those META tags.

Also, google often ignores them too.

The robots.txt file is a much better option for those crawlers.

There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.

John,

Don't worry about google, write the story for your readers.

Target it to their needs, rewrite, paraphrase, add extra information for one group and different information for another group.

I'm not suggesting you only change a few words. I mean to really target the article for each group.

If you want to see how much google cares about duplicate content, try this search on google:

Romney wins Ohio, 4 other Super Tuesday states

Check those results and tell me that google cares about duplicate content.

If google really cared about having only one original story, none of those sites would get any search traffic.

You can see this for every single AP story, all day every day - those sites are paying a lot of money for AP content. If google was penalizing them for always having duplicate content, they would never show up in search results.

I thnk you mean in the one you don't want to rank, the canonical tag should point to the one that you do want to rank. The canonical tag points to the original.

John, there is another option. You can edit one of the stories so that it directly addresses the visitors to that particular site. You may have a different headline, different description. There may be additional text. there may be text removed. Also, you can use a photo or a graphic or a video in this one, that isn't in the other. What this does is tailor the story for those readers.