Thanks for clearing that up and all of the help!

Hey Dmitrii,

I was planning on using two rewrites.

One rewrite for replacing the underscores with hyphens.

And another rewrite for removing the file extensions.

Just so I fully understand, you recommend implementing the rewrite for replacing the underscores with hyphens in our .htaccess file. Then once the new URLs are indexed, change the webpage file names themselves by replacing the underscores with hyphens, make the newly named files live and remove this rewrite from our .htaccess. Is my understanding correct?

Again...thanks for all of your help!

Another question just popped into my head...

Once our new website directory structure and URL format has been rewritten, redirected and indexed by search engines, would it make sense to edit the actual webpage file names (replacing the underscores w/ hyphens) and then remove the URL rewrite that replaces the underscores with the hyphens? Or is this not recommended?

Thanks for the help Dmitrii!

Both the rewrite I posted above and yours for removing file extensions failed to work. However, it seems this one does the trick (taken from the Apache help forums).

RewriteCond %{THE_REQUEST} ^[A-Z]{3,}\s([^.]+).htm [NC,OR]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,}\s([^.]+).php [NC]
RewriteRule ^ %1 [R,L]

Hey Dmitrii,

This rewrite that I posted above...

RewriteRule ^old/(.*)$ /new/$1 [L,R=301]

...isn't intended to remove the file extensions. I'm using it to redirect the old directory structure to our new directory structure.

I was asking if using this rewrite when changing my directory structure will be all I need in regards to having all the necessary redirects in place to not negatively affect our SEO/SERP rankings. Any idea?

Also, would you recommend the rewrite you provided above over the one below when removing file extensions?

RewriteBase /
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^(.*)$ $1.html

Let me know if I'm being clear enough Thanks!

Thanks for the response Dmitrii!

Thanks for for confirming that I don't need to update the webpage file names.

Do you know if redirecting the old directories to the new ones (using the the rewrite below) is all I need to do regarding redirects? In other words, when redirecting directories using the rewrite below is there any need to redirect the old URL format (small_blue_widget.htm) to the new (small-blue-widget)? My understanding is no, all I need to do is redirect the directories; but please share your knowledge.Thanks in advance!

<code>RewriteRule ^old/(.*)$ /new/$1 [L,R=301]</code> 

Hey Moz!

I’m getting ready to implement URL rewrites on my website to improve site structure/URL readability. More specifically I want to:

• Improve our website structure by removing redundant directories.
• Replace underscores with dashes and remove file extensions for our URLs.

Please see my example below:

Old structure: http://www.widgets.com/widgets/commercial-widgets/small_blue_widget.htm

New structure: https://www.widgets.com/commercial-widgets/small-blue-widget

I've read several URL rewriting guides online, all of which seem to provide similar but overall different methods to do this. I'm looking for what's considered best practices to implement these rewrites. From what I understand, the most common method is to implement rewrites in our .htaccess file using mod_rewrite (which will find the old URLs and rewrite them according to the rewrites I implement).

One question I can't seem to find a definitive answer to is when I implement the rewrite to remove file extensions/replace underscores with dashes in our URLs, do the webpage file names need to be edited to the new format? From what I understand the webpage file names must remain the same for the rewrites in the .htaccess to work. However, our internal links (including canonical links) must be changed to the new URL format. Can anyone shed light on this?

Also, I'm aware that implementing URL rewriting improperly could negatively affect our SERP rankings. If I redirect our old website directory structure to our new structure using this rewrite, are my bases covered in regards to having the proper 301 redirects in place to not affect our rankings negatively?

Please offer any advice/reliable guides to handle this properly.

Thanks in advance!

Thanks John!

Your company's optimization tools are very interesting.  I'd like to implement if possible...Do you have a way to integrate it w/ Amazon, similar to SERP Turkey's "Mechanical Turk?".  See: http://www.tomanthony.co.uk/blog/amazon-mechanical-turk-serp-turkey/

I suppose for a small business, you would recommend running an optimization tool like yours or SERP turkey, then tracking the implementation w/ before and after from WMT using spreadsheets, correct?

If so, what is the minimum amount of impressions that should be collected to consider the data reliable?  Our industry has relatively low search volume, so I would venture to guess that a set minimum time frame would make less sense than impression totals.

Hey John,

Thanks for your response. Do you believe the "Search Analytics" data in WMT is accurate? I just finished an experiment where I tracked clicks, impressions, CTR and avg. position (using data from WMT). However, I know that WMT data accuracy was an issue in the past which Google has supposedly fixed.

Since I started the experiments in mid March before the data accuracy was addressed, do you think my test could be thrown off?

Thanks!

Hey Solid_Gold and John,

I'm in the middle of rewriting our title tags to increase organic CTR as well. Did either of you find conclusive results with your tests?

John, I'm assuming you used your own tool Predikkta for testing, correct?

Solid_Gold, I see that you said you'd also run your own before/after tests to determine which title tags had a higher CTR. How did you go about testing this?

Thanks guys!

Thanks!  I figured as much, but I wanted to hear it from another guru rather than assuming on my own.

I appreciate it

Thanks!  I agree with everything you said.  We'll be sure to remove and disavow any negative links.

As you can see from the images I posted above, we are getting quite a bit of referral spam that we need to address.

Any idea if the referral spam can negatively impact Google measurement of user experience / dwell time?  I don't simply want to block it in analytics if it is poorly reflecting on our site.

Great links Patrick!

These are all things we're constantly working on.  The info on dwell time was of particular interest.   Thanks!

We might be wearing tin foil hats a bit, but the past actions of the competition have unfortunately led us down that path of thinking.

Is there any way for us to try to confirm whether click bots are being used to artificially boost the rankings of sites for a particular keyword?  I can't think of any way to detect that activity.

Here are what sites it appears to be coming from.

new

Great tip Tim!  You were right on the money.  Thank you so much

We did find Russian bots with awful user experience hitting our site starting in April.  My webmaster is looking in how to handle this.  Any suggestions to get him started?

8Deil9D

I don't believe they are clicking on my site and providing Google data that I have a bad user experience.

Instead, I think they may be providing an artificially good experience for their site and others that don't directly sell our product.  As a result, my user experience is lowered relative to the others.

Once again, it's just a theory based on their prior behavior and what is actually showing in the SERPS, but something I am concerned about.  If I'm right, I don't believe I'll be able to detect, nor do anything about it short of having my own click bot, which I won't do.

I guess that's the problem...there are thousands of reasons of why the rankings are what they are and there is no way to determine if a click bot is being used.  Is there?

Even if one is being used, I suppose the only thing I could do, is use one myself to counteract it, which I don't like the sound of.

Thanks for the reply Patrick

If we had 100% proof, we would have reported to Google and the FBI.  Unfortunately, like most DoS attacks, it could not be traced back to anyone.

We rank well for every other industry term.  The fact that sites that don't even directly sell the product make me question if something weird is going on here.  My dream scenario (as is theirs I'm sure) would be to be on the first page, as the lone website that actually sells my product directly.  The other sites don't rank well for any other keyword in our industry...just the biggest volume keyword.

I sent you a private message with our URL and the keyword.

I am questioning whether one of our competitors is using a click bot to do negative SEO on our CTR for our industry's main term.

Is there any way to detect this activity?

Background:

We've previously been hit by DoS attacks from this competitor, so I'm sure their ethics/morals wouldn't prevent them from doing negative SEO.

We sell an insurance product that is only offered through broker networks (insurance agents) not directly by the insurance carriers themselves.  However, our suspect competitor (another agency) and insurance carriers are the only ones who rank on the 1st page for our biggest term.  I don't think the carrier sites would do very well since they don't even sell the product directly (they have pages w/ info only)

Our site and one other agency site pops onto the bottom of page one periodically, only to be bumped back to page 2.  I fear they are using a click bot that continuously bounces us out of page 1...then we do well relatively to the other pages on page 2 and naturally earn our way back to page 1, only to be pushed back to page 2 by the negative click seo...is my theory.

Is there anything I can do to research whether my theory is right or if I'm just being paranoid?

I'm surprised as well.

I was thinking that creating PR release about how "I can't give away $25K" might get the fire started.

EGOL, I had the same mindset when creating our scholarship program (see: http://www.jwsuretybonds.com/scholarship/).  I figured the $25,000 prize and allowing anyone going to an accredited U.S. school would take care of it self.

We still decided to play it safe and have an internal resource reach out to universities and scholarship sites to get the seed started.  We started this in the beginning of the year and here is what we've found...

1. Sometimes a spectacular prize is something perceive as too good to be true.  This goes for the universities and students alike.
2. Universities WILL link to you, but you need to search out the ones that do.
3. Universities are AWFUL at letting their students know about scholarship resources.  If you want a lot of entries, it appears www.fastweb.com is by far the best resource.  We now get about 1,000 hits a day from FastWeb (just got listed 2 weeks ago) and are only now beginning to see entries pick up.

I'd like to do this every year, but in order to justify it we need more qualified entries and **a lot more buzz.  **

In our experience, the spectacular prize did not care of things like you and I presumed.  Our internal resource is still working hard to spread the word.