Should /dev folder be blocked?

[BoulderJoe]

I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...

Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?

The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.

Thoughts?

What would be the best way to block out your /dev area?

[BoulderJoe]

Hey Ryan,

It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic.  I just haven't been able to figure what/how I am getting penalized.  However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...

The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.

Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?

[AdoptionHelp]

It is a great idea to block the development portion of your site. You can do that with your robots.txt file:

User-Agent: *

Disallow: /dev

But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.