Should /dev folder be blocked?
-
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...
Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?
The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.
Thoughts?
What would be the best way to block out your /dev area?
-
Hey Ryan,
It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic. I just haven't been able to figure what/how I am getting penalized. However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...
The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.
Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?
-
It is a great idea to block the development portion of your site. You can do that with your robots.txt file:
User-Agent: *
Disallow: /dev
But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Reason for robots.txt file blocking products on category pages?
Hi I have a website with thosands of products. On the category pages, all the products are linked to with the code “?cgid” in the URL. But “?cgid” is also blocked in the robots.txt file for some reason. So I'm thinking it's stopping all my products getting crawled by Google. Am I right here? Is there any reason why a website would want to limit so many URL's? I'm only here a week and the sites getting great traffic, so don't want to go breaking it!!! Thanks
Web Design | | Frankie-BTDublin0 -
Block parent folder in robot.txt, but not children
Example: I want to block this URL (which shows up in Webmaster Tools as an error): http://www.siteurl.com/news/events-calendar/usa But not this: http://www.siteurl.com/news/events-calendar/usa/event-name
Web Design | | Zuken0 -
Using Multiple links/names for the same product?
I am being asked to change these product links on the home page: Home/Condo
Web Design | | RoxBrock
Watercraft/Boat to Home
Condo
Watercraft
Boat (Along with several other product links) How does this affect the customer experience/usability, and SEO? Is it a good idea or is it confusing? Thank you.0 -
How to split organic traffic for A/B testing
This might be a silly questions as I may be missing something completely obvious here, but we are completely new to A/B testing. Our site doesn't receive a phenomenal amount of traffic although we are looking to set up some A/B testing for our popular products. Is there a way to split organic traffic for a specific product page. I'm aware that we need to experiment which one performs better in Analytics but I'm unsure how to redirect 50% of the organic traffic.
Web Design | | Jseddon920 -
How to rewrite/redirect a folder name with .htaccess
I have a folder in my site that I want to rename. I don’t want to just rewrite the URL and keep my old folder name, I want to change the folder name and then do whatever is necessary with .hataccess to not lose search engine rankings. The folder name I want to change has a space in it and also is misspelled (whoops x2) Example. Mysite.com/old foldr/page.html Mysite.com/newfolder/page.html How would I go about doing this with .htaccess? Do I just switch the folder name on my server and then set up a redirect, or do I do a rewrite? Sorry now familiar with the terms or .htacces. Thanks all
Web Design | | SheffieldMarketing0 -
Which Hat of SEO (Black/White) Goes with this Green Lace Dress?
Doing some research (honestly) I came across this page: http://www.modcloth.com/th/green-lace-dress That page ranks very well for the phrase "green lace dress" - #2 in Google for me. I'm sure there's a good amount of links coming into it, but on the surface there's only 1 mention of green on the page and 2 mentions of lace. Looking at the code, you can see ModCloth is using the Quick View links below the products to display a more detailed description of the product - wherein lies many more instances of green and lace. So I ask: Which color of SEO hat goes with this green lace dress - Black or White? Is it good SEO to only show the product descriptions when the user initiates the Quick View? Or is it shady SEO to hide so much text from the immediate view of the user? Please select one and explain why.
Web Design | | rball10 -
How to verify http://bizdetox.com for google webmaster tools
Hey guys i tried to to make a Preferred Domain choice in webmaster tools, but it is not allowing me to save my choice bec its asking me to verify that i own http://bizdetox.com How do i go about doing that and what are the steps I have already verified www.bizdetox.com
Web Design | | BizDetox0 -
Is it too late to change an IP from the linking c-block?
My main web development company is linked to many of our clients and our clients link back to us using footer links back. We obviously have a high volume of c-block relations. If I change my main site's location to a different server will it make any difference or is it too late?
Web Design | | sanchez19600