Should /dev folder be blocked?
-
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...
Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?
The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.
Thoughts?
What would be the best way to block out your /dev area?
-
Hey Ryan,
It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic. I just haven't been able to figure what/how I am getting penalized. However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...
The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.
Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?
-
It is a great idea to block the development portion of your site. You can do that with your robots.txt file:
User-Agent: *
Disallow: /dev
But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving youtube videos between accounts/channels
Hi.
Web Design | | corn2015
If you have a youtube channel that has a lot of videos on it and need to move the videos to another channel, is there an easy method to do this without having to download and re-upload them?0 -
Responsive image plugins and seo / crawlability
Note : For the background of this question please read the preface below. Ive been researching responsive image options the main issue i can see with them is that they are not semantic html so bots may not index them correctly. For instance many of the responsive image plugins use data-src for an image rather than src. Does any one have any experience with this and if it impacts on SEO ? Does any one know of a client side responsive image soltion that uses a normal img tag with the image stored in the src and with the option to set an alt attribute ? **Preface : ** Ive got a site we are currently developing, the site has a large full width responsive image slider. To serve images that wont be pixilated we are making the width of the images 1800px wide (which should cover most screens, but isn't actually big enough if the site was viewed on a 27" imac) these 1800px wide images weight about 350kb - 500kb per image and our image slider has about 20 of them. As you can see this would be a problem for anyone with a connection slower than c.10 mbps. This is especially true for mobile devices that will be downloading an image 1800px wide although only require a much smaller one, this coupled with a 3g connection will make the site really slow.
Web Design | | Sam-P0 -
Can multiple domains compete with one another if they have the same/similar content?
Could an ecommerce site with a .co.nz nd .com.au domain compete with one another and hard organic rankings if the content of the pages are the same? All the links would be identical (apart from .co.nz or .com.au) the product descriptions, pages titles etc... would all be identical or similar (our page titles are ever so slightly different). Could this be hurting us? Thanks in advance ^ Paul
Web Design | | kevinliao0 -
How to verify http://bizdetox.com for google webmaster tools
Hey guys i tried to to make a Preferred Domain choice in webmaster tools, but it is not allowing me to save my choice bec its asking me to verify that i own http://bizdetox.com How do i go about doing that and what are the steps I have already verified www.bizdetox.com
Web Design | | BizDetox0 -
Geo Targeting for a City / State - is it possible?
What are the best ways to geo-target for a specific city? (Please share your top 5 ways to do so)
Web Design | | HMCOE0 -
Question About Site Redesign and Nav / Page Structure
Hey guys, i am currently redesigning our company's site, and have come across some things that I'm not quite sure of. We used to have individual service pages in our main navigation (design, video, marketing) before the redesign. In this new design, i had the idea of making just one "services" or "capabilities" page, where these three services would each be outlined, and each service would have a list of links to more specific landing pages. Obviously, breaking it up correctly with HTML5 using the andtags. What I'm wondering is that if i'm going to be penalized for having those three services that aren't necessarily related too closely on the same page as opposed to having the one page for each service (like we have now). Any help would be greatly appreciated, and let me know if i need to elaborate more. Thanks in advance!
Web Design | | RenderPerfect0 -
Does Google follow links inside a <noscript>tag?</noscript>
I'm looking at making an embedable calculator and asking users to embed it to their website. I had the idea of using javascript to include the calculator which would also conatain a text link back to my site in order to gain some back links. If it's possible Google won't see the link (as they may not execute the javascript), is it safe to place the link in the <noscript>tag? If so, Will it be indexed and will Page Rank be passed?</span></p> <p>Thanks in advance for your answers. </p> <p>Anthony</p> <p><span style="color: #5e5e5e;"><br /></span></p></noscript>
Web Design | | BallyhooLtd0 -
SEOMoz crawl report shows a duplicate content and duplicate title for these two url's http://freightmonster.com/ and http://freightmonster.com/index.html. How do I fix this?
What page is attached to http://freightmonster.com/ if it is not the index.html ? Should I do a redirect from the index page to something more descriptive?
Web Design | | FreightBoy1