Should /dev folder be blocked?
-
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...
Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?
The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.
Thoughts?
What would be the best way to block out your /dev area?
-
Hey Ryan,
It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic. I just haven't been able to figure what/how I am getting penalized. However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...
The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.
Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?
-
It is a great idea to block the development portion of your site. You can do that with your robots.txt file:
User-Agent: *
Disallow: /dev
But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ability to Transition Completed Wordpress Website to New Coder/Developer
We have worked with the same Wordpress developer since 2012. They recently redesigned our Wordpress site. We purchased a real estate theme and they performed major modifications to it. The project took 8 months. There are many customized widgets and multiple plugins. We hired a new SEO. The SEO is very comfortable coding. The SEO performed certain modifications and the code broke. The original developer stepped in and and helped restore the code. The SEO stated that the site should not be so delicate; that too many plugins and widgets are used making it inherently unstable. The original developer is claiming that the SEO did not follow best practices (they did not use a dev server to test). For a non technical business owner this is very disturbing. We finally agreed that the new SEO would make changes on a dev server and the original developer will check these changes to ensure they do not break the code. My question is, shouldn't a Wordpress site be simple enough to hand over to a decent coder with little risk of breaking the code? Are there any standards regarding the hand over of a site? I am comfortable with my developers, but what if they change professions or close their company? How would I transition the site? There must be standards and protocols that allow a third party, such as an SEO to change code without causing havoc. Any one have some insight?
Web Design | | Kingalan11 -
Homepage Gateway for Website Divisions (Residential / Commercial) Bad for SEO?
When a website offers multiple divisions for products and services would it be a good SEO practice to implement something as Culligan does by having their homepage be a sort of gateway to the different divisions of site (home, office, commercial, industrial)? 2DaYz
Web Design | | m-johnson0 -
Redirects Not Working / Issue with Duplicate Page Titles
Hi all We are being penalised on Webmaster Tools and Crawl Diagnostics for duplicate page titles and I'm not sure how to fix it.We recently switched from HTTP to HTTPS, but when we first switched over, we accidentally set a permanent redirect from HTTPS to HTTP for a week or so(!).We now have a permanent redirect going the other way, HTTP to HTTPS, and we also have canonical tags in place to redirect to HTTPS.Unfortunately, it seems that because of this short time with the permanent redirect the wrong way round, Google is confused as sees our http and https sites as duplicate content.Is there any way to get Google to recognise this new (correct) permanent redirect and completely forget the old (incorrect) one?Any ideas welcome!
Web Design | | HireSpace0 -
How to rewrite/redirect a folder name with .htaccess
I have a folder in my site that I want to rename. I don’t want to just rewrite the URL and keep my old folder name, I want to change the folder name and then do whatever is necessary with .hataccess to not lose search engine rankings. The folder name I want to change has a space in it and also is misspelled (whoops x2) Example. Mysite.com/old foldr/page.html Mysite.com/newfolder/page.html How would I go about doing this with .htaccess? Do I just switch the folder name on my server and then set up a redirect, or do I do a rewrite? Sorry now familiar with the terms or .htacces. Thanks all
Web Design | | SheffieldMarketing0 -
Question #1: Does Google index https:// pages? I thought they didn't because....
generally the difference between https:// and http:// is that the s (stands for secure I think) is usually reserved for payment pages, and other similar types of pages that search engines aren't supposed to index. (like any page where private data is stored) My site that all of my questions are revolving around is built with Volusion (i'm used to wordpress) and I keep finding problems like this one. The site was hardcoded to have all MENU internal links (which was 90% of our internal links) lead to **https://**www.example.com/example-page/ instead of **http://**www.example.com/example-page/ To double check that this was causing a loss in Link Juice. I jumped over to OSE. Sure enough, the internal links were not being indexed, only the links that were manually created and set to NOT include the httpS:// were being indexed. So if OSE wasn't counting the links, and based on the general ideology behind secure http access, that would infer that no link juice is being passed... Right?? Thanks for your time. Screens are available if necessary, but the OSE has already been updated since then and the new internal links ARE STILL NOT being indexed. The problem is.. is this a volusion problem? Should I switch to Wordpress? here's the site URL (please excuse the design, it's pretty ugly considering how basic volusion is compared to wordpress) http://www.uncommonthread.com/
Web Design | | TylerAbernethy0 -
Nav / Sitemap Question. Using a "services" page vs just linking directly to individual service page?
Okay, so our company offers video production, web design, and web marketing services. While we do offer these services individually, our goal is to get our clients to integrate these services together. Our nav is currently like so : home - about - video - web design - web marketing - blog - contact Now I've seen businesses and agencies also use a nav with a "services" button instead of listing out their service offerings (if they have more than 1, like us). The services button usually links to a category page or has a drop down with links to the company's individual services. I'm wondering if there is any benefit to having a main services page like this and linking to the individual pages off of it (video ,web design, marketing, etc). Or if we should just keep it the way we have it now (since we've already got some page authority on the individual service pages). I know this may not be the most important aspect of our site and we may be over-thinking it but any thoughts/ideas would be greatly appreciated, thanks!
Web Design | | RenderPerfect0 -
Does anyone think the <figcaption>attribute from HTML5 will have any influence for image search?</figcaption>
There is a <figure>element that is supposed to provide better descriptions of image on the web in HTML5 - do you think that will replace the importance of the "Alt" tag? Link to figcaption description </figure>
Web Design | | RankSurge2 -
How do search engines interpret <hgroup>...</hgroup> tags?
Hi there. I'm building an HTML 5 site and through research of new HTML 5 elements I've seen little conclusive information about the interpretation of the new <hgroup>tag, in terms of SEO application and interpretation. In particular does Google interpret the nested heading tags as individual elements or does it combine them into one entity? For example, say I have: <hgroup> # Article Heading ## Article Sub-heading </hgroup> How is this interpreted by Google and what would be some good SEO best practices regarding the <hgroup>tag in HTML5: Is it interpretted as a single tag (" Article Heading: Article Sub-heading ") or two separate heading tags (one and one )? Also, how much does the ordering of the tags matter (say for example I wanted something like the following for visual purposes? <hgroup> ## Article Sub-heading # Article Heading </hgroup> One last thing: is it safe to assume that it is indeed OK to have multiple tags on a single page (as referenced by Matt Cutts a while back in a Webmaster Video)? Thanks! </hgroup> </hgroup>
Web Design | | LMDNYC2