Should /dev folder be blocked?
-
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...
Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?
The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.
Thoughts?
What would be the best way to block out your /dev area?
-
Hey Ryan,
It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic. I just haven't been able to figure what/how I am getting penalized. However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...
The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.
Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?
-
It is a great idea to block the development portion of your site. You can do that with your robots.txt file:
User-Agent: *
Disallow: /dev
But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ability to Transition Completed Wordpress Website to New Coder/Developer
We have worked with the same Wordpress developer since 2012. They recently redesigned our Wordpress site. We purchased a real estate theme and they performed major modifications to it. The project took 8 months. There are many customized widgets and multiple plugins. We hired a new SEO. The SEO is very comfortable coding. The SEO performed certain modifications and the code broke. The original developer stepped in and and helped restore the code. The SEO stated that the site should not be so delicate; that too many plugins and widgets are used making it inherently unstable. The original developer is claiming that the SEO did not follow best practices (they did not use a dev server to test). For a non technical business owner this is very disturbing. We finally agreed that the new SEO would make changes on a dev server and the original developer will check these changes to ensure they do not break the code. My question is, shouldn't a Wordpress site be simple enough to hand over to a decent coder with little risk of breaking the code? Are there any standards regarding the hand over of a site? I am comfortable with my developers, but what if they change professions or close their company? How would I transition the site? There must be standards and protocols that allow a third party, such as an SEO to change code without causing havoc. Any one have some insight?
Web Design | | Kingalan11 -
Is Prerender.io/React going to negatively impact our SEO efforts?
On any page on the site (https://theadventurepeople.com/), the same short code appears. Having investigated Google index pages, Google's cache and Fetch & Render, it does look like Google can view the content and index it, but we're not 100% convinced. Background technical information from the web developer: The website is a single page application built using React. The site is setup with Prerender: https://prerender.io/ (which renders the javascript in a browser, saves the static HTML, and returns that to crawlers). Is Prerender.io/React going to negatively impact our SEO efforts?
Web Design | | Wagada0 -
When rel canonical tag used, which page does Google considers for ranking and indexing? A/B test scenario!
Hi Moz community, We have redesigned our website and launched for A/B testing using canonical tags from old website to new website pages, so there will be no duplicate content issues and new website will be shown to the half of the website visitors successfully to calculate the metrics. However I wonder how actually Google considers it? Which pages Google will crawl and index to consider for ranking? Please share your views on this for better optimisation. Thanks
Web Design | | vtmoz0 -
Hidden Text w/ Java Script _ Is it Bad?
Just came across an article that stated that Google is looking negatively at sites that attempt to hide text or use javascripts to expand text on websites. We are about to launch our new website and believe we are using this technique but im not certain if what we are doing will hurt us. Our website tends to be a little heavy on the text so used a "read more" scrpit that will expand when clicked on. Three sections that use this on the new website Take a look and let me know your thoughts http://joomplateshop.com/demos/catdi.com/
Web Design | | ChopperCharlie0 -
Duplicate home page /index.asp /index.php etc
We recently moved www.devoted2vntage.co.uk to shopify but seem to have multiple home page variants still in google index. I am concerned that these will be causing duplicate content. I have redirected the offending URLs below to www.devoted2vintage.co.uk/ and have set up a canonical URL but need an expect to tell me if I have taken the current steps and if not, exactly what I need to do. www.devoted2vintage.co.uk/index.php www.devoted2vintage.co.uk/index.htm www.devoted2vintage.co.uk/index.html www.devoted2vintage.co.uk/index.shtml www.devoted2vintage.co.uk/index.aspx www.devoted2vintage.co.uk/index.cfm www.devoted2vintage.co.uk/index.pl www.devoted2vintage.co.uk/index.asp
Web Design | | devoted2vintage0 -
Landing Page/Home Page issues
Hi. I was speaking with my designer last night (we are setting up a new website) and we were discussing the design of our homepage, now the designer said he wanted the first page of the website to be a sort of landing page page were the visitor has to click and enter, im sure everyone has all come across these before. However, I am concerned as to the SEO implications of this? Any help guys?
Web Design | | CompleteOffice0 -
SEOMoz crawl report shows a duplicate content and duplicate title for these two url's http://freightmonster.com/ and http://freightmonster.com/index.html. How do I fix this?
What page is attached to http://freightmonster.com/ if it is not the index.html ? Should I do a redirect from the index page to something more descriptive?
Web Design | | FreightBoy1 -
IP block in Google
Our office has a number of people performing analysis and research on keyword positions, volume, competition etc. We have 1 external static IP address. We installed the static IP so we can filter out our visits in Google Analytics. However by 10 AM we get impssible CAPTCHA's or even get blocked in Google. Do you have any experience with such an issue? Any solutions you can recommend? Any help would be appreciated! SXI5A.png
Web Design | | Partouter0