Blue-Green Deployment – Minimizing Downtime and Risks

Blue-Green Deployment – Minimizing Downtime and Risks

by Anand Suresh

For DevOps practitioners, achieving automation and continuous delivery at all levels of production is the holy grail. Also, avoiding downtimes and mitigating risks are high up on the list of priorities. Blue-green deployment provides you with a simple way to achieve all of these goals.

In a blue-green deployment process, instead of updating the current production (blue) environment with the new application version, a new production (green) environment is created. When it’s time to release the new application, version traffic is routed from the blue environment to the green environment. So, if there are any problems, deployment can be easily rolled back. Companies like Amazon and Facebook regularly use blue-green deployments. Netflix uses the same technique on the cloud but calls it “red/black push”.
Blue production

Reasons for Using Blue-Green Deployment

Here are some reasons for using blue-green deployment:

No Downtime

In a traditional deployment, the IT team has to shut down the production environment for updates. Depending on the complexity of the software, the downtime can be minutes to hours to even days. In a blue-green deployment, the traffic is switched to the new production environment. The process is instantaneous. So it eliminates downtimes.

Lower Risk

Direct update of a production environment is risky. There is no way to rollback quickly when a problem happens. Even if the previous environment is backed up, the rollback can take a long time. In a blue-green deployment, the previous production environment is on standby mode. So roll-back is easy and fast. If there is a problem with the new version, the DevOps team can go back to the previous environment, resolve the issue and then reattempt a deployment. It minimizes the risk of pushing new code.

Simplifies the Process

Traditional deployments need coordination between multiple parties, both internal and external. Businesses have to find downtime windows that are customer-friendly. In a blue-green process, you have fewer variables to worry about. Also, you can run deployments without interrupting business operations.

Blue-Green Deployment Process

The process is straightforward. Here are the simple steps:

1. Select New Environment

Determine the current (blue) and future (green) environments. Make sure DevOps processes clearly identifies the blue-green environments throughout the organization.

2. Deploy and Test

You can deploy the new application or version into the green environment and run tests. Once the green environment is ready for primetime, its time to plan the switch.

 3. Switch Production Environments

Depending on your objectives and needs, you can use various strategies for the switch. You can use a rolling model to start routing users slowly to the green environment. Or you can cut off the blue environment and switch to the green environment for all users in a single sweep. Generally, most organizations prefer the rolling model because it minimizes risk.

Challenges of Blue-Green Deployment

Even though the process has many advantages, it’s not totally seamless. Here are some issues that need consideration:

Database Changes

Databases can pose a challenge for blue-green deployment, especially if you are using relational databases. Even though your application might be easy to rollback, the database might end up with inconsistent records. So you need to think about how to handle the database connections in a rollback situation. In some cases, you might be forced to schedule downtimes. Additionally, if you are conducting a schema change in your new deployment environment, you can end up with issues in the current production version. Unless done carefully, for certain schema changes like a new not-null field being added, this can result in increased error rates for the servers still running the outgoing version of the code. If you are planning zero downtime deployment with a new schema, make sure you closely monitor the database after a transition to avoid unwanted schema drifts.

Complex Dependencies

For organizations that have built a complex web of dependencies, the blue-green process can face challenges. Especially, organizations that have mixed monolith and microservice architectures in their systems. It might still be possible to apply the blue-green deployment to parts of the application. However, it will require careful examination to determine the right granularity.

Infrastructure Support

Blue-green deployment uses infrastructure duplication. Not every organization can afford the expense of setting up fully switchable production environments. Or even if they can afford it, they just might not have the necessary infrastructure in place. So your planning stage should include evaluation steps to ensure that your infrastructure is capable of handling the process.

Scheduled and Background Jobs

Overtime production environments become complex with various scheduled background jobs and health checks. For example, you might be running cronjobs to trigger certain server backups or protecting your environment through firewall settings. Make an inventory of all the scheduled background jobs as part of the blue-green deployment automation. Manually doing these tasks can be time-consuming and error-prone. However, if you don’t have good DevOps practices in place, figuring out all the background and scheduled jobs can also be a complex task in itself.

Comparing Blue-Green Deployment to Other Common Terms

When discussing blue-green deployment, often the term A/B testing and canary releases are mentioned. It can cause confusion.

A/B testing is not a deployment process. It’s used to compare the features of a software. A business shows a feature to a certain set of users to gather data. Then, it repeats the process with another feature and compares the information. That’s A/B testing. In contrast, blue-green deployments concentrate on the release and rollback process of software versions.

A canary release also reduces deployment risks. It rolls out production versions to a subset of users to check how the release performs. If there is a problem, the version is rolled back instantly. It’s a more controlled way of releasing an application. Canary releases are more complicated to organize than blue-green releases. But it’s useful when you have doubts about the new application version. The process provides early warnings of possible problems.

Blue-Green Deployment Best Practices

Here are a few ways you can ensure the best results from your blue-green deployments:

Use Load Balancers: Avoid the temptation of switching DNS records to route traffic. DNS records take time to propagate. So it will not be a clean switch and end up creating confusion about which servers are serving the users. With load balancers, you will have more control over the switching process.

Use Rolling Updates: Even though it is possible to switch all of your servers at once from blue to green, it’s not recommended. Instead, use a gradual roll-out process switching one server at a time. It’s a better risk mitigation strategy.

Implement Automation: Your DevOps processes should accommodate blue-green deployment. Scripts and automation tools should help you remove manual steps and optimize the process.

Use Health Checks:  Setup health checks for your non-production environment during the testing phase. The non-production environment should have the exact same health checks as the production environment. You also need to design the health checks in such a way that you can easily distinguish the origin of any warning or alert. There should not be any confusion between blue and green environment problems.

Use the Power of the Cloud: On the cloud, you can easily create production environments from scratch from code. So, with proper design and automation in place, the cloud can provide additional opportunities for more efficient blue-green deployment.

Pay Attention to Compatibility: In order to make the blue-green switch transparent, the versions of the application need to be both backward and forward compatible. Ensure the application is thoroughly tested for compatibility. Otherwise, the blue-green environments might not be interchangeable.

Shutdown Inactive Environment: When you have successfully completed a transition, it’s tempting to keep the Blue or Green environment running. However, if you keep running the inactive environment, you’ll have to process messages, events, and alerts for it. So make sure that you properly shut it down. The practice will result in cost savings and stop resource wastage.

Keep it Simple: The purpose of the blue-green deployment is to create a simple and reliable deployment process. If your process is becoming too complex, then take a step back and reevaluate. 

Blue-Green Deployment: Efficient and Cost-Effective

Blue-green deployment has been around for a while now. As mentioned, companies like Facebook, Amazon, and Netflix regularly use it. It has proven to be an invaluable tool. Software companies are using blue-green deployment to increase efficiency and decrease costs of new releases every day.

Anand Suresh is a Senior Technical Project Manager at Practical Logix. He specializes in bridging the gap between the technical and non-technical teams while maintaining budget, timeline and quality of projects.

Leave a Reply

Your email address will not be published. Required fields are marked *