Cost-optimization – getting the most out of your cloud services

Cost-optimization – getting the most out of your cloud services

by Vijay Kumar

You are likely already familiar with the advantages of moving many of your business operations to the cloud. The added flexibility and scalability alone makes it easy for your company to rapidly adapt to changing environments. You have the ability to quickly change applications or provide more resources for your business with ease and minimal expense. You also have the advantage of being able to run various aspects of your business remotely from pretty much anywhere on the globe to be able to increase the opportunities for collaboration even if all of your business partners are not located in the same place.

You are also likely attracted to the major reduction in the need to purchase expensive hardware; it enables you to ramp up quickly without having to commit large amounts of massive capital expenditure.

However, just because you have reduced your initial expenses, does not mean that you can ignore them now or in the future. Typically if you are managing your resources on-premise, you have a finite amount of resources available for your use. While this had the drawback of limiting your scalability and could be very costly up-front, you at the very least knew exactly how to manage your expenses. Many cloud providers, such as Amazon Web Services (AWS) use a “pay as you go” model. You pay only for resources that you are using. This may be very attractive to businesses starting out, however it is quite possible that when you grow, your costs can increase exponentially, especially if your usage is not monitored or optimized.

As a result it’s a good idea to adopt a few strategies to optimize your expenses vs the value you receive.

Monitoring

The very first step you should take is to make sure you are aware of what resources you have allocated and which ones are actually being used. Running instances when they are not being used can increase the expense in areas where they are not necessary.

By monitoring your systems carefully, you can get an idea of whether or not you have particular places where you have provisioned your cloud servers to be too large, and end up paying for resources which are not necessary.  If you have database instances that have not been used for ages, they typically can safely be removed, reducing your costs immediately.  Most cloud providers provide some monitoring tools along with the packages, however if you plan on using multiple cloud services (which itself is a good idea), you may wish to use a third-party tool which can enable you to monitor and compare performance without having to jump back and forth between providers.

Auto-Scaling

As mentioned above, one of the great advantages of using cloud services is to be able to quickly increase capacity if your business grows larger than initially anticipated. It’s very easy to provision more resources to meet increased need.   The drawback here is that this can become increasingly cost-intensive, as you are paying for each provisioned resource.

You may have noticed through your monitoring that there are peaks and valleys for when resources are needed, such as at specific times of the year, or if there was a sudden change in demand. Typically this demand and need for these resources decreases at other times.  If you have provisioned more resources to handle this growth, while this may be extremely helpful for peak times, you may be paying for resources that are not being used at other times.

One of the features that cloud providers such as AWS provide is auto-scaling, which will automatically provision or de-provision resources as the need grows or increases.  This way you will only pay for resources that are actually being used.  Google Cloud and Azure also provides some auto-scaling functionality for managed instances.

auto-scaling

Source: Amazon https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html

While it may seem easy and a good idea to allow your cloud provider to handle your auto-scaling processes, there are a number of things of which you need to be aware. It is important to monitor how these resources are actually being used, as it can be complicated to fine-tune scaling to accurately meet need.  For example, if you are restricting your resources too tightly, you may appear to be saving money, however this can have an impact on the quality of speed that you are providing, which itself can be a drain on income/capacity if it creates a negative experience for the end-user.

AWS does not scan any more often than every 5 minutes, and it can take another 5 to 10 minutes for an instance build to take effect. If you suddenly get a huge influx of traffic (be it legitimate or not) that lasts less than this time, AWS will not be able to handle this and could take down your site or services. For this reason, you may wish to maintain a minimum level of instances regardless of actual traffic at a given time.

Manage Your Storage

One of the factors which can create unnecessary cost is the continuing cost of storage.   Cloud providers will charge you periodic fees for every byte of data you keep with them.  For small amounts of data, this is unnoticeable, however as data has a way of growing, you will want to make sure that you are only paying for storage of data that you actually need.

Automation

One of the best ways of handling this problem is by automating your processes as much as possible.  You can run scripts that run periodically that will automatically cleanup old data that is no longer being used.  You likely have data that was very important at one time, but will never be used again, such as old database snapshots.   You will want to run processes that will regularly get rid of this sort of data.

For some data, such as archives, which you likely wish to keep, storing it in active areas of your cloud is not particularly cost-efficient.   For this reason you will likely wish to create some lifecycle policies, where you move your rarely used resources to low-cost storage options.

Smaller Disk Volumes

If you use large disk volumes, there a strong risk of these filling up quickly. It is not as easy to clean these up as it is to archive them. For this reason you will likely want to use use smaller disk volumes on your servers to manage the amount of data that is being stored (whether you want it to or not)

If you are running large applications, typically you are logging every single request. As a result, your log files alone can quickly fill up your disks. For obvious reasons, you don’t want to do away with your logging, so will likely want to use log rotation to archive your logs on a regular basis. This way, while you can keep all of your ever important logs, you can limit the size of the log files that you retain in order to save storage costs.

Reserved Instances

While the on-demand managed resources provided by AWS and other providers is useful for only spending on resources you are using at a given time, the pricing model can be a bit steeper than using reserved instances.  If you know you need to maintain a set amount of storage, and wish not to go below a certain amount, using reserved instances for storing some of your capacity when there is not a level of volatility to your systems.   If you know that you will be maintaining a steady amount of traffic, but also need to be able to maintain a basic level of capacity, this may be a good method of reducing your expenses.

Serverless Services

As we know, time is money; the more time it takes to create and monitor your instances, this can increase the hit to your bottom time.

If you find issues with the amount of time that it is taking to monitor your cloud instances, you might wish to use serverless services. Despite their name, this does not mean that they do not run on servers; they instead run on remote servers managed by the cloud provider. These are also commonly referred to as Backend as a Service or BaaS. An example of this is AWS Lambda. Google Cloud Services and Microsoft Azure also offer serverless services.Serverless Services

Source: Amazon https://aws.amazon.com/blogs/compute/better-together-amazon-ecs-and-aws-lambda/

With serverless services, you can create your applications, and you can let the cloud provider handle all of the details of deploying the infrastructure. With serverless services, it becomes unnecessary to provision or manage your servers. These provide continuous scaling, and provisioning at a subsecond level.

It’s important, however, to be a bit wary of using solely the serverless architecture provided to you by the vendor, which can introduce a strong risk of vendor lock-in.  While your vendor may like this, it may not necessarily be in your interest to become dependent on one-provider, which can create some problems further down the line,  and can make it difficult to migrate if a need arises.

As a solution to this problem, it’s a good idea to use the the serverless framework (available for free  at https://github.com/serverless/serverless ) which will assist in preventing this problem by allowing you to create portable apps, using microservices that will run in different environments.  By using this framework for all of your serverless services, to move from one provider to another becomes significantly easier.

Containers

Containers, such as Docker or CoreOS, enable you to run entire virtual machines for each application and all of its dependencies. By using containerization you already have a clear understanding of what is required to deploy your applications and services; they are in enclosed units, and deploying them becomes significantly easier, while also making it possible for you to easily migrate between environments. Containers reduce hardware costs, simplify maintenance, and enable you to deliver software much faster, which themselves can help reduce cost.containerized-applications

Source: Docker https://www.docker.com/resources/what-container

Using containers will also have the secondary positive effect of reducing your dependency on a particular environment, and become reliant on using Amazon, Google, Azure or any other popular cloud environments.

It’s important to note that containers are also more beneficial for some application types than they are for others. They will work fine for discrete microservices, but if you require a lot of interaction between services they may not be appropriate for you.

Use up-to-date Servers

One area where you may be accruing unnecessary expense is through the use out of date technology. It may have been a while since you evaluated which servers you are running on.  Server technology changes rapidly, and typically the newer servers will run a lot more efficiently.  If it takes less processing power to complete the same tasks, the less you will need to pay for processing time.  It is generally a good idea to regularly migrate your systems to the latest generation of servers to get better performance.

Overall, using any or all of the above methods can help you gain the best bang for the buck using the advantages that cloud services provide.  While managing your costs in the cloud can be a little complicated,  the benefits remain great.

Vijay is a Senior DevOps Engineer and Systems Architect at Practical Logix

Leave a Reply

Your email address will not be published. Required fields are marked *