(Or, any day could be Black Friday thanks to CORONA)
The Cloud. Maybe you have already embraced it and migrated business critical platforms to an IaaS provider. Maybe you are in the planning stages of such a migration. There are a lot of reasons to do so, no more racking up pizza-box servers in data centers, crawling through floors and ceilings to pull cables, testing backup generators etc. Let’s be honest, there’s a lot not to miss. And don’t forgot the flexibility, scaling your platform out horizontally or up vertically with a few clicks or API calls.
Then the coronavirus came along to show us that all those t-shirts and laptop stickers you’ve seen at meetups and conventions (when those were still possible) were true: “There is no cloud, it’s just someone else’s computer”.
Unprecedented demand for many online platforms reminded a lot of business of the reality of running in the cloud. IaaS stands for Infrastructure as a Service which is sometimes confused, either accidently or wishfully, with SaaS or Software as a Service. With the latter, the responsibility for responding to demand with additional capacity while maintaining performance SLA’s lies indeed with the vendor.
With IaaS however, ultimately the platform owner is responsible for all of that along with the underlying architecture design, monitoring and forecasting in order to avail of the flexibility offered by the cloud.
No one should feel too bad about platforms that collapsed under unexpectedly high demands in the short term as a result of the coronavirus, I am pretty sure even a few SaaS providers shared the same pains. No one likes to get caught by surprise like that more than once however and I would like to share some of my pre- and post-corona lockdown load testing experiences that may help future proof your platform. You only need to look at a few of the online services such as Amazon who actually realized significant revenue increases resulting from lockdowns to see why it’s a good idea.
1. Less risk, cloning your environment
First off, with a cloud-based platform you theoretically have the ability (and some businesses do) to make an exact clone of your production platform and test-torture it to extremes during the more civilized business hours rather than rousting your teams out of bed at 02:30 to test against production. That is of course if your platform licensing allows such things. Then when you’re done a few days later, poof-back into the cloud it goes. That is a luxury few self-hosted platforms have. And of course, you’re not trying to build a business case in order to beg, borrow or steal CAPEX to buy enough hardware to meet expected demand for the next year. This is where IaaS really shines, you’re really just trying to validate your design and scaling configurations-someone else is taking care of the servers, storage and so on for you.
2. One size doesn’t fit all
There are other motivations for load testing in the cloud. We have tested some platforms for clients (particularly e-commerce platforms) that while they run great in an IaaS environment, they still have some legacy limitations such as requiring manual intervention to add or remove front end servers for instance. Also, some platforms do not handle auto-scaling (scaling out or in) as smoothly as they should and sometimes give end users stability or performance issues while servers are being added or removed from a pool. Some clients will want a recent baseline and capacity test of their legacy platform when performing a “lift and shift” to an IaaS provider and then repeat this process on the cloud-based platform. This allows product owners to compare the before and after pictures of capacity and performance before end users do. Lastly, I have worked with most of the larger IaaS providers and can confidently say that they are not all created equal. They all have their strong and weak points and the question most relevant to your business is: “which provider is strongest in the areas most critical to my platform(s) and how do I validate this?”
3. Forecast modelling
You may already be performing load tests on your production systems and not even realize it. Marketing decided to send a mailshot during your daily peak hours? Your monitoring data may already be giving clues to your platform’s performance during both expected and unexpected peak loads. The downshod of this approach of course is if you’ve managed to convince a user to follow a link in an email you don’t want them to be greeted by a blank page, spinning wheel etc. In this graphic the bottom example shows the desired linear relationship between demand and throughput.
The coronavirus has also highlighted a very important factor for accurate load testing and, well, platform design in general. Good Forecast Modeling is critical. Beyond the standard baseline testing levels, the quiet hours where your ops team gets a chance to drink their coffee and browse alerts and tickets since their last shift, I now recommend forecast modeling to determine at least the following 3 test traffic levels:
· What do you expect your peak season to look like, not only based on historical numbers but also other factors such as organic growth, pandemic-driven increases in online business etc
· What would “exceeding expectations” look like? The best peak season you can imagine, everyone will make their bonus this year-not just the CEO
· What does an unimaginable demand look like as the result of, oh let’s say a virus pandemic, zombie apocalypse or alien invasion?
Not every business will be able to accommodate the extreme traffic levels of that last point with business as usual, however they certainly could be handled more gracefully in many cases. I for instance have been ordering groceries online 2 or 3 times a week for years, long before the coronavirus made it fashionable and I was let down with the number of blank pages and sever 5xx errors my online supermarket presented me with during the lockdown period here in The Netherlands.
4. Not all SaaS providers are created equal
Finally, I would also mention that sometimes it is even a good idea to load test a SaaS provider-at least for your expected peak seasons. I have done this for several clients and can vouch for the fact that not all SaaS providers are created equal either. There are additional considerations with SaaS providers, some do not allow load testing at all. You’ll need to check your service agreement.
Many SaaS providers will allow load testing only when requested well in advance to plan enough resources for your tenant during the desired testing window, at least 2 weeks in advance and sometimes even a month. Most SaaS providers back up their SLA promises with discounts in their billing when they fail to meet those SLA’s. It is up to you however to decide if those discounts are truly adequate compensation when compared to potential lost revenue for a given platform.
You don’t want to be “penny wise, pound foolish” when it comes to your bottom line!
This article is published before on LinkedIn.
