For successful e-commerce, digital performance is increasingly important. For more than 10 years, MeasureWorks has been publishing the performance of the Twinkle100, the list of best-performing e-commerce companies. What is their secret to scoring so high on digital performance? In this series, MeasureWorks performance engineers engage with leading, fast-paced online retailers. What is the impact of performance on the business? And how does the organization integrate performance into their daily work? We are proud to start with our client bol.com! The #1 of the Twinkle100 with leading site performance for years.
We talk to Berend Marttin (Engineering Manager), Lars de Bruijn (Product Tech Lead Shop) and Edwin van der Graaf (Software Engineer). “When it comes to digital performance, bol.com was once inspired by the Velocity conference and by the many web performance pioneers such as Steve Souders and Patrick Meenan. It’s now in our DNA to continuously improve website performance,” says Marttin. “We know how to do it!” jokes De Bruijn. All three are happy to share their knowledge and experience, including on the bol.com Techlab blog. This knowledge and experience goes a long way: bol.com has been a leader in website performance for years.
The daily challenge: guiding customers through the store as quickly as possible
Bol.com is the store for millions of Dutch and Belgians, with many products in its own store. It is also a platform for thousands of entrepreneurs to sell products online. With such a size and multiple daily releases, you have to be able to rely on a reliable IT landscape. The entire landscape consists of many microsystems; the complete landscape is as strong as its weakest link. “If one system is slow, it creates a ripple effect in the entire landscape. That comes at the expense of the customer experience,” says Van der Graaf.
“The customer experience must be consistent and predictable.”
Edwin van der Graaf, Software Engineer at bol.com
“We know from our own testing that improving performance actually has an impact on conversion,” says Van der Graaf. “Our customers really appreciate it when we pay attention to good site performance. After all, that makes shopping easier. On all fronts, both front-end and back-end, there must be good performance. With that, the customer experience is consistent and predictable. Just like in a supermarket, customers want to get through the store as quickly as possible.”
Performance in practice: Site Reliability Engineering
To keep a grip on site performance and stability, bol.com has embraced the Site Reliability Engineering (SRE) method. The principles of SRE are based on Service Level Objectives, where the customer experience is central. “How good and how fast do we want to be? This is a choice that business and IT make together, and for which they are therefore jointly responsible,” says De Bruijn. “It also gives room to pay attention to site performance.”
An objective of bol.com: visitors should be able to browse a list of products with a performance of less than one second (90th percentile). There are also additional objectives for filtering or paging products. “This way, we have a concrete definition of ‘what is a good customer experience?’ and therefore also a good definition of the website’s performance,” says De Bruijn. “With this we can make an explicit consideration of whether or not to release improvements or functionalities.”
“With objectives, we can make explicit trade-offs for releases and improvements.”
Lars de Bruijn, Product Tech Lead Shop at bol.com
Improving from good performance to perfect performance is no easy task. “We are looking to improve every millisecond. And 10 small adjustments do add up to a nice profit,” says Van der Graaf. “To do that, we work with Real User Monitoring, open-source tools like WebPageTest, DevTools and Lighthouse, supplemented by our own monitoring on the back-ends.”
To keep site performance fast and consistent, the team tests in advance what impact the (daily) release will have. De Bruijn is adamant: “Performance must not slow down.” The team also conducts performance experiments because the business wants to know what the effect of changes is. De Bruijn: “In doing so, we test both A/A (same page, different implementation) and A/B (modified page). We test these experiments based on two data sources: Real User Monitoring and the data from the web server in question.”
With SRE’s efforts, bol.com makes a difference in online shopping
Ambition: #1 retailer with perfect score on Web Vitals
The work around performance is never finished; new technologies with opportunities and threats come along all the time. “We are on top of that. We have weekly meetings to exchange knowledge, hypotheses and test results in the area of performance. In addition, we mirror our performance against public benchmarks, such as the Chrome User Experience data (CRUX). That is based on the Core Web Vitals. These vitals are the metrics for a perfect customer experience,” said Van der Graaf.
“Of course, our ambition is to remain the #1 retailer, but also to conquer the #1 ranking in the Twinkle100 performance list and other benchmarks. Our ambition for 2021: a perfect score on all web vitals! Our IT management will be happy about that. But more importantly, it makes our customers happy!”
“Our customers really appreciate it when we pay attention to good site performance.”
Edwin van der Graaf, Software Engineer at bol.com
