We work hard to keep Dotdigital reliable and highly available. The impact from our fortnightly releases was frustrating our engineers just as much as our customers. This isn’t the first time we’ve felt like this and over the years our release mechanism has seen various improvements. We’ve gone from slow and manual updates with lots of disruption to quicker releases with work queued instead of lost. But with continuous improvement in mind we set out to do better and today we’re happy to announce releases will cause zero downtime from now on!

This means we’ll stop bugging you with our status page alerts every two weeks. And, most importantly, you can use Dotdigital anytime day or night without being slowed down by our regular updates. We’ll still publish our release notes as normal and you can find those on our Help Centre.

The technology story

dotdigital is one of the largest SaaS providers in the UK and I think it’s fair to say we had outgrown our previous mechanism for running and updating our applications.

Until recently our applications were hosted on virtual machines running in the Azure cloud. While we’ve taken advantage of the easy scaling and automated provisioning the cloud brings, our applications had strict dependencies on machines being configured in a certain way which was a hindrance. This coupling made it very difficult to release new versions of our software incrementally. This meant we had to resort to stopping, updating, and starting all our applications in one go, as fast as we could.

A common solution to this problem is containers. They provide a common way to package an application so it can be deployed in isolation on a machine without prior custom setup. The most common system for deploying containers is called Kubernetes and in the past, this typically prohibited the use of Windows and this was a problem. At dotdigital, we use several programming languages but our most common is C# running in the .NET Framework, which needs to be run on Windows (more on this later).

Fortunately, last spring Azure announced support for Windows with their Azure Kubernetes Service. After a lot of hard work from our development and infrastructure teams, we’re now proud to be able to update our applications seamlessly.

Since adopting containers and Kubernetes we’re now looking to remove our dependency on Windows and the .NET Framework. Back in 2019, Microsoft announced the .NET Framework will be superseded by a new version of .NET able to run on both Windows and Linux. At dotdigital, we feel it’s important to keep pace with technology and will soon be rolling out .NET 5 applications running on Linux.

Joining our team

If you’d like to work at a fast-paced company to solve challenging technical problems with the latest technologies then please check out our vacancies.

It’s that last statistic which really stands out for the Dotdigital technical teams. We need to be ready for that level of increased usage and so we spend the weeks prior trawling over telemetry data from our platform to scale our infrastructure accordingly. This year we already knew a lot of the work had been done due to the incredibly busy period running up to the GDPR in May.

There are four major elements to our server infrastructure: website clusters, background service clusters, database servers and email servers. Because we’re cloud-based, each can be scaled separately and so we:

• added extra web servers to handle the 37 million hits our click-and-open tracking website saw, plus another 34 million for Web Behavior Tracking.
• increased the numbers, size and performance of some of our servers which handle background tasks such as sending and importing.
• added a new email sending capacity to our fast-growing US deployment which has seen rapid growth in the last 12 months. At peak we sent over 320GB of email an hour!
• optimized email delivery throughput with improved compliance with email receiver guidelines.

Throughout the day, our engineers monitor system metrics, making notes of areas that will soon need attention. It’s busy days like this which enable us to see early warning signs of weaknesses in our different systems.

Sometimes concerns can be addressed quickly and easily with more computing power, or by altering slightly the behavior of a system via a setting. Alternatively, bottlenecks are fed into the development roadmap so systems can be overhauled for the coming year. As demand from customers continues to increase, we continue to reinvest in our platform and we’re already looking forward to next November!