News
BackMitigation of CVE-2023-20593 (Zenbleed)
On Monday, 2023-07-24, it was announced that researcher Tavis Ormandy had found a CPU vulnerability in the AMD Zen 2 platform. Tavis Ormandy described the hole in detail on his website. As we now mainly rely on AMD CPUs here at cloudscale.ch, we realized immediately that we were affected by this security hole. Using the proof-of-concept code that was published together with the description, we were then able to confirm this.
There was no question for those involved in the escalation process that this vulnerability required an immediate reaction / immediate mitigation in order to ensure the security of our customers in the best possible way.
In a next step, three possible mitigation approaches (BIOS update, "chicken bit" and microcode update) were discussed and the latter two were tested in our lab environment. These two problem-solving approaches have the advantage that they can be used during live operations, although the "chicken bit" option would probably have been associated with a considerable loss of performance, which is why we opted for mitigation by means of a microcode update.
After successful application of the microcode update in our lab, we were able to verify that it was no longer possible to exploit the vulnerability using the proof-of-concept code and that operations remained stable.
After our test suite passed successfully, we decided to roll out the update in batches in the production environment. Given the urgency, we did not set a two-week notice period for the maintenance window, as we usually do, but scheduled it with immediate effect – in both cloud locations at once, despite usually scheduling maintenance for two separate days. We initially applied the microcode update on individual compute hosts and then in batches.
The last compute host was finally patched at 01:33 (CEST) on Tuesday, 2023-07-25.
Setting priorities:
Your cloudscale.ch team
PS: In order to be continuously updated about incidents and planned maintenance work, subscribe to the updates on the channel of your choice on our status page.