10 Minutes to Doom: Security Patches

In many cases, from a patched binary, an exploit can be generated automatically within minutes. As Slammer and other worms have proved, only a few more minutes are required to infect all vulnerable hosts on the Internet. What a frightening world we live in.

There are many such adversaries: those seeking control of bandwidth, IPs, or information (e.g., CryptoLocker). There are also those interested in penetrating very small attack surfaces. Essentially, any adversary interested in controlling large swaths of the Internet has their work done for them by the very people trying to secure the Internet. Of course, the adversary may simply have access to the patch before the good guys, which is a problem in itself. But what about the situation where the good guys have the patch first? Are we still shooting ourselves in the foot?

Yes, we are shooting ourselves in the foot. Since I am an avid Debian user, I will be taking a look at the situation in Debian, but the situation in all Linux distributions is quite bad in this respect.

3, 2, 1… do nothing!

It is quite possible a patch for a vulnerability is released before the Debian security team releases one. However, I will be focusing on the case where the Debian security team is the first to release the patch (or releases it in conjunction with others).

This is the question:

How long does it take for the majority of affected machines on the Internet to install the new binary from the moment it is publicly available?

Well, it’s certainly longer than 10 minutes.

By default, Debian machines do not automatically upgrade packages. Some users set up automatic upgrades, but usually not at a frequency near fast enough to beat an adversary using automatic patch-based exploit generation.

But without needing to dive into the details, it can be simply put: due to bad defaults, the large majority of Debian machines are harmed when the security team releases a patch.


This is not an easy problem. The paper linked to at the top of this post proposes three possible solutions:


The idea here is to obfuscate the binary patch in such a way that automated tools cannot generate exploits from it.

While a nice idea possibly worth implementing anyways, it is not possible for Debian to ensure that everyone else releasing a patch for the vulnerability will obfuscate it adequately, and the source patch is almost certainly not going to be obfuscated. No-go.

Fast distribution

Default Debian to download security updates every minute. This would brilliantly solve the problem by crashing the Debian mirrors and no one would get the patch! :-)

The economics of building out a massive cluster to support the hundreds of thousands of simultaneous downloads is tremendously frightening. There is some discussion of using CDNs on debian-project. The economics are slightly improved by using a CDN (that is kind of what they are for), but most likely still not feasible for a volunteer project like Debian.

A P2P system may be viable here, but whether Debian users want to a priori donate uplink bandwidth to security updates is probably quite the formidable political issue. The two attempts at P2P package distribution have both died out: debtorrent and apt-p2p. (I am personally skeptical that an opt-in system would have nearly the amount of necessary opt-in to handle this sort of traffic.)


Instead of releasing a binary, release an encrypted binary, and then the decryption key at a later date & time. Then the binary distribution could be slightly staggered to improve the economics (though it should certainly still be quite aggressive to avoid needlessly delaying the fix).

Some extra precautions may want to be taken here. If encrypted binaries are released before the vulnerability disclosure, metadata may be important. To avoid leaking it, patched binaries would have to be distributed to all Debian machines regardless of what packages they actually have installed.

This seems like the most viable solution to me. I think most in the security community would be okay with encrypted binaries being distributed before the vulnerability is announced / the actual patches are released.

To be more specific:

  1. A vulnerability is found in a critical component by a security researcher.

  2. Upstream is informed, and works with major distributions such as Debian to do a coordinated disclosure at a specific time.

  3. Debian releases an encrypted binary and the disclosure time a few hours before the disclosure time.

  4. Debian machines poll, say, every hour for new binaries.

  5. Debian machines wait for the disclosure time and then try to fetch the decryption key and use it to install the binary.

When does it make sense to do this?

Such concerns have been voiced before. Fully automated upgrades on all Debian machines across the world comes with the obvious risk: a buggy patch, simultaneously breaking all Debian machines across the world. No one, including the Debian security team, is perfect.

Currently there is some protection against such a buggy patch, via the early adopters raising the alarm if they notice oddities. Naturally, in the proposed system, this protection would not exist.

So, it may be reasonable to only do fully automated security upgrades on a certain class of security vulnerabilities: remotely-exploitable ones. What exactly constitutes remotely-exploitable is not clear – there is always going to be someone piping traffic from a socket into every possible input you can think of to a binary – but there are at least very obvious cases where having a system like this would be quite nice. Imagine serious remotely-exploitable security vulnerabilities in openssl, Apache, openssh, etc.

There is a balance here that requires tuning, but the way it is currently tuned is resulting in a doomsday scenario for serious vulnerabilities.

Reaching the new world

To get to a world similar to the one just outlined, there are sociotechnical changes needed. Here are some of them: