Discussion:
Travis CI MITM RCE
(too old to reply)
Jakub Wilk
2018-08-25 21:49:23 UTC
Permalink
Travis CI <https://travis-ci.org/> is a popular continuous integration
service used to build and test software projects hosted at GitHub.
(Travis itself is free software, but the primary reason I'm writing this
is mail is that the service is used by many free software projects.)

I discovered multiple bugs in the way Travis CI uses APT and GnuPG that
defeat the package authentication mechanism. As a consequence, the bugs
allow man-in-the-middle attackers to execute arbitrary code in the
context of a Travis build that uses the APT add-on.

This is probably not a big deal if use Travis CI only for running tests.
But Travis also supports deployment (grep for “Deployments and Uploads”
on <https://docs.travis-ci.com/>); if you entrust Travis with any
secrets, this is bad news for you.

Here are the bugs details:

------------------------------------------------------------------------

1) On 2015-04-13, the “--force-yes” option was added to apt-get
invocations: https://github.com/travis-ci/travis-build/pull/424

This option name doesn't sound very dangerous, but it makes APT assume
the “yes” answer to all questions, including the question about
installing packages that couldn't be authenticated…

On 2017-10-12, I reported this bug to the Travis CI security team.

On 2018-07-05, --force-yes was replaced with --allow-downgrades
--allow-remove-essential --allow-change-held-packages:
https://github.com/travis-ci/travis-build/pull/1422

I'm not sure how could this change possibly work, because APT in the
Ubuntu versions Travis CI supports (precise, trusty) doesn't have these
options… So a few days later --force-yes was added back:
https://github.com/travis-ci/travis-build/pull/1433

------------------------------------------------------------------------

2) On 2017-10-12, code was added to refresh an expired signing key:
https://github.com/travis-ci/travis-build/pull/1192

The code used 32-bit key ID to retrieve the key from the keyserver.
I reported this on 2017-12-06:
https://github.com/travis-ci/travis-build/pull/1269

My patch was not accepted. Instead, more generic code to refresh expired
keys was added: https://github.com/travis-ci/travis-build/pull/1290

The new code looks like this:

apt-key list | awk -F'[ /]+' '/expired:/{printf "apt-key adv --recv-keys --keyserver keys.gnupg.net %s\\n", $3}' | sudo sh

The “apt-key list” format varies with GnuPG version, but on Ubuntus
Travis supports, it uses… 32-bit key IDs.

(For extra fun, this code happily executes code embedded in user IDs.
I don't believe this is exploitable on Travis CI, though.)

Apparently some joker already exploited this bug to add their own key to
the APT keyring <https://travis-ci.org/jwilk/testbed/jobs/420519943>:

$ apt-key list | grep -A1 -w A15703C6
pub 4096R/A15703C6 2016-01-11 [expires: 2020-01-05]
uid MongoDB 3.4 Release Signing Key <***@mongodb.com>
--
pub 1024R/A15703C6 2016-06-25
uid Totally Legit Signing Key <***@example.org>

I can neither confirm nor deny that it was me. It might be a mere
coincidence that I wrote a tool to brute-force 32-bit key IDs:
https://github.com/jwilk/stopgp32

------------------------------------------------------------------------

3) For many repositories in the APT source whitelist
<https://github.com/travis-ci/apt-source-whitelist>, the signing key is
downloaded over HTTP. For example:

{
"alias": "cassandra",
"sourceline": "deb \"http://www.apache.org/dist/cassandra/debian\" 39x main",
"key_url": "http://ha.pool.sks-keyservers.net/pks/lookup?search=0xA278B781FE4B2BDA&op=get"
},
--
Jakub Wilk
Jeremy Stanley
2018-08-26 14:54:06 UTC
Permalink
Post by Jakub Wilk
apt-key list | awk -F'[ /]+' '/expired:/{printf "apt-key adv --recv-keys --keyserver keys.gnupg.net %s\\n", $3}' | sudo sh
...
Post by Jakub Wilk
$ apt-key list | grep -A1 -w A15703C6
pub 4096R/A15703C6 2016-01-11 [expires: 2020-01-05]
[...]
If you're building infrastructure which needs to get data from off-site,
then consider whether or not you can provide template directives which
people can include in their command lists, and you then populate the
template with the correct current commands for that directive. Eg, if
I'm talking to Docker inside Circle CI, I don't set a bunch of variables
myself, I just say `setup_remote_docker` and let Circle CI figure out
which commands should be run. For "everything is a shell command"
setup, then perhaps `$CICMD_APT_KEYS_UPDATE` could be made available.
Indeed, as someone who helps design and run very large CI systems, I
can say with certainty that every extra request you make in your
jobs to retrieve something over a network connection is one more
false negative failure waiting to happen. The Internet is _not_
reliable, and it becomes obvious when you start looking at
connection failures and random API errors at scale from lots of
different places on the planet. If there's basically static data
that your job needs (especially public keys/certs) just bake it
directly into the job itself, and for things that change more often
than that cache as much of it as you can local to (or even directly
on the filesystems of) the systems which run those jobs.

Unfortunately a lot of this sort of silliness comes about because
people write CI jobs by translating their own developer environment
configuration documentation or user guides into scripts and don't
think about (or perhaps don't even understand in many cases) how
technologies like OpenPGP work.
--
Jeremy Stanley
Daniel Kahn Gillmor
2018-08-28 16:43:16 UTC
Permalink
The keyservers are a swamp; if you want to include one key, then include
the key as static data in your builds/CI configuration, so that it's
coming from a trusted source each time: your own data.
This is great advice, and not just for builds/CI configuration.

I made a similar suggestion recently to clean up the starttls-everywhere
datafile updater:

https://github.com/EFForg/starttls-everywhere/pull/65/commits/eb0a28e3fa141d4fb445c00df3ab7f3765ded859

In some ways, the keyserver network has done the OpenPGP community a
disservice, by encouraging OpenPGP users to refer to keys by
fingerprints (or even worse, by key IDs). While this is a useful
shorthand in some contexts, it's really a security/reliability
anti-pattern when it comes to secure programming.

--dkg
z***@spornkuller.de
2018-08-30 16:13:34 UTC
Permalink
Hi Daniel,
Post by Daniel Kahn Gillmor
In some ways, the keyserver network has done the OpenPGP community a
disservice, by encouraging OpenPGP users to refer to keys by
fingerprints (or even worse, by key IDs). While this is a useful
shorthand in some contexts, it's really a security/reliability
anti-pattern when it comes to secure programming.
I agree about the "key ID" part, but not about the "fingerprint" part.
Pinning a cryptographic hash over a public key isn't a security
antipattern by any strech of the imagination. Sure, you could argue that
the SHA-1 used by GPG isn't state-of-the-art anymore, but we're not
talking about collision attacks, but second preimage attacks. Far worse
for the attacker.

The way you phrased it, however, all applications of fingerprints/hashes
would be broken (SSH fingerprints, HPKP, etc.), regardless of the hash
function they use.

Cheers,
Joe
t
--
"A PC without Windows is like a chocolate cake without mustard."
v***@riseup.net
2018-08-31 12:18:03 UTC
Permalink
Post by z***@spornkuller.de
I agree about the "key ID" part, but not about the "fingerprint" part.
Pinning a cryptographic hash over a public key isn't a security
antipattern by any strech of the imagination. Sure, you could argue that
the SHA-1 used by GPG isn't state-of-the-art anymore, but we're not
talking about collision attacks, but second preimage attacks. Far worse
for the attacker.
True, yes, harder to brute-force a identical private key, than a key with an identical fingerprint.

However, if someone hadn't considered the possibility of a SHA1 collision attack, and a signature verification fails, despite the fingerprint they see matching, what % of GPG users would skip signature verification?
Perhaps due to confusion/self-doubt/inexperience/other.
Admittedly, this could be stepping into the realm of social engineering.
z***@spornkuller.de
2018-08-31 20:25:50 UTC
Permalink
Post by v***@riseup.net
Post by z***@spornkuller.de
I agree about the "key ID" part, but not about the "fingerprint" part.
Pinning a cryptographic hash over a public key isn't a security
antipattern by any strech of the imagination. Sure, you could argue that
the SHA-1 used by GPG isn't state-of-the-art anymore, but we're not
talking about collision attacks, but second preimage attacks. Far worse
for the attacker.
True, yes, harder to brute-force a identical private key, than a key with an identical fingerprint.
Hmm, not so sure. Let's say we're talking about RSA-4096, then we have a
security level of around 144 bit. Bruteforcing a second preimage SHA-1
(pretending it's an ideal hash function for a second) would have
complexity of around 159 bit. I.e., even for RSA-4096, it would be
easier to create the *identical* private key by factoring the modulus
(thus obviously creating a keypair with the identical fingerprint) than
just randomly generating keypairs and checking their private key hash.

I.e., my point was that for a given key that's uploaded with a fixed
fingerprint, we're not talking about 2^(b/2) collision complexity, but
2^(b-1) second preimage complexity.
Post by v***@riseup.net
However, if someone hadn't considered the possibility of a SHA1 collision attack, and a signature verification fails, despite the fingerprint they see matching, what % of GPG users would skip signature verification?
Perhaps due to confusion/self-doubt/inexperience/other.
Admittedly, this could be stepping into the realm of social engineering.
I think the attacker model that Daniel referred to was that someone
states "my key's fingerprint is XYZ" and someone downloading a forged,
same-fingerprint key from the keyserver.

Cheers,
Joe
--
"A PC without Windows is like a chocolate cake without mustard."
Jakub Wilk
2018-10-18 15:10:38 UTC
Permalink
Post by z***@spornkuller.de
I.e., my point was that for a given key that's uploaded with a fixed
fingerprint, we're not talking about 2^(b/2) collision complexity, but
2^(b-1) second preimage complexity.
Nitpicking, but for an ideal n-bit hash function, on avergage you need
2ⁿ (not 2ⁿ⁻¹) evalutations of the function to find the preimage.
--
Jakub Wilk
z***@spornkuller.de
2018-10-19 18:17:54 UTC
Permalink
Hey Jakub,
Post by Jakub Wilk
Nitpicking, but for an ideal n-bit hash function, on avergage you need
2ⁿ (not 2ⁿ⁻¹) evalutations of the function to find the preimage.
Huh, wow! I would also have had the gut-feeling of 2^(n-1) and had to
code a little program to verify the facts:

import hashlib
ps = [ ]
for q in range(20000):
for p in range(10000):
z = q | (p << 32)
z = int.to_bytes(z, length = 8, byteorder = "little")
if hashlib.md5(z).digest()[0] == 0:
break
ps.append(p)
print(sum(ps) / len(ps))

And indeed, you're absolutely correct. Thanks for the comment!

Have a great weekend,
Joe
Daniel Kahn Gillmor
2018-08-31 15:52:16 UTC
Permalink
Post by z***@spornkuller.de
I agree about the "key ID" part, but not about the "fingerprint" part.
Pinning a cryptographic hash over a public key isn't a security
antipattern by any strech of the imagination. Sure, you could argue that
the SHA-1 used by GPG isn't state-of-the-art anymore, but we're not
talking about collision attacks, but second preimage attacks. Far worse
for the attacker.
The way you phrased it, however, all applications of fingerprints/hashes
would be broken (SSH fingerprints, HPKP, etc.), regardless of the hash
function they use.
sorry, i think i wasn't clear enough about my complaint. I'm not
claiming that fingerprints are broken, or that second preimage attacks
against sha-1 are possible today. I'm saying that they're ill-suited to
many of the specific use cases where they show up.

If all i send you is a fingerprint, you *still* need to get the public
key somewhere. This is a point of potential failure.

In nearly every case where we're talking about automated signature
checking, the cost of shipping the public key instead of (or in addition
to) the fingerprint is negligible. and shipping just the fingerprint
introduces robustness and reliability problems for the signature
verification.

This is not to say that these sorts of things shouldn't consider looking
for updates to the keys that they have -- revocation checks, new
subkeys, etc all might be useful in some contexts. But there's no good
reason to ship a sophisticated, signature-verifying package with just a
fingerprint in it, when you could ship the whole key instead.

so, where are fingerprints useful? they're useful in *extremely
bandwidth-limited* cases, such as situations dealing with human
attention spans (e.g. fingerprint verification) or technically or
socially constrained channels like twitter, visible e-mail .signatures,
or SMS. They're also useful internally in programs that deal with many
keys, as concise references to known keys, or placeholders for unknown
keys.

Fingerprints are even arguably too long for most human attention spans,
so we need additional user research to look into better ways to do
verification that involves humans.

--dkg
z***@spornkuller.de
2018-08-31 20:54:47 UTC
Permalink
Post by Daniel Kahn Gillmor
In nearly every case where we're talking about automated signature
checking, the cost of shipping the public key instead of (or in addition
to) the fingerprint is negligible. and shipping just the fingerprint
introduces robustness and reliability problems for the signature
verification.
Ah, fair enough. Thanks for clarifying this, you're making good points.
The robustness issue is indeed something I completely disregarded.

Luckily, we've already arrived at a point where keys can be as short as
hash values. Ed25519 keys are 32 bytes, i.e., the same length as a
SHA256 hash. So there's that :-)

All the best,
Cheers,
Joe
--
"A PC without Windows is like a chocolate cake without mustard."
Jakub Wilk
2018-10-27 14:54:46 UTC
Permalink
Response from Travis CI:
https://blog.travis-ci.com/2018-08-29-addressing-reported-mitm-rce
Post by Jakub Wilk
On 2018-07-05, --force-yes was replaced with --allow-downgrades
https://github.com/travis-ci/travis-build/pull/1422
I'm not sure how could this change possibly work, because APT in the
Ubuntu versions Travis CI supports (precise, trusty) doesn't have
these options…
It did work, because Travis CI folks installed backported APT 1.2.X,
with support for these options...
Post by Jakub Wilk
https://github.com/travis-ci/travis-build/pull/1433
...but this fix had an off-by-one bug in version check, which made APT
1.2.X still use --force-yes. The bug was fixed soon after my advisory:
https://github.com/travis-ci/travis-build/commit/1ee43f25e45cad99c283b8fe53145617fd115dbb
Post by Jakub Wilk
https://github.com/travis-ci/travis-build/pull/1192
The code used 32-bit key ID to retrieve the key from the keyserver. I
https://github.com/travis-ci/travis-build/pull/1269
My proposed fix was to use "gpg --recv-key" with full fingerprint. But I
now discovered that even this is not resistant against MitM attacks:

https://dev.gnupg.org/T3398

"[...] modern gpg automatically applies an import screener that only
accepts OpenPGP certificates that have the given fingerprint [...]

However, it's possible for someone else to make a new OpenPGP
certificate that includes the key in question without knowledge of the
secret key (e.g. as a non-cross-signed subkey).

As a result, an attacker can bypass the import screener and inject new
primary keys into the keyring. [...]"
--
Jakub Wilk
Daniel Kahn Gillmor
2018-10-29 12:52:25 UTC
Permalink
Post by Jakub Wilk
My proposed fix was to use "gpg --recv-key" with full fingerprint. But I
https://dev.gnupg.org/T3398
"[...] modern gpg automatically applies an import screener that only
accepts OpenPGP certificates that have the given fingerprint [...]
It may be even worse than this, because the version of gpg used by
default in travis is not "modern gpg", it's either gnupg2
2.0.22-3ubuntu1.4 or gnupg 1.4.16-1ubuntu2.6. I don't think either of
these has the baseline "import screener" functionality, let alone a fix
for T3398 :(

--dkg
Jakub Wilk
2018-10-31 14:29:15 UTC
Permalink
Post by Daniel Kahn Gillmor
Post by Jakub Wilk
My proposed fix was to use "gpg --recv-key" with full fingerprint. But
https://dev.gnupg.org/T3398
"[...] modern gpg automatically applies an import screener that only
accepts OpenPGP certificates that have the given fingerprint [...]
It may be even worse than this, because the version of gpg used by
default in travis is not "modern gpg", it's either gnupg2
2.0.22-3ubuntu1.4 or gnupg 1.4.16-1ubuntu2.6. I don't think either of
these has the baseline "import screener" functionality
Ubuntu Precise and later releases have the import screener backported to
gnupg(2) packages:
https://bugs.launchpad.net/ubuntu/+source/gnupg2/+bug/1409117
--
Jakub Wilk
Loading...