Documenting or creating a TAP to describe keyid_hash_algorithms

[erickt]

Hello again,

Would you consider documenting, or possibly writing a TAP, to describe the keyid_hash_algorithms field in a key definition? As I upgrade go-tuf (and eventually rust-tuf) to the TUF 1.0 Draft, I am also updating it to be compatible with the metadata produced and consumed by this project. In order to do so, I need to also produce a compatible keyid_hash_algorithms field the metadata I produce. However, I'm not completely sure what I am supposed to do with this field when I receive it from python-tuf.

It appears this field originally came from secure-systems-lab/securesystemslib#37. As best as I can tell, it is present so that a client knows which algorithm was used to compute a keyid, so it can verify that a keyid is correct. Is this right, and are there other ways the keyid_hash_algorithms is used?

[awwad]

((Background note for others: KeyIDs matter in TUF. It is technically possible, I think, to construct a slower client for the reference implementation that could verify metadata without knowing how keyids were calculated, without substantial security issues. You could do this by -- to take signature checking as an example -- trying to verify a signature using all authorized keys rather than just the one with a matching keyid the signature lists.... I'm not recommending that.))

There's been some debate (#442 and #434, for example), about keyid_hash_algorithms and keyid calculation in general. Personally, I don't think the keyid_hash_algorithms field is of use, and I would like to remove it from keys, as I think it's probably sensible to assume across an implementation and deployment that all keyid calculations employ a specific hash function.

As for your question, there are two answers:

1. The hypothetical feature keyid_hash_algorithms is supposed to provide is the ability to limit the client's calculation of possible keyids for a key to only the algorithms listed there, to reduce collision space for keys or exclude, say, outdated hash algorithms from keyid calculations for that key. You could have keys that expect to only be hashed to produce keyids using SHA512 or something, I guess. (But really, if a hash algorithm is unacceptable, it should be unacceptable for all keys in the implementation, IMHO.)

2. The current code in the reference implementation includes keyid_hash_algorithms in the keyid calculation, which means that:

   • changing the values in that field for a key changes the key's keyid
   • removing the field changes all keys' keyids, which is a bit of a backward compatibility headache

Sorry for the headache. ^_^ Does that clear things up enough to ensure compatibility?

[awwad]

Oh, sorry, I didn't answer the TAP question:

If we thought that keyid_hash_algorithms was an important element of the system, it'd be a good thing to add to the spec (using a TAP or not), but I don't think we do.... We just need to discuss it a bit more at some point and (hopefully) remove it from the reference implementation in a breaking release. So I'd say this is a compatibility-with-the-reference-implementation issue rather than a specification issue, if that makes sense.

Please let me know if I missed anything else. :)

[erickt]

@awwad hello! Since it looks like 0.12 is coming up soon, has there been a discussion about removing keyid_hash_algorithms? I'm actually working on a conformance suite to make sure my go-tuf and rust-tuf are compatible with python-tuf, but it's a little unfortunate that the hard requirement of python-tuf to have keys contain keyid_hash_algorithms forces me to add support, even though you think it doesn't add much value.

Is it too late to get removing keyid_hash_algorithms into 0.12? Or to allow for a more transition path, allow python-tuf to parse metadata that doesn't contain keyid_hash_algorithms?

[trishankatdatadog]

Cc @lukpueh

[trishankatdatadog]

I'm wary of removing keyid_hash_algorithms in 0.12 for two reasons:

1. Security. There might be some unexpected issues we haven't thought about.
2. Backwards-compatibility. I'm not a big fan of removing harmless things at the expense of unnecessarily breaking old clients.

@JustinCappos thoughts?

[JustinCappos]

I believe our thoughts here were that we should support multiple hash algorithms in metadata and support updating of those algorithms so that a client which supports both can use them together unambiguously. We might as well make this a "party issue". Others? @mnm678 @SantiagoTorres

…

On Tue, Sep 24, 2019 at 2:58 PM Trishank K Kuppusamy < ***@***.***> wrote: I'm wary of removing keyid_hash_algorithms for two reasons: 1. Security. There might be some unexpected issues we haven't thought about. 2. Backwards-compatibility. I'm not a big fan of removing harmless things at the expense of unnecessarily breaking old clients. @JustinCappos <https://github.com/JustinCappos> thoughts? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#848>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGROD525DYMOTQT2J4I44TQLJPMBANCNFSM4HDU36MA> .

[erickt]

If there's security advantages to it, I'm fine implementing it in go-tuf/rust-tuf, I'm just not really sure how to do it. As best as I can tell, even though python-tuf has keyid_hash_algorithms=["sha256","sha512"], I don't see any metadata being generated with sha512 key ids, or anything using it. Is that a bug, or is there something I'm missing?

My first thought is that we're supposed to list keys multiple times, once for each item in the keyid_hash_algorithms. The problem I see is what happens if we have a TUF implementation that doesn't verify key ids, it just uses the key id listed in the signatures block to find the key in the root metadata. If an attacker steals one private key, they could sign metadata twice with the different key ids, thus defeating the threshold count. I tripped over this in go-tuf while I was migrating from the TUF 0.9 to 1.0 key structure (fortunately I caught it before committing it to the repo).

Regarding backwards compatibility, at the very least tuf 0.12 could do what I do, and treat the field as optional so python-tuf can parse metadata produced by another library that doesn't emit keyid_hash_algorithms. I'm doing this in go-tuf / rust-tuf.

[SantiagoTorres]

. As best as I can tell, even though python-tuf has keyid_hash_algorithms=["sha256","sha512"], I don't see any metadata being generated with sha512 key ids, or anything using it. Is that a bug, or is there something I'm missing?

I believe this happens because the runtime settings aren't the same as the ones being checked in, althoughI could be wrong.

[trishankatdatadog]

@erickt Yes, I see what you mean now. I think right now there isn't a good way in TUF to group together otherwise identical keys with different keyids. Until we solve this problem (perhaps in a backwards-incompatible TAP), I agree we should ignore keyid_hash_algorithms (it's not like we use SHA-512 keyids anywhere AFAICT?), simply assume SHA-256, and document it clearly in the spec.

[JustinCappos]

Okay, so the idea would be to have there be a TAP that addresses specifically this issue come out soon? We would make a (likely) backwards incompatible change at that point to add in multiple hash algorithms. Assuming I'm understanding this correctly, does anyone disagree with this course of action?

…

On Wed, Sep 25, 2019 at 3:39 PM Trishank K Kuppusamy < ***@***.***> wrote: @erickt <https://github.com/erickt> Yes, I see what you mean now. I think right now there isn't a good way in TUF to group together otherwise identical keys with different keyids. Until we solve this problem (perhaps in a backwards-incompatible TAP), I agree we should ignore keyid_hash_algorithms (it's not like we use SHA-512 keyids anywhere AFAICT?), simply assume SHA-256, and document it clearly in the spec. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#848>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGROD5VORJJN2CQVKISNKDQLO47DANCNFSM4HDU36MA> .

[SantiagoTorres]

spec-wise, I'm a little bit worried about backwards compatibility and what it entails on the semver angle...

[trishankatdatadog]

@SantiagoTorres True, it would break the reference implementation if we remove it from the code. What I'm saying is: keyid_hash_algorithms should not make it into the spec. Rather, we assume that the keyid is the SHA-256 of the canonical JSON of the key object. Other implementations SHOULD ignore keyid_hash_algorithms when they see it in metadata produced by the reference implementation.

This also means that while other implementations can consume reference implementation metadata w/o breaking, the converse is not true (because it silently needs keyid_hash_algorithms). We can update the reference implementation to assume some sane default if it is not present.

As for backwards-incompatibility produced by the TAP, we can worry about that later.

Is this all clear?

[erickt]

The complication is that the keyid_hash_algorithm is included in the keyid computed by python-tuf, so we couldn't just remove it from python-tuf without forcing the ids to be recalculated. I suppose python-tuf could be written in such a way that for some period of time it automatically compute two keyids for a key, one with, and one without the keyid_hash_algorithms. Then during verification make sure to only count a key once.

This is essentially the approach I'm taking in go-tuf, where I automatically trust both keyids, even if both aren't listed in the metadata, and only count the keys once during verification (looking at the code though I really should be ignoring unknown keyid algorithms though). Anyway, with this approach, once all the clients were updated to the latest version, you could switch towards generating the standard TUF-1.0 keyids. Perhaps 0.12 could implement parsing both IDs, and 0.13 could switch the code generation to remove keyid_hash_algorithms?

Also, perhaps the spec state machine should be updated to explicitly include rules on how to deduplicate keys. Would it be sufficient to dedup on the keyval field, or should we dedup on the (keytype, scheme, keyval) tuple?

[lukpueh]

Regarding what others have said about python-tuf only generating sha256 keyids, it indeed does not seem straight forward to generate keyids using any other algorithm, at least not with python-tuf/securesystemslib methods. The corresponding function is a barely documented "private" helper that has a hardcoded "sha256" default argument (see securesystemslib.keys._get_keyid()), and none of the functions that use it allow to change that argument. The only function that does call _get_keyids with different hash algorithms is securesystemslib.keys.format_metadata_to_key(). It does so by consulting a global setting for hash algorithms and returning the list of keyids as undocumented second return value, which makes me think that this feature was added a bit after the fact. That function is used by python-tuf to add the same key under different keyids (different hash algos) to the keydb (tuf's core trust management facility).

I don't think that the desired flexibility gained by supporting different hash algorithms for keyids, justifies the complexity of the (current) implementation, and the associated maintenance cost.

[lukpueh]

Btw. and since this has been an argument here, the next tuf release will already be backwards incompatible.

See first line of tentative changelog:

Add backwards incompatible TUF spec version checks (#842, #844, #854, #914)