That's not arbitrary: it's X9.62 format, even though that's not called out anywhere.
See:
American National Standards Institute (ANSI), ANSI X9.62-2005: The Elliptic Curve Digital Signature Algorithm (ECDSA), 2005.
Post
Replies
Boosts
Views
Activity
And the answer is, the public key needs to be serialized as an X9.62 Uncompressed Point in order for the digest to match... this is not called out anywhere.
Found a few examples on GitHub, but what's worrying is that they don't closely follow the steps outlined in Validating Apps that Connect to Your Server.
In particular, this part:
Create the SHA256 hash of the public key in credCert, and verify that it matches the key identifier from your app.
But this is far from being unambiguous. I'd have thought that this referred to the entire DER of the public key:
% openssl x509 -in k -noout -pubkey | openssl asn1parse -inform pem -dump
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 09 1a ae 9f d2 0b-89 e6 6b ab 68 3e 70 e1 ..........k.h>p.
0010 - 6d 0f b1 2f 8b 4b bd c9-d2 54 ec 15 2c b4 fc 4c m../.K...T..,..L
0020 - 8d fb e1 49 0d 90 34 80-10 82 08 6c 49 58 7e 2c ...I..4....lIX~,
0030 - 5b 90 2b 80 2d 1f f3 e9-36 59 51 d2 3e 1d d2 f8 [.+.-...6YQ.>...
0040 - 75 e3 u.
But from what I can tell, the checksum is only on the BIT STRING itself from looking at other examples... but even then, extracting just this portion and checksumming only that, I can't match the keyId.
Hmm... Can't seem to format block code...