Pebble stores issued serials in one encoding and looks them up in another when validating an ARI replaces field, so a newOrder that carries a spec-compliant certID gets rejected with a 500 whenever the serial's most significant byte is >= 0x80.
Tested against v2.10.1, and main (200b05c) still has both call sites unchanged.
What happens
POST /order-plz
{"identifiers": [{"type":"dns","value":"*.example.com"}],
"replaces": "RTAINB8QAcdWz54id2aMkYdCnDI.AIWf94paRX4"}
HTTP 500
{
"type": "urn:ietf:params:acme:error:serverInternal",
"detail": "could not find an order for the given certificate: could not find order resulting in the given certificate serial number",
"status": 500
}
The certificate is there, it was issued by this same Pebble process minutes earlier, and the same certID works fine against renewalInfo.
AIWf94paRX4 is base64url for 00 85 9F F7 8A 5A 45 7E, i.e. the serial 859FF78A5A457E with the DER sign byte in front. RFC 9773 section 4.1 asks for the DER encoded value bytes of serialNumber, and DER prepends 0x00 when the first byte has the high bit set, so a client that produces that byte is doing the right thing. Boulder accepts it.
Where the two encodings diverge
Stored key, from big.Int.Bytes(), which is the minimal magnitude with no sign byte:
// ca/ca.go:166
hexSerial := hex.EncodeToString(cert.SerialNumber.Bytes())
Lookup key, from the raw base64url decoded bytes, sign byte included:
// core/types.go:232 (NewCertID)
id: hex.EncodeToString(serial),
validateReplacementOrder then passes that through CertID.SerialHex():
// wfe/wfe.go:1695
originalOrder, err := wfe.db.GetOrderByIssuedSerial(certID.SerialHex())
so the map in db/memorystore.go is keyed on 859ff78a5a457e and queried for 00859ff78a5a457e, and misses.
What makes me fairly confident this is unintentional rather than a deliberate strictness choice: the other consumer of the same parsed certID normalizes correctly, because it uses the *big.Int rather than the hex string.
// wfe/wfe.go:1931 (RenewalInfo)
cert := wfe.db.GetCertificateBySerial(certID.SerialNumber)
That is also why this is easy to miss in practice. The ARI GET succeeds, the client gets a renewal window back and happily proceeds, and only the newOrder that follows blows up.
How often
makeSerial draws from [0, math.MaxInt64). An 8 byte magnitude can never have a top byte >= 0x80, since that would need a value of at least 2^63, so this is driven by the 7 byte case and works out at about 2^55/2^63, or 1 in 256. That is rare enough to look like an intermittent client bug in CI, which is how I ended up here.
Reproducing it
Since the natural rate is 1 in 256, the quickest way is to force the condition in ca/ca.go:
func makeSerial() *big.Int {
serial, err := rand.Int(rand.Reader, big.NewInt(1<<55))
if err != nil {
panic(fmt.Sprintf("unable to create random serial number: %s", err.Error()))
}
serial.Or(serial, new(big.Int).Lsh(big.NewInt(1), 55)) // [2^55, 2^56): 7 bytes, top byte >= 0x80
return serial
}Then issue a certificate with any ARI aware client and renew it. With acme.sh (which adds the sign byte per RFC 9773) I get a clean A/B:
serial=6D569DBC36BFE7 top byte < 0x80 renew rc=0 "Cert success."
serial=859FF78A5A457E top byte >= 0x80 renew rc=1 "could not find an order for the given certificate"
Dropping the Or above, so the top byte lands under 0x80, makes it pass every time.
Possible fix
Normalizing in NewCertID looks like the smallest change, something like keying id off new(big.Int).SetBytes(serial).Bytes() so both sides agree on the minimal encoding. That also keeps SerialHex consistent with what GetCertificateBySerial already does. Happy to send a PR if that is the direction you would want, or if you would rather have the fix on the db side I can do that instead.
One side note while you are in there: the resulting problem document is serverInternal, and at least one client only retries without replaces when the response mentions malformed, alreadyReplaced, replaces or ARI. So the mismatch is currently fatal rather than degrading to a plain order. A malformed problem here would be friendlier regardless of whether the encoding is fixed.
Pebble stores issued serials in one encoding and looks them up in another when validating an ARI
replacesfield, so a newOrder that carries a spec-compliant certID gets rejected with a 500 whenever the serial's most significant byte is >= 0x80.Tested against v2.10.1, and
main(200b05c) still has both call sites unchanged.What happens
The certificate is there, it was issued by this same Pebble process minutes earlier, and the same certID works fine against
renewalInfo.AIWf94paRX4is base64url for00 85 9F F7 8A 5A 45 7E, i.e. the serial859FF78A5A457Ewith the DER sign byte in front. RFC 9773 section 4.1 asks for the DER encoded value bytes ofserialNumber, and DER prepends0x00when the first byte has the high bit set, so a client that produces that byte is doing the right thing. Boulder accepts it.Where the two encodings diverge
Stored key, from
big.Int.Bytes(), which is the minimal magnitude with no sign byte:Lookup key, from the raw base64url decoded bytes, sign byte included:
validateReplacementOrderthen passes that throughCertID.SerialHex():so the map in
db/memorystore.gois keyed on859ff78a5a457eand queried for00859ff78a5a457e, and misses.What makes me fairly confident this is unintentional rather than a deliberate strictness choice: the other consumer of the same parsed certID normalizes correctly, because it uses the
*big.Intrather than the hex string.That is also why this is easy to miss in practice. The ARI GET succeeds, the client gets a renewal window back and happily proceeds, and only the newOrder that follows blows up.
How often
makeSerialdraws from[0, math.MaxInt64). An 8 byte magnitude can never have a top byte >= 0x80, since that would need a value of at least 2^63, so this is driven by the 7 byte case and works out at about 2^55/2^63, or 1 in 256. That is rare enough to look like an intermittent client bug in CI, which is how I ended up here.Reproducing it
Since the natural rate is 1 in 256, the quickest way is to force the condition in
ca/ca.go:Then issue a certificate with any ARI aware client and renew it. With acme.sh (which adds the sign byte per RFC 9773) I get a clean A/B:
Dropping the
Orabove, so the top byte lands under 0x80, makes it pass every time.Possible fix
Normalizing in
NewCertIDlooks like the smallest change, something like keyingidoffnew(big.Int).SetBytes(serial).Bytes()so both sides agree on the minimal encoding. That also keepsSerialHexconsistent with whatGetCertificateBySerialalready does. Happy to send a PR if that is the direction you would want, or if you would rather have the fix on the db side I can do that instead.One side note while you are in there: the resulting problem document is
serverInternal, and at least one client only retries withoutreplaceswhen the response mentionsmalformed,alreadyReplaced,replacesorARI. So the mismatch is currently fatal rather than degrading to a plain order. Amalformedproblem here would be friendlier regardless of whether the encoding is fixed.