JWKS URL not working

Questions
1

Hi, I got this strange problem that the ledger doesn’t work with the JWKS URL. Below is the error in the ledger-api log.

00:03:29.862 [ForkJoinPool-4-worker-1] WARN  c.d.ledger.api.auth.AuthServiceJWT$ - Authorization error: Could not verify JWT token: com.auth0.jwk.SigningKeyNotFoundException: Cannot obtain jwks from url http://someurl.com/jwt_auth0.html
00:03:29.877 [ForkJoinPool-4-worker-1] DEBUG com.daml.ledger.api.auth.Authorizer - No authenticated claims found in the request context. Returning UNAUTHENTICATED
00:03:29.892 [grpc-nio-worker-ELG-3-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xc4eedb1e, L:/127.0.0.1:6865 - R:/127.0.0.1:59948] OUTBOUND HEADERS: streamId=1 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-status: 16] padding=0 endStream=true

I tried invalid token, incorrect JWK url, incorrect content inside JWK url to identify the issue. However the error message didn’t change at all… Any idea how to find out which part goes wrong? The Jwk url; the Ledger API token or something else?

2

Hi @Frankie, I don’t know how to get better error reporting from the ledger. However from the error and my past experience running into similar errors I can only tell that it seems the URL you provided is not working.

That URL should always be the one returning the public key (that corresponds to the private key used for generating tokens) and the response format should be like this:

{"kty":"EC",
      "crv":"P-256",
      "x":"f83OJ3D2xF1Bg8vub9tLe1gHMzV76e8Tus9uPHvRVEU",
      "y":"x_FEzRu9m36HLN_tue659LNpXW6pCyStikYjKIWI5a0",
      "kid":"Public key used in JWS spec Appendix A.3 example"
     }

Source is from the RFC itself (Searchterm JWKS).

Did you check with a simple curl, that the endpoint you provide to the ledger is responding?

3

Further to @victor.pr.mueller’s point, the JWKS endpoint is supposed to return JSON. In your error message, you seem to use an HTML URL instead. You also hint at using Auth0; for that particular provider, the JWKS endpoint is at

https://<your-tenant>.us.auth0.com/.well-known/jwks.json
1 Like
4

I changed the original url for privacy reason…

Part of the url looks like this

image
image1362×608 72.5 KB

The url is hosted by our internal IAM. I compare it with the auth0 well-known jwks and they look very similar.

5

Another possiblity is that the key id which is used as reference in the token is wrong. What is the (protected) header of the token? E.g one useable key from the picture you posted has the kid wU3ifIIaLOUAReRB/FG6eM1P1QM= (because it is RS256).

6

I’ve opened an issue for your problem, so we can track progress on error message improvements.

7

Hi @Frankie,

If the JWKS endpoint you’re using returns the same shape of JSON as the Auth0 one, that should be good. Another thing you need to ensure is that the keys are referenced correctly. A JWKS endpoint returns a list of keys, along with a Key ID (kid) for each one.

Your JWT then needs to specify the same Key ID as the kid field in the header. For example, in the default example from jwt.io, the token is

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

The header is the first part, before the first dot. In this case it decodes to:

$ echo "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9" | base64 -d | jq .
{
  "alg": "HS256",
  "typ": "JWT"
}
$

You’d want to change it to add the key you’re using to sign your tokens, e.g. if you are using the key with "kid": "wU3ifIIaLOUAReRB/FG6eM1P1QM=" (the middle one on your screenshot), the header should be changed to:

$ echo '{"alg":"HS256","typ":"JWT","kid":"wU3ifIIaLOUAReRB/FG6eM1P1QM="}' | base64
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IndVM2lmSUlhTE9VQVJlUkIvRkc2ZU0xUDFRTT0ifQ==

Of course, you likely don’t have to care about the base64 encoding yourself; presumably whatever you’re using to generate the tokens can take care of that.

8

Just to provide some updates. We connect the ledger to a different IAM instance and it is working. It seems that it is something to do with the CA certificate. (We uses VMBC and creates our own CA)

The ledger doesn’t check the JWK url when it starts and we only find issues later. I’d suggest that 1) can we make a connection to the url and check when the ledger is started 2) provide more debug level messages when the JWK url is having issue.

1 Like
9

Bit of a shot in the dark here, but is your JWKS URL an HTTPS one, using a self-signed cert? If so, it may be blocked at the TLS verification stage, before it ever gets to the public key.

10

@Gary_Verhaegen thanks. We will try that. We suspect that it is probably something to do with iss. Do you know how iss is verified in the ledger?

11

I do not, no.

12

You can also take a look at our sample ex-secure-daml-infra as this goes through the details of JWKS and JWT and provides some examples. Repo also contains some scripts to check JWT and you can also use jwt.io. You may need to ensure the ledger trusts the root signing key of any non-public CA for any TLS actions (as Gary mentions).

Also note that HS256 is not recommended as this is a shared secret and open to brute force attacks.

1 Like
13

Thanks for that. I will check it up.

It turned out that the root CA was installed on the client node after we found it’s missing. However the container needs to be restarted to get it working… … :grinning_face_with_smiling_eyes: