Issue enabling TLS for DAML JSON Api

Questions
, ,
1

I’m running DAML JSON Api and DAML on SQL process as two different pods in a kubernetes cluster. Trying to enable TLS communication using self signed certs for both the processes.

Following commands to generate the certs,

openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -out ca.crt -subj '/CN=[0.0.0.0.ca](http://0.0.0.0.ca/)' -days 3650# 
Generate server key, csr and crt
openssl genrsa -out daml-on-sql.key 4096
openssl pkey -in daml-on-sql.key -out daml-on-sql.pem
openssl req -new -key daml-on-sql.key -out daml-on-sql.csr -subj '/CN=0.0.0.0.server'
openssl x509 -req -in daml-on-sql.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out daml-on-sql.crt -extfile openssl-extensions.cnf -extensions req_ext -days 3650# Generate client key, csr and crt

openssl genrsa -out daml-json-api.key 4096
openssl pkey -in daml-json-api.key -out daml-json-api.pem
openssl req -new -key daml-json-api.key -out daml-json-api.csr -subj '/CN=0.0.0.0.client'
openssl x509 -req -in daml-json-api.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out daml-json-api.crt -extfile openssl-extensions.cnf -extensions req_ext -days 3650

Added the kubernetes service name of daml-on-sql process as a subject alternative name in the extensions file while creating the certs.

Have set the cert path, pem path and ca cert path for the containers in the deployment yaml.

The DAML on sql is running fine as per the logs below,

INFO: Listening on 0.0.0.0:6865 over TLS. (context: {participantId=sandbox-participant})
INFO: Initialized sandbox version 1.2.0 with ledger-id = yyy, port = 6865, dar file = List(yyy.dar), time mode = wall-clock time, ledger = postgres, auth-service = AuthServiceWildcard$, contract ids seeding = strong

The DAML JSON seems to have started ok, and it has loaded the existing package id’s as well,

11:45:09.314 [http-json-ledger-api-akka.actor.default-dispatcher-9] INFO com.daml.http.Main$ - Started server: ServerBinding(/0.0.0.0:7575)

But the issue is I’m unable to access the endpoint using https.

When I run a curl command inside the container, I get the below error

curl -v https://localhost:7575/v1/query
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 7575 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Also I see the below warning in the JSON Api logs,

[WARN] [08/12/2020 11:45:53.610] [http-json-ledger-api-akka.actor.default-dispatcher-8] [akka.actor.ActorSystemImpl(http-json-ledger-api)] Illegal request, responding with status '400 Bad Request': Unsupported HTTP method: The HTTP method started with 0x16 rather than any known HTTP method. Perhaps this was an HTTPS request sent to an HTTP endpoint?
2 Likes
2

Hi @senrav, welcome to the forum!

The JSON API deliberately does not provide TLS support for its endpoints. The CLI parameters you used enable TLS for the connect between the JSON API and the ledger not for the endpoints of the JSON API itself.

To secure the JSON API itself, you should put it behind a reverse proxy like Nginx. The JSON API will even emit a warning by default if you do not do that and your requests do not have an X-Forwarded-Proto header with an https url. (That warning is disabled in daml start since that is only intended for development).

1 Like
3

Thanks @cocreature, for a quick reply.

I’m having a similar setup, our UI is hosted using nginx and we have a reverse proxy configuration for our backend service and the JSON Api. The ingress will be like https://FQDN/v1

Is the X-Forwarded-Proto header, a must for the JSON Api for this to work? This is missing in my configuration, I can add this and check

Just to confirm, in this setup, the cert and pem parameters for the JSON Api can use the same trusted cert which I have created for the UI, as the forwarded hostname will be the UI FQDN. I’ll be using a self signed cert for the DAML-on-Sql, and it’s ca cert configured in the JSON Api.

Is that warning (in the issue description above) something to worry about?

1 Like
4

You are expected to provide either X-Forwarded-Proto or Forwarded with an https URL. If you don’t, you will get a warning but things will still work.

The cert and pem parameters passed to the JSON API are only used as client certificates for mutual TLS for the connection to DAML-on-SQL. They will be validated by DAML-on-SQL according to the value of --cacrt (passed to DAML-on-SQL) or the default trusted root CAs if you do not pass --cacrt.

1 Like
5

@senrav It is something to worry about. We made it a warning instead of an error to preserve compatibility for a version transition, but it has been warned about for a while, so could be changed to an error in any release, as mentioned in the PR that made it a warning.

If you have reasonably straightforward means for fixing the warning, I think it best to apply them now. Otherwise, you will have to worry about fixing it as part of an upgrade, at which point your mind may be on other issues.

edit: I refer to the warning about the Forwarded header above. Regarding the “The HTTP method started with 0x16” warning, that’s just fallout from trying an https request directly to JSON API, which as already discussed doesn’t work. You can think of it as an additional notice that such requests must go through reverse proxy, as you have already set up.

2 Likes
6

If you are sure you’re setup is secure, you can also disable the warning by setting --allow-insecure-tokens. This less safe than setting the correct headers on your reverse proxy so only advised of headers are not an option.

1 Like
7

A post was split to a new topic: How do I produce tokens for an authorized ledger?

8

If nginx is doing TLS termination itself, you will need to add the header as part of the nginx configuration, probably as proxy_set_header.

If nginx is not doing TLS termination, it is very likely whatever process is doing it will already have added a correct header for this, and AFAICT nginx will by default proxy that unchanged. You may want to look at the $proxy_add_x_forwarded_for variable if you want to add nginx to the chain of proxies in that header.

You’ll probably want to take a look at the nginx API doc for proxy while working on this.

9

Thanks for the comments and suggestions. I solved it by using nginx as a sidecar container, enabling tls for the nginx and use localhost http to communicate with the json api.

1 Like
10

Hi Senrav, How do i test it to know https is enabled for JSON API? Do we need to browse https://localhost:7575? But it doesn’t work