Enable TLS Sandbox <-> Navigator

Questions Features & Bugs
,
1

Hi, we are trying to enable TLS for the communication between our java app and the ledger. Due the fact that we were facing a too generic issue in the java app, we decided to go one step backwards trying to check the same behaviour between the Sandbox and the Navigator(following the samples in the documentation). After testing Sandbox <-> Navigator communication we realized that there is a similar issue like the one using the java app.

We are using daml version 1.11.1, and we are executing this:
daml start --sandbox-option "--pem=sandbox.key" --sandbox-option "--crt=sandbox.crt" --navigator-option "--pem=sandbox.key" --navigator-option "--crt=sandbox.crt" --navigator-option "--cacrt=sandbox.crt"

and the output is:

INFO: Listening on localhost:6865 over TLS.
…
Exception in thread "main" io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:534)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
        at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:464)
        at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:428)
        at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:461)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/0:0:0:0:0:0:0:1:6865
Caused by: java.net.ConnectException: Connection refused: no further information
        at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)

We are assuming that mTLS(instead of TLS) is used when daml start is executed, besides for ease this test we are reusing same private key and cert in both servers, could you please advice about what may be wrong or even if it may be a daml version issue?

1 Like
2

--cacrt needs to point to your CA certificate not the ledger certificate. You are also not specifying the CA certificate on the server side used to verify the client certificate via --sandbox-option --cacrt=… on the server side.

i recommend reading @nycnewman’s excellent blogpost on how to set this up Secure Daml Infrastructure - Part 1 PKI and certificates.

1 Like
3

After adding the missing param to the command, we realized that we are able to setup TLS between Sandbox and Navigator if we execute them separately:

Terminal 1 -> daml sandbox darfile.dar --pem sandbox.key --crt sandbox.crt --cacrt ca.crt
Terminal 2 -> daml navigator server localhost 6865 --pem nav.key --crt nav.crt --cacrt ca.crt

But if we try to execute it using daml start using the same options, navigator fails to reach the sandbox. It looks like that navigator is using 127.0.0.1 instead of localhost, and may be the reason of the failure. Even after checking the help and documentation, we coudn’t manage to override that value using options.

In order to discard the main issue could you please advice if there is a way or not to do so?

4

I am able to get TLS working via daml start and I did check that it connects to localhost. There are two things you need to pay attention to:

  1. daml start also runs daml script if you have an init-script field in your daml.yaml. For testing you can just remove this. If you want to keep it you also need to set TLS flags for that.
  2. daml start also runs daml json-api. You can disable that via --json-api-port none or again set tls flags for that as well.

In the end, something like the following works if you deleted the init-script line:

daml start --sandbox-option --cacrt=certs/ca.crt --sandbox-option --pem=certs/server.pem --sandbox-option --crt=certs/server.crt --navigator-option --cacrt=certs/ca.crt --navigator-option --pem=certs/client.pem --navigator-option --crt=certs/client.crt

That said, daml start is really just a convenience for development. Once you get to the point where you worry about TLS I think you’re probably better off starting the individual components manually.

1 Like