Configuring TLS on Participant

Hi, I have a participant1_v1.conf file as shown below and I was configured the TLS option using self-signed certs

canton {
  participants {
    participant1 {
      admin-api.port = 5011
      admin-api.address = "0.0.0.0"
      ledger-api {
        address = "0.0.0.0"
        port = 5012
        tls {
          cert-chain-file = "/canton/canton-node/keys/participant1.crt"
          private-key-file = "/canton/canton-node/keys/participant1.pem"
          trust-collection-file = "/canton/canton-node/keys/root-ca.crt"
        }
      }
      storage = {
        type = postgres
        config {
          dataSourceClass = "org.postgresql.ds.PGSimpleDataSource"
          properties = {
            user = "canton"
            password = "canton"
            databaseName = "participant1"
            serverName = "participant1"
            portNumber = 5432
          }
        }
      }
    }
  }
}

while starting the node using below command, it is not starting as expected

docker run -it --name participant1 -p 5012:5012 --volume "$PWD/canton/canton-node:/canton/canton-node" --volume "$PWD/canton/canton-node/keys:/canton/canton-node/keys" -e JAVA_TOOL_OPTIONS="-Djavax.net.debug=all" digitalasset/canton-open-source --config /canton/canton-node/participant1_v1.conf --log-profile=container

logs from the container

Picked up JAVA_TOOL_OPTIONS: -Djavax.net.debug=ssl,handshake,certpath
INFO  c.d.c.CantonCommunityApp$ - Starting Canton version 2.7.6
INFO  c.d.c.CantonCommunityApp$ - Registered shutdown-hook.
javax.net.ssl|DEBUG|01|main|2025-02-10 15:03:31.515 UTC|SSLCipher.java:464|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|DEBUG|01|main|2025-02-10 15:03:31.519 UTC|SSLCipher.java:464|jdk.tls.keyLimits:  entry =  ChaCha20-Poly1305 KeyUpdate 2^37. CHACHA20-POLY1305:KEYUPDATE = 137438953472
DEBUG c.d.c.c.TlsServerConfig$ - Using 5 out of 11 Canton's default TLS ciphers
DEBUG c.d.c.e.CommunityEnvironment - participant1:admin-api=5011,ledger-api=5012
INFO  c.d.c.e.CommunityEnvironment - Deriving 4 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
DEBUG c.d.c.c.ExecutionContextMonitor - Monitoring canton-env-execution-context
javax.net.ssl|DEBUG|03|Finalizer|2025-02-10 15:03:31.933 UTC|SSLSocketImpl.java:578|duplex close of SSLSocket
javax.net.ssl|DEBUG|03|Finalizer|2025-02-10 15:03:31.933 UTC|SSLSocketImpl.java:1760|close the SSL connection (passive)
javax.net.ssl|DEBUG|03|Finalizer|2025-02-10 15:03:31.934 UTC|SSLSocketImpl.java:578|duplex close of SSLSocket
javax.net.ssl|DEBUG|03|Finalizer|2025-02-10 15:03:31.934 UTC|SSLSocketImpl.java:1760|close the SSL connection (passive)
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.CommunityEnvironment tid:b6a8d25871c644e261a87c1d4192b390 - Automatically starting all instances
INFO  c.d.c.e.ParticipantNodes - Setting up database schemas for participant1
INFO  c.d.c.r.DbStorage:participant1 tid:9b6b05a7cb46b6bd1410761c2ed77e0f - Overriding numThreads from 1 to 2 for the purpose of db migration, as flyway needs at least 2 threads.
DEBUG c.d.c.r.DbStorage:participant1 tid:9b6b05a7cb46b6bd1410761c2ed77e0f - Initializing database storage with config: Config(SimpleConfigObject({"connectionTimeout":5000,"dataSourceClass":"org.postgresql.ds.PGSimpleDataSource","initializationFailTimeout":1,"numThreads":2,"poolName":"slick-participant1-1","properties":{"databaseName":"participant1","password":"****","portNumber":5432,"serverName":"participant1","user":"canton"}}))
INFO  c.z.h.HikariDataSource - slick-participant1-1 - Starting...
INFO  c.z.h.HikariDataSource - slick-participant1-1 - Start completed.
DEBUG c.d.c.r.DbVersionCheck$:participant1 - Performing version checks
INFO  o.f.c.i.l.VersionPrinter - Flyway Community Edition 9.15.2 by Redgate
INFO  o.f.c.i.l.VersionPrinter - See release notes here: https://rd.gt/416ObMi
INFO  o.f.c.i.l.VersionPrinter -
INFO  o.f.c.i.d.b.BaseDatabaseType - Database: jdbc:postgresql://participant1:5432/participant1 (PostgreSQL 14.15)
DEBUG c.d.c.r.CommunityDbMigrations:participant1 tid:7a344acf2521067af5b45d763a041665 - Skip flyway migration on non-empty database
DEBUG c.d.c.r.CommunityDbMigrations:participant1 - Finished setting up database schemas after 717 milliseconds
INFO  c.z.h.HikariDataSource - slick-participant1-1 - Shutdown initiated...
INFO  c.z.h.HikariDataSource - slick-participant1-1 - Shutdown completed.
INFO  c.d.c.p.ParticipantNodeBootstrap:participant=participant1 - Starting admin-api services on CommunityAdminServerConfig(0.0.0.0,Some(5011),None,Some(KeepAliveServerConfig(40s,20s,20s)),10485760)
INFO  c.d.c.r.DbStorageSingle$:participant=participant1 - Creating storage, num-combined: 2
DEBUG c.d.c.r.DbStorage:participant=participant1 tid:3aa5013e1db133994292122afc850076 - Initializing database storage with config: Config(SimpleConfigObject({"connectionTimeout":5000,"dataSourceClass":"org.postgresql.ds.PGSimpleDataSource","initializationFailTimeout":1,"numThreads":2,"poolName":"slick-participant1-2","properties":{"databaseName":"participant1","password":"****","portNumber":5432,"serverName":"participant1","user":"canton"}}))
INFO  c.d.c.p.ParticipantNodeBootstrap:participant=participant1 - Node is not initialized yet. Performing automated default initialization.
DEBUG c.d.c.p.ParticipantNodeBootstrap:participant=participant1 tid:8919e861e4c7696102eb9f229959843a - Skipping existing NamespaceDelegation(12202e51b003..., SigningPublicKey(id = 12202e51b003..., format = Tink, scheme = Ed25519), true)
DEBUG c.d.c.p.s.CantonSyncService:participant=participant1 tid:a97f21f9b2bf0402250c1d92200ad7d3 - Invoke crash recovery or initialize active participant
INFO  c.d.c.p.s.CantonSyncService:participant=participant1 tid:a97f21f9b2bf0402250c1d92200ad7d3 - Recovering published timely rejections
INFO  c.d.c.p.s.CantonSyncService:participant=participant1 tid:a97f21f9b2bf0402250c1d92200ad7d3 - Publishing the unpublished events from the ParticipantEventLog

Hi Bhas, What symptoms are you experiencing that you are saying it didn’t start as expected? I don’t see anything immediately fishy in your logs.

Hi @bernhard the container is exiting and not even starting, and it supposed to open canton console as I am running in interactive mode

In that case, there should be more logs that what you shared. Is there any more?

I don’t see any logs after that, and also to note here that I am using canton version 2.7.9, is their any restriction that TLS work only in Enterprise Edition?

@Bhas, can you try this?

Start the node in daemon mode like this:

docker run -it --name participant1 -p 5012:5012 --volume "$PWD/canton/canton-node:/canton/canton-node" --volume "$PWD/canton/canton-node/keys:/canton/canton-node/keys" -e JAVA_TOOL_OPTIONS="-Djavax.net.debug=all" digitalasset/canton-open-source daemon --config /canton/canton-node/participant1_v1.conf --log-profile=container

Are the logs more complete in this case?

@WallaceKelly it is complaining about the certificate

2025-02-10 17:18:21,274 [canton-env-execution-context-20] ERROR c.d.c.p.a.LedgerApiService:participant=participant1 - Failed to create LedgerApiServer
java.lang.IllegalArgumentException: Input stream not contain valid certificates.
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:411)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:341)
        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:84)
        at io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:126)
        at com.digitalasset.canton.ledger.api.tls.TlsConfiguration.buildServersSslContext(TlsConfiguration.scala:105)
        at com.digitalasset.canton.ledger.api.tls.TlsConfiguration.$anonfun$server$2(TlsConfiguration.scala:74)
        at scala.util.Using$.resources(Using.scala:298)
        at com.digitalasset.canton.ledger.api.tls.TlsConfiguration.server(TlsConfiguration.scala:70)
        at com.digitalasset.canton.platform.apiserver.LedgerApiService.$anonfun$acquire$3(LedgerApiService.scala:36)
        at scala.Option.flatMap(Option.scala:283)
        at com.digitalasset.canton.platform.apiserver.LedgerApiService.$anonfun$acquire$1(LedgerApiService.scala:36)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
        at com.daml.executors.QueueAwareExecutorService$TrackingRunnable.run(QueueAwareExecutorService.scala:98)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.security.cert.CertificateException: found no certificates in input stream
        at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:98)
        at io.netty.handler.ssl.SslContext.toX509Certificates(SslContext.java:1200)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:409)
        ... 18 common frames omitted
2025-02-10 17:18:21,293 [canton-env-execution-context-22] INFO  com.zaxxer.hikari.HikariDataSource - daml.index.db.connection.api-server - Shutdown initiated...
2025-02-10 17:18:21,300 [canton-env-execution-context-22] INFO  com.zaxxer.hikari.HikariDataSource - daml.index.db.connection.api-server - Shutdown completed.
2025-02-10 17:18:21,301 [canton-env-execution-context-39] INFO  c.d.c.p.i.RecoveringIndexer:participant=participant1 - Stopping Indexer Server
2025-02-10 17:18:21,301 [canton-env-execution-context-39] INFO  c.d.c.p.i.h.KillSwitchCaptor:participant=participant1 - Shutdown called!
2025-02-10 17:18:21,302 [canton-env-execution-context-39] INFO  c.d.c.p.i.h.KillSwitchCaptor:participant=participant1 - Shutdown call delegated!
2025-02-10 17:18:21,305 [canton-env-execution-context-39] INFO  com.zaxxer.hikari.HikariDataSource - daml.index.db.connection.indexer - Shutdown initiated...
2025-02-10 17:18:21,322 [canton-env-execution-context-39] INFO  com.zaxxer.hikari.HikariDataSource - daml.index.db.connection.indexer - Shutdown completed.
2025-02-10 17:18:21,324 [ha-coordinator-0] DEBUG c.d.c.p.i.h.HaCoordinator$:participant=participant1 - Releasing periodic checker of the exclusive Indexer Main Lock on the main connection...
2025-02-10 17:18:21,324 [ha-coordinator-0] INFO  c.d.c.p.i.h.HaCoordinator$:participant=participant1 - Stepping down as leader, stopping DB connectivity polling
2025-02-10 17:18:21,324 [ha-coordinator-0] DEBUG c.d.c.p.i.h.HaCoordinator$:participant=participant1 - Step 7: Released periodic checker of the exclusive Indexer Main Lock on the main connection
2025-02-10 17:18:21,325 [ha-coordinator-0] DEBUG c.d.c.p.i.h.HaCoordinator$:participant=participant1 - Releasing main connection...
2025-02-10 17:18:21,325 [ha-coordinator-0] DEBUG c.d.c.p.i.h.HaCoordinator$:participant=participant1 - Step 8: Released main connection
2025-02-10 17:18:21,325 [ha-coordinator-0] INFO  c.d.c.p.i.h.HaCoordinator$:participant=participant1 - Stepped down as leader, IndexDB HA Coordinator shut down
2025-02-10 17:18:21,326 [canton-env-execution-context-39] INFO  c.d.c.p.i.RecoveringIndexer:participant=participant1 - Successfully finished processing state updates
2025-02-10 17:18:21,326 [canton-env-execution-context-39] INFO  c.d.c.p.i.RecoveringIndexer:participant=participant1 - Stopped Indexer Server
2025-02-10 17:18:21,329 [canton-env-execution-context-22] INFO  c.d.c.p.DispatcherState:participant=participant1 - Shutting down Ledger API offset dispatcher state.
2025-02-10 17:18:21,335 [canton-env-execution-context-22] INFO  c.d.c.p.DispatcherState:participant=participant1 - Ledger API offset dispatcher shutdown.
2025-02-10 17:18:21,336 [canton-env-execution-context-39] ERROR c.d.c.p.l.a.StartableStoppableLedgerApiServer:participant=participant1 - Failed to start ledger API server
java.lang.IllegalArgumentException: Input stream not contain valid certificates.

Which certificate it could be root-ca or participant cert?

My suspicion is that it is the participant cert. But, I agree, the logs do not say specifically.

If you are willing to create new certificates, you could use the scripts documented here and distributed here. I have had success using these for local, self-signed testing.

certs-common.sh
#!/bin/bash

# Copyright (c) 2025 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# architecture-handbook-entry-begin: GenTestCertsCmds
DAYS=3650

function create_key {
  local name=$1
  openssl genrsa -out "${name}.key" 4096
  # netty requires the keys in pkcs8 format, therefore convert them appropriately
  openssl pkcs8 -topk8 -nocrypt -in "${name}.key" -out "${name}.pem"
}

# create self signed certificate
function create_certificate {
  local name=$1
  local subj=$2
  openssl req -new -x509 -sha256 -key "${name}.key" \
              -out "${name}.crt" -days ${DAYS} -subj "$subj"
}

# create certificate signing request with subject and SAN
# we need the SANs as our certificates also need to include localhost or the
# loopback IP for the console access to the admin-api and the ledger-api
function create_csr {
  local name=$1
  local subj=$2
  local san=$3
  (
    echo "authorityKeyIdentifier=keyid,issuer"
    echo "basicConstraints=CA:FALSE"
    echo "keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment"
  ) > ${name}.ext
  if [[ -n $san ]]; then
    echo "subjectAltName=${san}" >> ${name}.ext
  fi
  # create certificate (but ensure that localhost is there as SAN as otherwise, admin local connections won't work)
  openssl req -new -sha256 -key "${name}.key" -out "${name}.csr" -subj "$subj"
}

function sign_csr {
  local name=$1
  local sign=$2
  openssl x509 -req -sha256 -in "${name}.csr" -extfile "${name}.ext" -CA "${sign}.crt" -CAkey "${sign}.key" -CAcreateserial  \
               -out "${name}.crt" -days ${DAYS}
  rm "${name}.ext" "${name}.csr"
}

function print_certificate {
  local name=$1
  openssl x509 -in "${name}.crt" -text -noout
}

# architecture-handbook-entry-end: GenTestCertsCmds
sample calls
# include certs-common.sh from config/tls
. ./certs-common.sh

# create root certificate such that we can issue self-signed certs
create_key "root-ca"
create_certificate "root-ca" "/O=TESTING/OU=ROOT CA/emailAddress=canton@digitalasset.com"
print_certificate "root-ca"

# create participant ledger-api certificate
create_key "ledger-api"
create_csr "ledger-api" "/O=TESTING/OU=PARTICIPANT/CN=localhost/emailAddress=canton@digitalasset.com" "DNS:localhost,IP:0.0.0.0"
sign_csr "ledger-api" "root-ca"
print_certificate "ledger-api"

(HT: @sormeter)

Hi @WallaceKelly, Thanks, I am able to successfully spin up the node by configuring the tls certs as per your instruction.
I am trying to connect to domain from participant console using below command

participant1.domains.connect("mydomain","https://<domain-ip>:5022")

but it is giving below error

DEBUG c.d.c.a.a.c.GrpcCtlRunner tid:ccebee38cbe759037625c327dd6c0e81 - Retry has not been configured for GrpcClientError, giving up.
ERROR c.d.c.e.CommunityConsoleEnvironment - Request failed for participant1.
  GrpcClientError: UNAVAILABLE/DOMAIN_IS_NOT_AVAILABLE(1,ccebee38): Cannot connect to domain Domain 'mydomain'
  Request: ConnectDomain(Domain 'mydomain',false)
  CorrelationId: ccebee38cbe759037625c327dd6c0e81
  RetryIn: 1 second
  Context: HashMap(participant -> participant1, domain -> mydomain, reason -> Request failed for mydomain. Is the server running? Did you configure the server address as 0.0.0.0? Are you using the right TLS settings? (details logged as DEBUG)
  GrpcServiceUnavailable: UNAVAILABLE/Network closed for unknown reason
  Request: get domain id and sequencer id, alias -> Domain 'mydomain', tid -> ccebee38cbe759037625c327dd6c0e81)
  Command ParticipantAdministration$domains$.connect invoked from cmd1.sc:1
com.digitalasset.canton.console.CommandFailure: Command execution failed.

here is my conf file details

Slightly dated but this goes through the setup for TLS on Canton: GitHub - digital-asset/ex-secure-canton-infra: Reference deployment of a secure Canton infrastructure (PKI, JWT, HA)

You need to specify the root cert in the domain connect call otherwise the cert is untrusted by Canton.

Something like the following:

participants.local.foreach(_.domains.connect(“domain”, “https://0.0.0.0:4401”, certificatesPath = “./certs/domain/intermediate/certs/ca-chain.cert.pem”))

1 Like

Hi @nycnewman, I still have same error in my console

This error is usually a couple of things:

  • TLS certificates are not trusted (thus need to pass the root certificate in as a parameter to connect command)
  • Domain is listening on a different port - looks like you are using the right port though
  • On MacOS in Docker, localhost is literally just the localhost within the Linux VM that the container is running. You have to use “host.docker.internal” instead of locahost to connect to another container listening on loopback.
  • Finally check that the sequencer is exposing and binding correctly to the 5022 port. You should be able to use tools like grpcurl to check the reflection endpoint of the service