What causes "Network closed for unknown reason" when trying to connect to ledger-api running through Docker?

Hi all,

When running Canton in my local machine with a given conf and bootstrap everything runs fine and I can connect to the ledger-api. But when I try to run Canton through a Dockerfile providing the same conf and bootstrap, it starts up but when my application tries to connect I get:
java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason

This repeats when my application retries, and I am currently unable to find any logging or other data explaining why this might happen. Any advice would be greatly appreciated!


Double-check that the Docker image opens the port for inbound connections, as opposed to just for connections from localhost?

I had a port issue before and in that case I was just getting Connection Refused, now I’m not getting that, so it does seem like the request is getting to the right place.

There are two levels that may need to be open here:

  • First, the Docker container itself needs to make the port available, which is typically done by using the -p option in the docker run command (though details may vary depending on how you run docker; e.g. if through Docker Compose it may be a ports: entry in the YAML file).
  • Second, the Canton server itself can be bound to either localhost or In the former case, it sill drop connections from another machine, and the Docker container is (from a networking perspective) another machine than your host laptop.

If your current configuration (config file + defaults; I’m not very familiar with Canton defaults) binds the Canton server to localhost, that would work seamlessly when running on your machine but reject the connections when running in Docker.

My Dockerfile has the below as the final command:

ENTRYPOINT ["bin/canton", "-c", "/conf/canton.conf", "--bootstrap", "/conf/bootstrap-script.canton", "-Dcanton.participants.participant1.ledger-api.address="]

I am exposing port 5011 in my Dockerfile which is participant1’s ledger-api, and starting with -p 5011:5011. I believed that the last argument in the above entrypoint resolves the Canton server point.

Yes, that all looks correct to me. That’s as far as I was able to help, sorry.

I’ll see if I can find someone else to chime in.

1 Like

My comment might not be overly helpful, but it works for me:

docker run -p5011:5011 --rm -it digitalasset/canton-open-source:latest -Dcanton.participants.participant1.ledger-api.address= --config examples/01-simple-topology/simple-topology .conf --bootstrap examples/01-simple-topology/simple-ping.canton

The config setting gets picked up:
@ participant1.config.ledgerApi.address res1: String = ""

And I can poke it with grpcurl:
canton@db-testing:~$ grpcurl -plaintext localhost:5011 list com.daml.ledger.api.v1.ActiveContractsService com.daml.ledger.api.v1.CommandCompletionService com.daml.ledger.api.v1.CommandService com.daml.ledger.api.v1.CommandSubmissionService com.daml.ledger.api.v1.LedgerConfigurationService com.daml.ledger.api.v1.LedgerIdentityService com.daml.ledger.api.v1.PackageService com.daml.ledger.api.v1.TransactionService com.daml.ledger.api.v1.VersionService com.daml.ledger.api.v1.admin.ConfigManagementService com.daml.ledger.api.v1.admin.MeteringReportService com.daml.ledger.api.v1.admin.PackageManagementService com.daml.ledger.api.v1.admin.ParticipantPruningService com.daml.ledger.api.v1.admin.PartyManagementService com.daml.ledger.api.v1.admin.UserManagementService grpc.health.v1.Health grpc.reflection.v1alpha.ServerReflection canton@db-testing:~$

Ohh … actually, the issue is: when you override configuration settings on the command line, you effectively have two ways to do that: JVM properties or Canton configuration flags.

So, you can use -C:

./bin/canton -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton -C canton.participants.participant1.ledger-api.port=1234 

or -D

./bin/canton -Dcanton.participants.participant1.ledger-api.port=1234 -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton 

The JVM property -D is ignored unless it is the first argument. The -C works everywhere. Looks like the docker example is still using -D.

That got it! Thanks Ratko, wasn’t aware of that!