What causes "Network closed for unknown reason" when trying to connect to ledger-api running through Docker?

Questions
, ,
1

Hi all,

When running Canton in my local machine with a given conf and bootstrap everything runs fine and I can connect to the ledger-api. But when I try to run Canton through a Dockerfile providing the same conf and bootstrap, it starts up but when my application tries to connect I get:
java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason

This repeats when my application retries, and I am currently unable to find any logging or other data explaining why this might happen. Any advice would be greatly appreciated!

Thanks!

2

Double-check that the Docker image opens the port for inbound connections, as opposed to just for connections from localhost?

3

I had a port issue before and in that case I was just getting Connection Refused, now I’m not getting that, so it does seem like the request is getting to the right place.

4

There are two levels that may need to be open here:

  • First, the Docker container itself needs to make the port available, which is typically done by using the -p option in the docker run command (though details may vary depending on how you run docker; e.g. if through Docker Compose it may be a ports: entry in the YAML file).
  • Second, the Canton server itself can be bound to either localhost or 0.0.0.0. In the former case, it sill drop connections from another machine, and the Docker container is (from a networking perspective) another machine than your host laptop.

If your current configuration (config file + defaults; I’m not very familiar with Canton defaults) binds the Canton server to localhost, that would work seamlessly when running on your machine but reject the connections when running in Docker.

5

My Dockerfile has the below as the final command:

ENTRYPOINT ["bin/canton", "-c", "/conf/canton.conf", "--bootstrap", "/conf/bootstrap-script.canton", "-Dcanton.participants.participant1.ledger-api.address=0.0.0.0"]

I am exposing port 5011 in my Dockerfile which is participant1’s ledger-api, and starting with -p 5011:5011. I believed that the last argument in the above entrypoint resolves the Canton server point.

6

Yes, that all looks correct to me. That’s as far as I was able to help, sorry.

I’ll see if I can find someone else to chime in.

1 Like
7

My comment might not be overly helpful, but it works for me:

docker run -p5011:5011 --rm -it digitalasset/canton-open-source:latest -Dcanton.participants.participant1.ledger-api.address=0.0.0.0 --config examples/01-simple-topology/simple-topology .conf --bootstrap examples/01-simple-topology/simple-ping.canton

The config setting gets picked up:
@ participant1.config.ledgerApi.address res1: String = "0.0.0.0"

And I can poke it with grpcurl:
canton@db-testing:~$ grpcurl -plaintext localhost:5011 list com.daml.ledger.api.v1.ActiveContractsService com.daml.ledger.api.v1.CommandCompletionService com.daml.ledger.api.v1.CommandService com.daml.ledger.api.v1.CommandSubmissionService com.daml.ledger.api.v1.LedgerConfigurationService com.daml.ledger.api.v1.LedgerIdentityService com.daml.ledger.api.v1.PackageService com.daml.ledger.api.v1.TransactionService com.daml.ledger.api.v1.VersionService com.daml.ledger.api.v1.admin.ConfigManagementService com.daml.ledger.api.v1.admin.MeteringReportService com.daml.ledger.api.v1.admin.PackageManagementService com.daml.ledger.api.v1.admin.ParticipantPruningService com.daml.ledger.api.v1.admin.PartyManagementService com.daml.ledger.api.v1.admin.UserManagementService grpc.health.v1.Health grpc.reflection.v1alpha.ServerReflection canton@db-testing:~$

8

Ohh … actually, the issue is: when you override configuration settings on the command line, you effectively have two ways to do that: JVM properties or Canton configuration flags.

So, you can use -C:

./bin/canton -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton -C canton.participants.participant1.ledger-api.port=1234

or -D

./bin/canton -Dcanton.participants.participant1.ledger-api.port=1234 -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton

The JVM property -D is ignored unless it is the first argument. The -C works everywhere. Looks like the docker example is still using -D.

9

That got it! Thanks Ratko, wasn’t aware of that!