DAR Uploaded, but No Templates in Daml Navigator (Minikube + Canton Setup)

We have recently encountered an issue while uploading DAR files to my Canton participant, and I wanted to share my experience .

Issue:

While uploading multiple DAR files to my Canton participant, one of the uploads failed with the following error:

An exception was thrown during the upload-dar command
- GRPCIOTimeout

This occurred while uploading sor-mvp-init-1.2.22.dar , which is 5.1MB in size. The error message suggests that the DAR file size might exceed the gRPC maximum message size, which is typically set to 4MB by default.

My Setup:

  • Participant Configuration:
ledger-api {
    address = "0.0.0.0"
    port = 4001
    postgres-data-source.synchronous-commit = off
    max-inbound-message-size = 209715200
    command-service.max-commands-in-flight = 100
    max-contract-state-cache-size = 100
    max-contract-key-state-cache-size = 100
    max-transactions-in-memory-fan-out-buffer-size = 100
}
  • DAR File Sizes:
    • sor-mvp-data-dependencies-1.2.6.dar ( Uploaded successfully)
    • sor-mvp-models-1.2.22.dar (Uploaded successfully)
    • sor-mvp-integration-acord-1.2.0.dar ( Uploaded successfully)
    • sor-mvp-init-1.2.22.dar ( Failed due to GRPCIOTimeout)

Possible Causes:

1. Increase gRPC Maximum Message Size

Since my participant configuration already sets max-inbound-message-size = 209715200 (200MB), the issue might not be the default 4MB limit

I have set the max-inbound-message-size = 209715200 as we have discussed
and we are using port forward cmd while we upload the dar

daml ledger upload-dar --host=127.0.0.1 --port=4001 /Users/zinnia_india/Documents/qa_deployment/deleting_daml_backup/SOR-DAML/.packages/init/.daml/dist/sor-mvp-init-1.2.22.dar

Port forward cmd

kubectl port-forward svc/partcipant-service-name 4001:4001 -n canton

and this cmd says connection lost and ledger participant pod get restarted when we upload dar