We have recently encountered an issue while uploading DAR files to my Canton participant, and I wanted to share my experience .
Issue:
While uploading multiple DAR files to my Canton participant, one of the uploads failed with the following error:
An exception was thrown during the upload-dar command
- GRPCIOTimeout
This occurred while uploading sor-mvp-init-1.2.22.dar
, which is 5.1MB in size. The error message suggests that the DAR file size might exceed the gRPC maximum message size, which is typically set to 4MB by default.
My Setup:
- Participant Configuration:
ledger-api {
address = "0.0.0.0"
port = 4001
postgres-data-source.synchronous-commit = off
max-inbound-message-size = 209715200
command-service.max-commands-in-flight = 100
max-contract-state-cache-size = 100
max-contract-key-state-cache-size = 100
max-transactions-in-memory-fan-out-buffer-size = 100
}
- DAR File Sizes:
sor-mvp-data-dependencies-1.2.6.dar
( Uploaded successfully)sor-mvp-models-1.2.22.dar
(Uploaded successfully)sor-mvp-integration-acord-1.2.0.dar
( Uploaded successfully)sor-mvp-init-1.2.22.dar
( Failed due to GRPCIOTimeout)
Possible Causes:
1. Increase gRPC Maximum Message Size
Since my participant configuration already sets max-inbound-message-size = 209715200
(200MB), the issue might not be the default 4MB limit
I have set the max-inbound-message-size = 209715200 as we have discussed
and we are using port forward cmd while we upload the dar
daml ledger upload-dar --host=127.0.0.1 --port=4001 /Users/zinnia_india/Documents/qa_deployment/deleting_daml_backup/SOR-DAML/.packages/init/.daml/dist/sor-mvp-init-1.2.22.dar
Port forward cmd
kubectl port-forward svc/partcipant-service-name 4001:4001 -n canton
and this cmd says connection lost and ledger participant pod get restarted when we upload dar