How do I resolve time out error for DAML?

Questions
1

Hi all!

I’m trying to run a daml script to create contracts on our test ledger using oracle for storage and using java SDK(daml-lf-1.8-archive-java-proto) connect for expose interfaces to the front-end.

Sometimes I got TIME OUT error when creating a contract. I’m not sure if it’s really a daml script issue or an issue with the network environment.

Would appreciate your advice!

request response:

java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: REQUEST_TIME_OUT(XXXX): Timed out while awaiting for a completion corresponding to a command submission.

springboot error log:

ERROR 37172 --- [nio-8090-exec-9] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalStateException: getOutputStream() has already been called for this response] with root cause

java.lang.IllegalStateException: getOutputStream() has already been called for this response

daml error log:

[canton-env-execution-context-22] INFO  c.d.p.a.services.ApiCommandService - REQUEST_TIME_OUT(3,f19c3ae8): Timed out while awaiting for a completion corresponding to a command submission., context: {participant: "XXX", readAs: [], parties: ["XXX"], submissionId: "XXX", actAs: ["XXX"], commandId: "XXX", err-context: "{definiteAnswer=false, location=ErrorFactories.scala:77}"}
[canton-env-execution-context-22] INFO  c.d.c.n.g.ApiRequestLogger:participant=XXX  - Request c.d.l.a.v.CommandService/SubmitAndWaitForTransaction by /XX.XXX.XXX.XXX:XXXXX: failed with DEADLINE_EXCEEDED/REQUEST_TIME_OUT(3,f19c3ae8): Timed out while awaiting for a completion corresponding to a command submission.
[canton-env-execution-context-70] INFO  c.d.p.a.s.tracking.TrackerMap - Shutting down tracker for Key(MS004App,Set(XXX)) after inactivity of PT5M, context: {participant: "XXX"}
2

Which ledger are you running and which SDK version are you using?

3

Thanks for your reply!
canton version: 1.0.0-rc
daml Sdk version :1.18.1
springboot(bindings-rxjava version:1.18.1, daml-lf-1.8-archive-java-proto version:1.18.1, java version: 1.8)

4

I recommend to try to reproduce this with 2.2.

5

Is there any instruction for upgrading the DAML version? It seems we need to change some script code when upgrading the DAML version.

6

The release notes should describe the required changes.

7

Thanks a lot!

8

I have already upgraded the canton to 2.2.1，enabled debug level and triggered the Timeout error again. The log file is uploaded to the google dive: canton (2).log - Google Drive
Also, when we try to use memory only option, this timeout error wont happen. That means, it is something to do with the JDBC

Furthermore, can we modify the timeout time? How to do it? If I want to make the whole business process smoother, can I set the timeout time to a few seconds? Thanks for your help!

9

Where in the log do you see the timeout? I don’t see the message you showed in the first post.

10

I got the time out error in canton log :
[0;39m[34mINFO c.d.c.n.g.ApiRequestLogger:participant=MS002 tid:c2132131cd83bea628ecb45a536ee887 - Request c.d.l.a.v.CommandService/SubmitAndWaitForTransaction by /10.233.103.231:41656: failed with DEADLINE_EXCEEDED/REQUEST_TIME_OUT(3,24f3cccf): Timed out while awaiting for a completion corresponding to a command submission.

This request has a timeout problem： tid:c2132131cd83bea628ecb45a536ee887

11

The log file you uploaded does not seem to contain any DEADLINE_EXCEEDED line. Did it maybe get truncated?

12

The log file seems does not contain 'TIME_OUT" error, it can be found in the console and in the response of the request.

The log file conteain ’ tid:c2132131cd83bea628ecb45a536ee887’:
‘2022-07-14 09:40:50,641 [canton-env-execution-context-106] DEBUG c.d.c.n.g.ApiRequestLogger:participant=MS002 tid:c2132131cd83bea628ecb45a536ee887 - Request c.d.l.a.v.CommandService/SubmitAndWaitForTransaction by /10.233.103.231:41656: received a message
Request tid:c2132131cd83bea628ecb45a536ee887’,

could you locate the problem by it?

13

Hi there,

the log file looks truncated, but I do see that there are 3 transactions being rejected due to a duplicate contract key:

  CommandRejected(
    recordTime = 2022-07-14T09:40:53.249329Z,
    completionInfo = CompletionInfo(
      actAs = ExpoProAdmin::12203ad0d871...,
      commandId = d43a549f-d727-4d5e-afb6-2d1e7fbd5eb3,
      applicationId = MS002App,
      deduplication period = DeduplicationOffset(offset=Offset(Bytes(000000000000000001)),
      submissionId = Some(24f3cccf-6fb4-4b63-9d4c-157372b9b323)
    ),
    reason = FinalReason(
      Status(
        ALREADY_EXISTS,
        DUPLICATE_CONTRACT_KEY(10,5f47c01f): Inconsistent rejected transaction would create a key that already exists (DuplicateKey),
        details = Seq(
          Any(type.googleapis.com/google.rpc.ErrorInfo,<ByteString@60d1bdb4 size=108 contents="\n\026DUPLICATE_CONTRACT_KEY\032\024\n\vparticipant\022\005MS002\032...">,UnknownFieldSet(Map())),
          Any(type.googleapis.com/google.rpc.RequestInfo,<ByteString@5d77d834 size=34 contents="\n 5f47c01f391706cd19547679e33e7e56">,UnknownFieldSet(Map()))
        )
      )
    )
  )
).

Can you check that your application handles contention on contract keys correctly?

14

I just want to triggered the Timeout error, repeat same request in few seconds, I will try to download the full log. thanks for your help!

15

When we launch two requests at the same time, it’s easy to timeout. After I changed this value (storage.max-connections) to 2000 in canton.conf, it does not time out when 5 requests are triggered at the same time. In our other test environment, storage.max-connections is 4, and 50 requests are triggered at the same time without any problem. storage.max-connections does not seem to be set to a large number, what is the root problem? How can I optimize it?

16

We redownload the log file which contain the timeout error, please see this in attachment.canton.zip - Google Drive

we find the time out error in log file as below:

2022-07-19 06:29:08,499 [canton-env-execution-context-161] INFO c.d.p.a.services.ApiCommandService - REQUEST_TIME_OUT(3,a3fcac8d): Timed out while awaiting for a completion corresponding to a command submission., context: {participant: "MS002", readAs: [], parties: ["ExpoProAdmin::12203ad0d871ab3ef01815843d727518a74bb046bd3d6084e29b06c63b1f62de083f"], submissionId: "a3fcac8d-9b81-4f09-b2ab-3842cc679814", actAs: ["ExpoProAdmin::12203ad0d871ab3ef01815843d727518a74bb046bd3d6084e29b06c63b1f62de083f"], commandId: "83660b36-632e-4db9-82fb-f49f933e43dc", err-context: "{location=CompletionResponse.scala:121}"}
2022-07-19 06:29:08,509 [canton-env-execution-context-161] INFO c.d.c.n.g.ApiRequestLogger:participant=MS002 tid:dceb664577d23208d8f52880b68d002c - Request c.d.l.a.v.CommandService/SubmitAndWaitForTransaction by /10.233.79.163:52688: failed with DEADLINE_EXCEEDED/REQUEST_TIME_OUT(3,a3fcac8d): Timed out while awaiting for a completion corresponding to a command submission.
17

We plan to add a retry mechanism, but the current timeout is a bit long, how can we set the timeout to half a minute to speed up the retry pace? We found the timeout related parameters as below in the documentation, but we don’t know how to change them properly, please help guide us how to change the timeout to 30 seconds. Thank you！

@ mydomain.service.get_dynamic_domain_parameters
res1: com.digitalasset.canton.protocol.DynamicDomainParameters = DynamicDomainParameters(
participantResponseTimeout = 1m,
mediatorReactionTimeout = 1m,
transferExclusivityTimeout = 1m,
topologyChangeDelay = 0.25s,
ledgerTimeRecordTimeTolerance = 1m
)