Participant node running for multiple days with dataload - DEADLINE_EXCEEDED

Questions
1

Hi team,

I encountered a DEADLINE_EXCEEDED error while my participant node was running for multiple days and received multiple dataload.

  GrpcClientGaveUp: DEADLINE_EXCEEDED/deadline exceeded after 106.829442472s. [closed=[UNAVAILABLE], committed=[buffered_nanos=35940812200, waiting_for_connection]]
  Request: send-async/tick-b339ace8-a001-4cc4-8a2e-7c95968ab87c
2022-04-19 09:52:17,949 [canton-env-execution-context-55135] WARN  c.d.c.s.c.t.GrpcSequencerClientTransport:participant=pUS/domain=domain tid:92473e7ac63e4da0734871c8f6c9cab1 - Failed to send acknowledgement for 2022-04-19T08:33:43.796203Z: Request failed for sequencer.
  GrpcClientGaveUp: DEADLINE_EXCEEDED/deadline exceeded after 119.999929159s. [closed=[], open=[[buffered_nanos=28762, remote_addr=/172.17.0.3:5018]]]
  Request: acknowledge/2022-04-19T08:33:43.796203Z
2022-04-19 09:48:48,307 [canton-env-execution-context-55135] DEBUG c.d.c.s.c.t.GrpcSequencerClientTransport:participant=pUS/domain=domain tid:ab47939cf9cefe2afbc9d09d602e9fa6 - Retry has not been configured for GrpcClientGaveUp, giving up.
2022-04-19 09:51:48,078 [canton-env-execution-context-55476] INFO  c.d.c.s.c.t.GrpcSequencerClientTransport:participant=pUS/domain=domain tid:ab47939cf9cefe2afbc9d09d602e9fa6 - Send [tick-b339ace8-a001-4cc4-8a2e-7c95968ab87c] returned an error however may still be possibility sequenced so we are ignoring the error: Request failed for sequencer.
  GrpcClientGaveUp: DEADLINE_EXCEEDED/deadline exceeded after 106.829442472s. [closed=[UNAVAILABLE], committed=[buffered_nanos=35940812200, waiting_for_connection]]
  Request: send-async/tick-b339ace8-a001-4cc4-8a2e-7c95968ab87c
2022-04-19 09:55:55,688 [canton-env-execution-context-55135] DEBUG c.d.c.t.TimeProofRequestSubmitterImpl:participant=pUS/domain-alias=domain tid:ab47939cf9cefe2afbc9d09d602e9fa6 - Sending time request
2022-04-19 10:00:05,330 [canton-env-execution-context-55135] DEBUG c.d.c.s.c.t.GrpcSequencerClientTransport:participant=pUS/domain=domain tid:ab47939cf9cefe2afbc9d09d602e9fa6 - Sending request send-async/tick-320fbf74-43b4-4d94-aac6-f47e6e11f6ac to sequencer.
2022-04-19 10:02:38,663 [canton-env-execution-context-55135] WARN  c.d.c.s.c.t.GrpcSequencerClientTransport:participant=pUS/domain=domain tid:ab47939cf9cefe2afbc9d09d602e9fa6 - Request failed for sequencer.

Now my participant node won’t respond, on my remote node when I try to do a health.status I have

Screenshot 2022-04-19 at 9.31.26 PM
Screenshot 2022-04-19 at 9.31.26 PM1424Ă—224 38.7 KB

How do I retrieve my participant node ?

Cheers,

Jean-Paul

1 Like
2

Hi @jaypeeda ,
Are you sure that the remote participant is still running or could it be that it crashed?

Best,

Rafael

3

How can I make sure whether it is running or it has crashed ?
The response from my screen is showing that it is waiting for connection no ?

1 Like
4

I think it was indeed crashed, as I couldn’t find the process on my machine. So I restart my canton environment.

A question on my side is, if you have a participant node and you didn’t extract the private keys (identity, signing, encryption) before it crashed. Is it possible to recover or it’s gone forever?

Thanks,
Jean-Paul

1 Like
5

Hi @jaypeeda ,
Sorry that I was a bit slow to come back to you. I was thinking that your node crashed (I interpreted the “waiting for connection” a bit differently than yourself).

The keys are written to the primary storage so if you don’t do everything in-memory, you should be able to restart your node and it will find the keys in the DB.

I hope this helps,

Rafael

1 Like