Recovery of Participant Node from Domain

Questions
1

Hi team,

I have successfully recovered a participant node (with no persistent database or simulating that even the database is crashed) from domain connection, through the keys downloaded from that participant node. All contracts are recovered in the participant node after the whole process.

My observation is that when the participant node is recovered, there are some warning messages on both the participant node and the domain.

On the recovered participant node

WARN  c.d.c.p.p.TransactionProcessor:participant=participant1/domain=mydomain tid:08ac7be8ebe4d2064fb2d1a89dbfe8e4 - Response message for request [2] timed out at 2022-04-09T06:24:38.468208Z
WARN  c.d.c.p.p.TransactionProcessor:participant=participant1/domain=mydomain tid:e32b66b45bffcebc86523dafcc30a29c - Response message for request [0] timed out at 2022-04-09T06:24:38.468208Z
WARN  c.d.c.p.p.TransactionProcessor:participant=participant1/domain=mydomain tid:f810c9c3b99e0e9422c8d537a49f7b89 - Response message for request [3] timed out at 2022-04-09T06:24:38.468208Z
WARN  c.d.c.p.p.TransactionProcessor:participant=participant1/domain=mydomain tid:faf5533e697ea66cbc59e22cbe4c9ea1 - Response message for request [1] timed out at 2022-04-09T06:24:38.468208Z

On domain side

WARN  c.d.c.d.s.s.WritePayloadsFlow$:domain=mydomain tid:e32b66b45bffcebc86523dafcc30a29c - The sequencer time [2022-04-09T06:24:38.248190Z] has exceeded the max-sequencing-time of the send request [2022-04-09T06:04:52.005521Z]: deliver[message-id:b4fde246-d54c-4234-86ec-5d00334541bf]
WARN  c.d.c.d.s.s.WritePayloadsFlow$:domain=mydomain tid:faf5533e697ea66cbc59e22cbe4c9ea1 - The sequencer time [2022-04-09T06:24:38.295222Z] has exceeded the max-sequencing-time of the send request [2022-04-09T06:04:52.868373Z]: deliver[message-id:43039a1c-1cd2-4082-b898-df5decbb9e1a]
WARN  c.d.c.d.s.s.WritePayloadsFlow$:domain=mydomain tid:08ac7be8ebe4d2064fb2d1a89dbfe8e4 - The sequencer time [2022-04-09T06:24:38.319855Z] has exceeded the max-sequencing-time of the send request [2022-04-09T06:04:54.651539Z]: deliver[message-id:24a5b90f-a88a-484e-8542-f2d7dc3420ae]
WARN  c.d.c.d.s.s.WritePayloadsFlow$:domain=mydomain tid:f810c9c3b99e0e9422c8d537a49f7b89 - The sequencer time [2022-04-09T06:24:38.327356Z] has exceeded the max-sequencing-time of the send request [2022-04-09T06:22:10.815760Z]: deliver[message-id:6ef550f0-3e93-4dc0-b5f2-234608cbf971]

I suspect it is due to the transactions are recovered from the domain sequenced data. (There were four transactions made before). Please let me know if this is the mechanism.

Besides, are we expecting similar behaviour if the domain is using DLT (Fabric and Besu)?

Thanks in advance.

kc

1 Like
2

Hi KC,

these messages indicate that the participant is sending responses to a request after the request’s participant response deadline. Consequently, the request times out and the sequencer will discard the responses.

Not sure what exactly you did, but could it be that you have in-flight requests when the crash occurs?

Best,
Matthias

3

Thanks Matthias. What I have done is to set up a simple canton environment (01-simple-topology-like) and have the participant1 completely recovered from the keys (namespace, encryption and signing). The recovery is successful (all active contracts, parties are back after I upload the DAR and connect it to the domain). These warning messages appear in both the participant1 and the domain side.

As I notice the quantity of transactions (4), I suspect it’s due to the recovery process.

Thanks again.

kc