How do I stop my trigger deployed on DamlHub from continually spamming the ledger with errors?

I have written a trigger that appears to have a bug leading to continuous DUPLICATE_CONTRACT_KEY errors that I see in my bot logs and in my participant logs.

Eventually the ledger becomes unresponsive and will not start when this happens.

How can I prevent this from happening and how can I rate limit what errors are being thrown from my trigger/bot?

bot
16:50:18.022 [TriggerRunner-akka.actor.default-dispatcher-5] WARN  com.daml.lf.engine.trigger.Runner - Command failed: DUPLICATE_CONTRACT_KEY(10,0ca44d4c): Inconsistent rejected transaction would create a key that already exists (DuplicateKey) , code: 6 , context: {triggerDefinition: "61c3eb23c38d91078633b3df143ac8eefc90297b3f54ad340022aabb17de9e75:AutoApproval:autoApprovalTrigger"
1 Like

Hi @sormeter,

In addition to making your model more resilient to these types of errors (for example instead of just using a create, use a function that will try to lookupByKey and if it is not found only then do the create), there are some trigger features that can be used to catch errors and protect the trigger from infinitely looping, an example of which can be found here: GitHub - digital-asset/trigger-failure: Example code for dealing with Daml Trigger failures

When a trigger using the wrapper set up in this repo reaches its maximum amount of failures, the trigger stops processing events and a TriggerFailureNotification contract is created that includes the failure messages that accumulated during the triggers last run.

The party running the trigger then has the opportunity to exercise the StartTrigger choice which will reset the amount of failures and let the trigger start processing commands again. This will allow the trigger operator time to either undeploy the trigger or manually get the state of the ledger into a state where the trigger can run correctly without the trigger overloading the ledger with errors.

In the example in the above repo, the trigger uses the updateState function that receives messages as the commands are completed or failed:

updateStateFailures : Message -> T.TriggerUpdateA FailureState ()
updateStateFailures m = case m of
  (MCompletion (Completion _ (Failed _ msg))) -> modify $ addFailure msg
  _                                           -> pure ()

If the trigger receives a failed message, it stores an attempt in its state which is checked later in the trigger rule and “pauses” the trigger if the maximum error threshold is hit, but the trigger could also use this feature to do specific operations in response to failures.

4 Likes