When is RESOURCE_EXHAUSTED returned by the command submission/completion services?

Questions
, ,
2

This is how ledger API services work with respect to RESOURCE_EXHAUSTED errors, to the best of my knowledge:

Command submission service

If the submission service returns RESOURCE_EXHAUSTED, it means the command submission was not accepted by the backing ledger due to back-pressure. There will therefore be no element in the completion stream. The client should back off exponentially and retry.

For reference, the submission service does:

  1. Validate the gRPC request
  2. Deduplicate the command (i.e., reject it if the “same” command was already submitted through this participant)
  3. Execute the DAML command (including choosing a transaction time)
  4. Send the resulting transaction to the backing ledger

The ledger API server does not backpressure itself when there are too many commands piling up in the first three steps (it should). However, the backing ledger can return Overloaded in the last step above. In that case, the submission service returns RESOURCE_EXHAUSTED.

See WriteService. submitTransaction() and SubmissionResult.Overloaded

Command completion service

The completion stream does not contain RESOURCE_EXHAUSTED elements.

See CompletionFromTransaction

Command service

The command service returns RESOURCE_EXHAUSTED when:

  • The command service itself is overloaded (too many concurrent commands)
  • The submission service returned RESOURCE_EXHAUSTED (the command service uses the command submission service and forwards any error returned by the submission service)

In both cases, the client should back off exponentially and retry.

See HandleOfferResult

4 Likes