We’re submitting circa ~400 independent transactions in one submission (so have ruled out contention) via the java bindings and getting the below error:
DEADLINE_EXCEEDED: REQUEST_TIME_OUT(3,e1d63187): Timed out while awaiting for a completion corresponding to a command submission.
What parameters can we change to increase this time out? Is this a client side issue or do the participant/sequencer nodes have a config to increase this.
To make it clear, you cannot submit independent transactions. You can submit a number of commands which are going to result in a single transaction.
Looking at the documentation for REQUEST_TIME_OUT I would say it’s reported by the participant node and can be solved by either bumping the configuration for that timeout (I don’t know the specific configuration key, I’ll let someone else reply). I would say it might be a possibility (but I’m not 100% sure) that despite being reported by the participant node the timeout is triggered by something still being in flight while the CommandService is doing the command completion tracking for you. A possibility is for you to use the CommandSubmissionService and CommandCompletionService instead of delegating completion tracking to the CommandService, which is more complex but allows you to manage this kind of issues on the application side (instead of setting the limit for the whole participant node).
Note that while you can bump some timeouts, you cannot make transactions arbitrarily large. At some point you’ll hit time model issues so you also have to bump time model skews and generally this is not how a Daml ledger is suposed to be used.
I’d stick to batches of around ~100 but ofc it depends on how big an individual element in your batch is.