We are facing an issue where we have 1 contract in our ledger that we no longer can transact with.
The contract itself can easily be retrieved through a Postman Command , but any action (Choice) execution is not allowed
The response message we get back from any choice execution is {"errors":["CommandService Error, io.grpc.StatusRuntimeException: ABORTED: Inconsistent: InconsistentKeys: at least one contract key has changed since the submission"],"status":500}
This seems to be similar previous discussion post → https://discuss.daml.com/t/understanding-inconsistent-keys-error-from-json-api/2478
Here are my basic questions :
What would cause a contract to be in the ledger but still identified with an “Inconsistent Key” ?
2, How can we scan our ledger and try to identify those contracts with “Inconsistent Key?”
How can we fix any potential other contracts with this issue ?
We have seen this issue pop-up a few times now, but we don’t have any good answer on how to prevent this or how to fix this ?
InconsistentKeys is usually a race condition/contention.
Daml transaction processing works by first running your submitted command against the latest committed ledger state known to the participant. The participant then submits it to the ledger. The ledger sequences it, validates that the transaction is model conformant and checks that all contracts used in the transaction are still active and that the contract keys map to the same contracts. If the contract keys map to different contracts (including none if it originally matched to a contract & the other way around you get an InconsistentKeys error).
So this is usually triggered by two concurrent transactions modifying the same contract key:
For simplicity, let’s assume we have a single participant & two transactions A & B and a contract key K.
Both transactions are submitted concurrently to the participant. Before those transactions K maps to a contract C0. The participant interprets the transactions concurrently so in both cases K maps to C0 originally. Let’s say A modifies the mapping to K → C1 and B to K → C2.
Now those transactions are submitted to the ledger. The ledger sequences them. If A comes first, A will be accepted. However B will be rejected since the mapping in the transaction K → C0 no longer matches the mapping K → C1 after applying A. Similarly, if B comes first A will be rejected.
At the moment, you don’t get information on which key is causing the race unfortunately.
Note that even in Sandbox you have parallel transaciton processing and can run into this issue.
As for fixing it, it somewhat depends on how frequent it is. In general, you can just retry if your command fails with this error and if this is a rare occasion I would not do anything else.
If this happens very frequently, i.e., you have a lot of contention on one contract key things are a bit different. While just retrying could work you have a chance of just running into contention again and you have to retry several times eventually significantly reducing your transaction throughput. In those cases you have to find some way to reduce contention on this contract key. This can range from sequencing the commands that operate on the same contract key to completely eliminate contention on that key to some form of sharding to make sure different commands operate on different keys. The specifics for this heavily depend on your workflow so there’s no single recommendation here.
Just a few clarifying comments/questions based on your comments
So this is usually triggered by two concurrent transactions modifying the same contract key:
For simplicity, let’s assume we have a single participant & two transactions A & B and a contract key K.
Both transactions are submitted concurrently to the participant. Before those transactions K maps to a contract C0. The participant interprets the transactions concurrently so in both cases K maps to C0 originally. Let’s say A modifies the mapping to K → C1 and B to K → C2.
We are a fully web-enabled application… All daml actions are basically triggered by buttons on a screens… Our user base is very small at this time, so it is unlikely that 2 people are trying to modify a contract simultaneously. Technically it could be possible that one user is modifying the contract in one web-browser and either the same or different user is doing the same on the same contract in a other browser … but I would find this highly unlikely>
Or do you think that if a user clicks a button twice (is impatient) this race condition may occur ?
Is my understanding correct that
while the situation of a race condition occurred the leger notices this situation and puts the contract in an “inconsistent key” state “forever” ?
From that point forward, no other JSON-API enabled action (choice) is possible.
Our observation is that in our front-end UX every future action is failing and we get the same JSON API return message on every attempt. Even when we do a manual Choice to Archive in Postman it is failing.
Now those transactions are submitted to the ledger. The ledger sequences them. If A comes first, A will be accepted. However B will be rejected since the mapping in the transaction K → C0 no longer matches the mapping K → C1 after applying A. Similarly, if B comes first A will be rejected.
So lets say K-> C1 state, why would any other future action fail ?
We simply are executing and transaction on the latest contractID we have in our system and trying to apply a new state to that Contractid. The contract should be in its final and correct state. There is no longer a race condition happening at this time. It is a simple single transaction that is based on the contract we can retrieve with all its details through the JSON-API
How can we look at these contract Key Mappings ? Is this viewable in Postgres directly ? We don’t pass any contract keys in the JSON-API message we are sending
How does the system know that they are wrong ?
Is there a tool available for us to look at all contracts and determine which contract keys are incorrect ? Maybe through grpc or ledger api ?
We do find inconsistent Key issues from time to time. It is not frequent, but the impact is now becoming bigger and bigger as our only option to solve it at this time is t replay the entire customer scenario manually. (since the existing contract is no longer transactionable)
Also the worst thing is that every time it is our customer who notices the problem (before we can) . The race condition may have happened long time ago and we are just noticing it on the next action performed by the user days later.
Happy to send more information or even demonstrate so we can prevent and fix this issue.
I really want to get to the bottom of this, so we can prevent some unhappy daml customers
A single user clicking the button twice could produce that.
No, one transaction will be rejected with the inconsistent key error but just resubmitting it should fix it. There is no permanent effect. If you get that error on every later request it sounds like maybe you have some rogue automation that causes contention all the time.
Not any other future action will fail. But the contract key assignment is fixed in the participant. So in the example, A and B both fix the assignment K → C0. For the one that gets sequenced first, that is still accurate. But given that this transaction changes the assignment, the other one will get rejected since the mapping fixed during the participant K->C0 is no longer accurate.
When I say “contract key mapping” I just mean the implicit mapping created by the active contracts with a key. If contract C is active and has key k that means key k currently maps to contract C. You can observe that via fetchByKey on the JSON API but not the intermediate states during transaction processing.
So this is where we have found consistent behavior for this one contract. We have a permanent effect of “an inconsistent key” error message. We will work with our partners BTP to identify this “rogue” behavior