Exception in Sandbox and looping continuously

Questions Features & Bugs
, ,
1

When ran a bulk process of creation and archival of contracts , Firstly i’ve got below error in sandbox.

http-json-ledger-api-akka.actor.default-dispatcher-715] ERROR com.daml.http.Endpoints - Future failed
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Inconsistent: InconsistentKeys: at least one contract key has changed since the submission
	at io.grpc.Status.asRuntimeException(Status.java:534)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

After that, i further reprocessed some set of messages… I’ve received below exception in sandbox and it started looping continuously for every 10 secs.

ERROR: Error while running indexer, restart scheduled after 10 seconds
org.h2.jdbc.JdbcBatchUpdateException: Unique index or primary key violation: "PUBLIC.INDEX_5 ON PUBLIC.PARTICIPANT_CONTRACTS(CREATE_KEY_HASH) VALUES 1475"; SQL statement:
merge into participant_contracts using dual
 on contract_id = ? when not matched then
 insert (contract_id, template_id, create_argument, create_argument_compression, create_ledger_effective_time, create_key_hash, create_stakeholders)
 values (?, ?, ?, ?, ?, ?, ?) [23505-200]
	at org.h2.jdbc.JdbcPreparedStatement.executeBatch(JdbcPreparedStatement.java:1235)
	at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
	at anorm.BatchSql.$anonfun$execute$2(BatchSql.scala:83)
	at resource.AbstractManagedResource.$anonfun$acquireFor$1(AbstractManagedResource.scala:85)
	at scala.util.control.Exception$Catch.$anonfun$either$1(Exception.scala:252)
	at scala.util.control.Exception$Catch.apply(Exception.scala:228)
	at scala.util.control.Exception$Catch.either(Exception.scala:252)
	at resource.AbstractManagedResource.acquireFor(AbstractManagedResource.scala:85)
	at resource.ManagedResourceOperations.apply(ManagedResourceOperations.scala:31)
	at resource.ManagedResourceOperations.apply$(ManagedResourceOperations.scala:31)
	at resource.AbstractManagedResource.apply(AbstractManagedResource.scala:48)
	at resource.ManagedResourceOperations.acquireAndGet(ManagedResourceOperations.scala:29)
	at resource.ManagedResourceOperations.acquireAndGet$(ManagedResourceOperations.scala:29)
	at resource.AbstractManagedResource.acquireAndGet(AbstractManagedResource.scala:48)
	at anorm.BatchSql.execute(BatchSql.scala:83)
	at anorm.BatchSql.execute$(BatchSql.scala:81)
	at anorm.BatchSql$Copy.execute(BatchSql.scala:222)
	at com.daml.platform.store.dao.events.ContractsTableH2$InsertContractsExecutable.$anonfun$execute$1(ContractsTableH2.scala:87)
	at scala.Option.foreach(Option.scala:407)
	at com.daml.platform.store.dao.events.ContractsTableH2$InsertContractsExecutable.execute(ContractsTableH2.scala:87)
	at com.daml.platform.store.dao.events.TransactionsWriter$PreparedInsert.$anonfun$writeState$5(TransactionsWriter.scala:52)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.daml.metrics.Timed$.$anonfun$value$1(Timed.scala:19)
	at com.codahale.metrics.Timer.time(Timer.java:118)
	at com.daml.metrics.Timed$.value(Timed.scala:19)
	at com.daml.platform.store.dao.events.TransactionsWriter$PreparedInsert.writeState(TransactionsWriter.scala:52)
	at com.daml.platform.store.dao.JdbcLedgerDao.$anonfun$storeTransaction$2(JdbcLedgerDao.scala:502)
	at com.daml.platform.store.dao.HikariJdbcConnectionProvider.$anonfun$runSQL$1(HikariJdbcConnectionProvider.scala:148)
	at com.daml.metrics.Timed$.$anonfun$value$1(Timed.scala:19)
	at com.codahale.metrics.Timer.time(Timer.java:118)
	at com.daml.metrics.Timed$.value(Timed.scala:19)
	at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:148)
	at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$2(DbDispatcher.scala:55)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

It did not let me proceed further with any other activity on ledger… Its merely dead per se.

DAML SDK : 1.11.1
HTTP-JSON : 1.11.1

Please advise.

cc: @yukikaze

2

The second issue is due to a bug in the duplicate contract key detection. This is fixed in SDK 1.11.2 or 1.13.0 and newer.

The same bug can also result in “DuplicateKeys” error being reported as “InconsistentKeys” so the first is either a race on a contract key leading to an actual InconsistentKeys error or a DuplicateKeys error meaning you tried to create a contract with a key that already exists. Given that it’s the same bug, it’s also the same fix so I recommend upgrading to at least 1.11.2.