Dear DAML community,
I am trying to upload our smart contracts DAR file using daml ledger upload-dar to a daml sandbox connected to a Postgres database. After the unsuccessful upload I get the following error message on the terminal (daml-helper: GRPCIOBadStatusCode StatusAborted (StatusDetails {unStatusDetails = "Request timed out"})) and continuously looping errors on the Server:
INFO: daml.index.db.connection.indexer - Starting...
DEBUG: daml.index.db.connection.indexer - Added connection org.postgresql.jdbc.PgConnection@35fd4c26
INFO: daml.index.db.connection.indexer - Start completed.
INFO: Found existing ledger with ID: b324b85f-3ff0-4181-90ce-ff5075b68702
INFO: Found existing participant id 'sandbox-participant'
INFO: Restarted Indexer Server
WARN: daml.index.db.connection.indexer - Connection org.postgresql.jdbc.PgConnection@35fd4c26 marked as broken because of SQLSTATE(08006), ErrorCode(0)
java.sql.BatchUpdateException: Batch entry 0 insert into packages(package_id, upload_id, source_description, size, known_since, ledger_offset, package)
select '6442383f2e52716c6c01d63e2bec689d92b0df6b1a046fa0600df87b3faba168', '94fad625-5b43-4543-8b9c-648e57e2957f', NULL, 1973317, '2020-09-24 14:01:44.418529+00'::timestamp with time zone, ledger_end, ?
from parameters
on conflict (package_id) do nothing was aborted: An I/O error occurred while sending to the backend. Call getNextException to see other errors in the batch.
at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:153)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:513)
at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:850)
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:873)
at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1562)
at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
at anorm.BatchSql.execute(BatchSql.scala:70)
at anorm.BatchSql.execute$(BatchSql.scala:69)
at anorm.BatchSql$Copy.execute(BatchSql.scala:203)
at com.daml.platform.store.dao.JdbcLedgerDao.executeBatchSql(JdbcLedgerDao.scala:858)
at com.daml.platform.store.dao.JdbcLedgerDao.uploadLfPackages(JdbcLedgerDao.scala:700)
at com.daml.platform.store.dao.JdbcLedgerDao.$anonfun$storePackageEntry$1(JdbcLedgerDao.scala:661)
at com.daml.platform.store.dao.HikariJdbcConnectionProvider.$anonfun$runSQL$1(HikariJdbcConnectionProvider.scala:131)
at com.daml.metrics.Timed$.$anonfun$value$1(Timed.scala:18)
at com.codahale.metrics.Timer.time(Timer.java:118)
at com.daml.metrics.Timed$.value(Timed.scala:18)
at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:131)
at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$2(DbDispatcher.scala:53)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:514)
... 27 common frames omitted
Caused by: javax.net.ssl.SSLProtocolException: Connection reset by peer (Write failed)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:126)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:259)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:988)
at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
at org.postgresql.core.PGStream.send(PGStream.java:307)
at org.postgresql.core.v3.SimpleParameterList.streamBytea(SimpleParameterList.java:287)
at org.postgresql.core.v3.SimpleParameterList.writeV3Value(SimpleParameterList.java:380)
at org.postgresql.core.v3.QueryExecutorImpl.sendBind(QueryExecutorImpl.java:1621)
at org.postgresql.core.v3.QueryExecutorImpl.sendOneQuery(QueryExecutorImpl.java:1861)
at org.postgresql.core.v3.QueryExecutorImpl.sendQuery(QueryExecutorImpl.java:1386)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:494)
... 27 common frames omitted
Caused by: java.net.SocketException: Connection reset by peer (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(SSLSocketOutputRecord.java:320)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:983)
... 35 common frames omitted
ERROR: Exception while executing SQL query. Rolled back. (context: {updateRecordTime=2020-09-24T14:01:44.418529Z, updateSourceDescription=, metric=store_package_entry, updateOffset=00000000000000020000000000000000, updateSubmissionId=94fad625-5b43-4543-8b9c-648e57e2957f})
java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy2.rollback(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.rollback(ProxyConnection.java:370)
at com.zaxxer.hikari.pool.HikariProxyConnection.rollback(HikariProxyConnection.java)
at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:144)
at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$2(DbDispatcher.scala:53)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
DEBUG: daml.index.db.connection.indexer - Closing connection org.postgresql.jdbc.PgConnection@35fd4c26: (connection is broken)
ERROR: Error while running indexer, restart scheduled after 10 seconds
java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy2.rollback(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.rollback(ProxyConnection.java:370)
at com.zaxxer.hikari.pool.HikariProxyConnection.rollback(HikariProxyConnection.java)
at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:144)
at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$2(DbDispatcher.scala:53)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
I tried different things including increasing the timeout using daml ledger upload-dar --timeout 300 and also cleaning the postgres database tables and restarting the Daml sandbox.
Best regards,
Pierre Besson
