DAML on SQL: Connection not available on startup

We came across the following issue today, started working fine after restarting the container.

ERROR: get_event_sequential_id_range: Got an exception while executing a SQL query. Rolled back the transaction. (context: {participantId=daml-on-sql})

java.sql.SQLTransientConnectionException: daml.index.db.connection.sandbox - Connection is not available, request timed out after 250ms.

at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)

at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)

at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:128)

at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$1(DbDispatcher.scala:54)

at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)a

at scala.util.Success.$anonfun$map$1(Try.scala:255)

at scala.util.Success.map(Try.scala:213)

at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)

at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)

at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

ERROR: Unhandled internal error (context: {participantId=daml-on-sql, startExclusive=00000000000002ff, endInclusive=, parties=[test2]})

java.sql.SQLTransientConnectionException: daml.index.db.connection.sandbox - Connection is not available, request timed out after 250ms.

at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)

at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)

at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:128)

at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$1(DbDispatcher.scala:54)

at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)

at scala.util.Success.$anonfun$map$1(Try.scala:255)

at scala.util.Success.map(Try.scala:213)

at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)

at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)

at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

ERROR: get_event_sequential_id_range: Got an exception while executing a SQL query. Rolled back the transaction. (context: {participantId=daml-on-sql})

java.sql.SQLTransientConnectionException: daml.index.db.connection.sandbox - Connection is not available, request timed out after 250ms.

at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)

at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)

at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)

at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:128)

at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$1(DbDispatcher.scala:54)

at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)

at scala.util.Success.$anonfun$map$1(Try.scala:255)

at scala.util.Success.map(Try.scala:213)

at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)

at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)

at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)
2 Likes

This error means that a database connection couldn’t be acquired from the connection pool within 250ms for a flat transactions or a transaction tree request. Do you have many transaction streams running at the same time? It’s difficult to say without knowing more details about how your applications interact with the ledger api.