DAML on sql connection not available error

Questions
,
1

Started seeing the below error on our daml-on-sql process today,

ERROR: Unhandled internal error (context: {participantId=sandbox-participant, startExclusive=0000000000000115, endInclusive=, parties=[Operator]})
java.sql.SQLTransientConnectionException: daml.index.db.connection.sandbox - Connection is not available, request timed out after 909ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
	at com.daml.platform.store.dao.HikariJdbcConnectionProvider.runSQL(HikariJdbcConnectionProvider.scala:77)
	at com.daml.platform.store.dao.DbDispatcher.$anonfun$executeSql$1(DbDispatcher.scala:54)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: SSL error: Connection reset
	at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:42)
	at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:441)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:135)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:211)
	at org.postgresql.Driver.makeConnection(Driver.java:458)
	at org.postgresql.Driver.connect(Driver.java:260)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:119)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467)
	at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:706)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:692)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 common frames omitted
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:210)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
	at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1761)
	at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:124)
	at sun.security.ssl.Handshaker.kickstart(Handshaker.java:1079)
	at sun.security.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1479)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1346)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
	at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:40)
	... 18 common frames omitted

Also on the following errors on the Azure PostgreSql logs,

could not receive data from client: An existing connection was forcibly closed by the remote host.

Is there any issue with the connection pool recycling? or is it do any resource pressure?
Also, is there a way to specify connection limits?

1 Like
2

What version of the SDK are you using?

1 Like
3

1.2.0 currently, but we are also in the process of moving to 1.4.0

1 Like
4

I see, thanks. Can you clarify what you mean with DAML on SQL? We recently released a standalone JAR, but it was not there in 1.2. Are you running daml sandbox or daml sandbox-classic? If it’s the former, it’s been addressed in this post but it won’t be a problem in DAML on SQL.

1 Like
5

We are using the standalone jar and this is the image provided by @georg http://digitalasset-docker-daml-on-sql.jfrog.io/digitalasset/daml-on-sql:1.2.0.

Are you saying that it won’t be problem in 1.4.0 jar?

1 Like
6

I spoke with @georg, the standalone JAR was based on sandbox-classic, just as DAML on SQL is since 1.4, so the post I shared does not apply.

It’s quite difficult to understand what could have caused this, but the root cause seems to be a closed socket (Caused by: java.net.SocketException: Connection reset). Could this have been a transient network error? If you are experiencing this on an ongoing basis, can you give a little bit more color over the issue? Under which conditions does it emerge?

1 Like
7

Yesterday was the first time we experienced this. It doesn’t look like a network error. We restarted the container and everything was back to normal. We are checking if there was any resource pressure.

No issues with the managed postgres service, the resource was healthy during the time.

Please let me know what else to look for.

1 Like
8

It’s very difficult to say, all I see is a connection reset. If you see this happening again it would be useful to understand the conditions that made this error emerge.

1 Like
9

2 posts were split to a new topic: DAML on SQL: Connection not available on startup