Trying to run the `ex-secure-daml-infra` refapp, getting `password authentication failed` error for PostgreSql

Questions
1

I’m trying to run the ex-secure-daml-infra refapp as per the the documentation.

I had to do some minor tweaks which seem to be obvious to run it without errors up to a point, where I got stuck.

I’m at this phase: ex-secure-daml-infra/StartingServices.md at 487ed79b1a62f0bd543ee7213e63337245dcbcb7 · digital-asset/ex-secure-daml-infra · GitHub

The script ./run-docker.sh runs ok.

When I try to run the next script, ./run-sandbox.sh, I get the following error:

18:32:36.503 [main] INFO  ROOT - Sandbox verbosity changed to DEBUG 
18:32:39.152 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - Driver class org.postgresql.Driver found in Thread context class loader jdk.internal.loader.ClassLoaders$AppClassLoader@277050dc 
18:32:39.164 [daml-on-sql-akka.actor.default-dispatcher-6] INFO  c.d.p.store.dao.HikariConnection - Creating Hikari connections with asynchronous commit disabled (context: {participantId=daml-on-sql}) 
18:32:39.177 [daml-on-sql-akka.actor.default-dispatcher-6] INFO  c.d.p.store.dao.HikariConnection - Attempting to connect to the database (attempt 1/600) (context: {participantId=daml-on-sql}) 
18:32:39.180 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - daml.index.db.connection.migrations - configuration: 
18:32:39.185 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - allowPoolSuspension.............false 
18:32:39.185 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - autoCommit......................false 
18:32:39.185 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - catalog.........................none 
18:32:39.185 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - connectionInitSql..............."SET synchronous_commit=ON" 
18:32:39.186 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - connectionTestQuery.............none 
18:32:39.186 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - connectionTimeout...............5000 
18:32:39.186 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - dataSource......................none 
18:32:39.186 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceClassName.............none 
18:32:39.186 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................none 
18:32:39.187 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceProperties............{password=<masked>, prepStmtCacheSqlLimit=2048, cachePrepStmts=true, prepStmtCacheSize=128} 
18:32:39.188 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - driverClassName................."org.postgresql.Driver" 
18:32:39.188 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckProperties...........{} 
18:32:39.188 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckRegistry.............none 
18:32:39.188 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000 
18:32:39.188 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - initializationFailTimeout.......1 
18:32:39.189 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false 
18:32:39.189 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - jdbcUrl.........................jdbc:postgresql://db.acme.com/ledger?user=ledger&password=<masked>&ssl=true&sslmode=verify-full&sslrootcert=/Users/gyorgybalazsi/ex-secure-daml-infra/certs/intermediate/certs/ca-chain.cert.pem&sslcert=/Users/gyorgybalazsi/ex-secure-daml-infra/certs/client/client1.acme.com.cert.der&sslkey=/Users/gyorgybalazsi/ex-secure-daml-infra/certs/client/client1.acme.com.key.der 
18:32:39.189 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........0 
18:32:39.190 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - maxLifetime.....................1800000 
18:32:39.190 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - maximumPoolSize.................2 
18:32:39.190 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - metricRegistry..................none 
18:32:39.190 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - metricsTrackerFactory...........none 
18:32:39.191 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - minimumIdle.....................2 
18:32:39.191 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - password........................<masked> 
18:32:39.191 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - poolName........................"daml.index.db.connection.migrations" 
18:32:39.192 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - readOnly........................false 
18:32:39.192 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - registerMbeans..................false 
18:32:39.192 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - scheduledExecutor...............none 
18:32:39.192 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - schema..........................none 
18:32:39.192 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - threadFactory...................internal 
18:32:39.193 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - transactionIsolation............default 
18:32:39.193 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - username........................none 
18:32:39.193 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.HikariConfig - validationTimeout...............5000 
18:32:39.199 [daml-on-sql-akka.actor.default-dispatcher-6] INFO  com.zaxxer.hikari.HikariDataSource - daml.index.db.connection.migrations - Starting... 
18:32:40.582 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.pool.PoolBase - daml.index.db.connection.migrations - Failed to create/setup connection: FATAL: password authentication failed for user "ledger" 
18:32:40.585 [daml-on-sql-akka.actor.default-dispatcher-6] DEBUG com.zaxxer.hikari.pool.HikariPool - daml.index.db.connection.migrations - Cannot acquire connection from data source 
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "ledger"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:613)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:161)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
	at org.postgresql.Driver.makeConnection(Driver.java:465)
	at org.postgresql.Driver.connect(Driver.java:264)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:119)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:541)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81)
	at com.daml.platform.store.dao.HikariConnection.$anonfun$acquire$3(HikariJdbcConnectionProvider.scala:70)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
18:32:41.589 [daml-on-sql-akka.actor.default-dispatcher-6] ERROR com.zaxxer.hikari.pool.HikariPool - daml.index.db.connection.migrations - Exception during pool initialization. 
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "ledger"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:613)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:161)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
	at org.postgresql.Driver.makeConnection(Driver.java:465)
	at org.postgresql.Driver.connect(Driver.java:264)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:119)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:541)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81)
	at com.daml.platform.store.dao.HikariConnection.$anonfun$acquire$3(HikariJdbcConnectionProvider.scala:70)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

And this keeps repeating with the subsequent attempts.

What am I doing wrong?

1 Like
2

Not familiar with this app but looking through the scripts you mentioned, the password configured in run-docker.sh is

ChangeDefaultPassword!

whereas the password used by run-sandbox.sh is

LedgerPassword!

I guess they need to be the same in order for the sandbox to connect.

3

Please disregard my previous message.

Looking at it more closely, it appears that the ledger user the run-sandbox.sh script expects should be created by pg-initdb/init-userdb.sh, which should be run by the PostgreSQL container at startup.

Can you make sure that script has run?

2 Likes
4

@Gary_Verhaegen is correct. As part of a TechNote on hardening PostgresQL, the system now created a new database and user (ledger) and should connect to that. If the URL is still showing ChangeDefaultPassword! then this is outdated. I’ll check the scripts again to confirm it works as per original article.

1 Like
5

@nycnewman there are two passwords in the code, ChangeDefaultPassword! and LedgerPassword!, still, it doesn’t seem to work properly to me.

6

For this specific example - postgres user was set to ChangeDefaultPassword!. This is set via the POSTGRES_PASSWORD parameter and is the default password for the admin account. I then do a scirpt (in pg_init) that creates a secondary account “ledger” as a non-admin account associated with a non-default tablespace “ledger”. This is then passed in via the JDBC URL parameter. One way to test this is working is to use psql cli command to test connectivity and account access (this can be run as a locally installed package or via Docker container). Once you have proven the database is up and account working then try starting ledger server.

1 Like
7

Another option is to revert run-docker.sh to the original simple setup by using the following docker command:

docker run --name daml-postgres -d -p 5432:5432 -e POSTGRES_PASSWORD=“ChangeDefaultPassword!” -e POSTGRES_HOST_AUTH_METHOD=trust -v “$(pwd)/certs/server/certs/db.$DOMAIN.cert.pem:/var/lib/postgre sql/db.DOMAIN.cert.pem:ro" -v "(pwd)/certs/server/private/db.$DOMAIN.key.pem:/var/lib/postgresql/db.DOMAIN.key.pem:ro" -v "(pwd)/certs/intermediate/certs/ca-chain.cert.pem:/var/lib/postgresql/ ca-chain.crt:ro” postgres:12 -c ssl=on -c ssl_cert_file=/var/lib/postgresql/db.$DOMAIN.cert.pem -c ssl_key_file=/var/lib/postgresql/db.$DOMAIN.key.pem -c ssl_ca_file=/var/lib/postgresql/ca-chain.c rt -c ssl_min_protocol_version=“TLSv1.2” -c ssl_ciphers=“HIGH:!MEDIUM:+3DES:!aNULL”

run-sandbox.sh would then revert to:

java -jar daml-on-sql-1.10.0.jar
./dist/ex-secure-daml-infra-0.0.1.dar
–client-auth $CLIENT_CERT_AUTH_PARAM
–sql-backend-jdbcurl “jdbc:postgresql://localhost/postgres?user=postgres&password=ChangeDefaultPassword!&ssl=on”
$SIGNER_URL
–log-level DEBUG
–ledgerid LEDGER_ID \ --cacrt "(pwd)/certs/intermediate/certs/ca-chain.cert.pem"
–pem “$(pwd)/certs/server/private/ledger.DOMAIN.key.pem" \ --crt "(pwd)/certs/server/certs/ledger-chain.$DOMAIN.cert.pem”

1 Like
8

Thank you!