"No access to: class..." error when attempting to start a domain in Canton

here the canton.log:

sh-4.4$ cat $PWD/tmp/canton.log 
2021-09-10 17:47:22,411 [main] INFO  c.d.canton.CantonEnterpriseApp$ - Starting Canton version 0.27.0
2021-09-10 17:47:23,141 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:47:23,143 [main] INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
2021-09-10 17:47:23,343 [canton-env-execution-context-19] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-09-10 17:47:23,392 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:47:23,485 [main] INFO  c.d.c.e.EnterpriseEnvironment tid:cdc2b1bbc3fa55e17a0aec41db02fa97 - Manual start requested.
2021-09-10 17:54:50,887 [main] INFO  c.d.canton.CantonEnterpriseApp$ - Starting Canton version 0.27.0
2021-09-10 17:54:51,895 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:54:51,898 [main] INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
2021-09-10 17:54:52,181 [canton-env-execution-context-19] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-09-10 17:54:52,249 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 17:54:52,325 [main] DEBUG io.grpc.NameResolverRegistry - Service loader found io.grpc.internal.DnsNameResolverProvider@4d192aef
2021-09-10 17:54:52,355 [main] INFO  c.d.c.e.EnterpriseEnvironment tid:2e24416c2789880eb605da153f34cc21 - Manual start requested.
2021-09-10 18:01:00,156 [main] INFO  c.d.canton.CantonEnterpriseApp$ - Starting Canton version 0.27.0
2021-09-10 18:01:00,813 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 18:01:00,816 [main] INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
2021-09-10 18:01:00,999 [canton-env-execution-context-18] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-09-10 18:01:01,047 [main] INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
2021-09-10 18:01:01,143 [main] DEBUG io.grpc.NameResolverRegistry - Service loader found io.grpc.internal.DnsNameResolverProvider@21f459fc
2021-09-10 18:01:01,174 [main] INFO  c.d.c.e.EnterpriseEnvironment tid:e12838ac803568950bf919dfd819276c - Manual start requested.

No difference, unfortunately.

What’s a bit odd is were not getting as far as we were earlier in this thread.

Let’s try:

./bin/canton daemon -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug

Yeah, I also think it’s strange… before with a “wrong” java image I could see the interactive console but couldn’t start the domain, when I switched to openJDK 11 this started being the behavior.

with this

./bin/canton daemon -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug

command now:

sh-4.4$ ./bin/canton daemon -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug
INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:c7047fb7ee307f9886a513fae9ad95e4 - Manual start requested.
ERROR c.d.c.ServerRunner - An internal error occurred while executing script.
java.lang.ExceptionInInitializerError: null
        at ammonite.Main$.apply$default$5(Main.scala:68)
        at com.digitalasset.canton.console.HeadlessConsole$.createDefaultAmmoniteOptions(HeadlessConsole.scala:59)
        at com.digitalasset.canton.console.HeadlessConsole$.$anonfun$apply$1(HeadlessConsole.scala:46)
        at better.files.Dispose.apply(Dispose.scala:81)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at os.package$.<clinit>(package.scala:19)
        ... 25 common frames omitted
ERROR c.d.c.ServerRunner - Unexpected error while running server: null
INFO  c.d.c.ServerRunner - Exception causing error is:
java.lang.ExceptionInInitializerError: null
        at ammonite.Main$.apply$default$5(Main.scala:68)
        at com.digitalasset.canton.console.HeadlessConsole$.createDefaultAmmoniteOptions(HeadlessConsole.scala:59)
        at com.digitalasset.canton.console.HeadlessConsole$.$anonfun$apply$1(HeadlessConsole.scala:46)
        at better.files.Dispose.apply(Dispose.scala:81)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at os.package$.<clinit>(package.scala:19)
        ... 25 common frames omitted

Okay, progress! Well back to the original problem…

So that still looks like the new temporary path isnt being picked up. Could you do the following just to really make sure it’s picking up our new temporary file path:

./bin/canton daemon -Djava.io.tmpdir=/home/app/temp -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug

Okay, so this is different:
when running the exact command you sent:

./bin/canton daemon -Djava.io.tmpdir=/home/app/temp -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug

we get:

INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:3d72cae7793ab98823e8071e6d7fdbcc - Manual start requested.
ERROR c.d.c.ServerRunner - An internal error occurred while executing script.
java.nio.file.NoSuchFileException: /home/app/temp/ammonite_cache8685047869954133097
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)        at java.base/java.nio.file.Files.createDirectory(Files.java:690)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:135)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:172)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:1007)
        at better.files.File$.newTemporaryDirectory(File.scala:1360)
        at better.files.File$.temporaryDirectory(File.scala:1369)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
ERROR c.d.c.ServerRunner - Unexpected error while running server: /home/app/temp/ammonite_cache8685047869954133097
INFO  c.d.c.ServerRunner - Exception causing error is:
java.nio.file.NoSuchFileException: **/home/app/temp/ammonite_cache8685047869954133097**
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
        at java.base/java.nio.file.Files.createDirectory(Files.java:690)
        at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:135)
        at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:172)
        at java.base/java.nio.file.Files.createTempDirectory(Files.java:1007)
        at better.files.File$.newTemporaryDirectory(File.scala:1360)
        at better.files.File$.temporaryDirectory(File.scala:1369)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)

showing that the path got picked up. However the folder name is tmp, so I changed that in the following command and:

sh-4.4$ bin/canton daemon -Djava.io.tmpdir=/home/app/tmp -c examples/01-simple-topology/simple-topology.conf --bootstrap=examples/01-simple-topology/simple-ping.canton --log-file-name=$PWD/tmp/canton.log --debug
INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:de2f7115e072bddf7b897dcabed0fbbc - Manual start requested.
ERROR c.d.c.ServerRunner - An internal error occurred while executing script.
java.lang.ExceptionInInitializerError: null
        at ammonite.Main$.apply$default$5(Main.scala:68)
        at com.digitalasset.canton.console.HeadlessConsole$.createDefaultAmmoniteOptions(HeadlessConsole.scala:59)
        at com.digitalasset.canton.console.HeadlessConsole$.$anonfun$apply$1(HeadlessConsole.scala:46)
        at better.files.Dispose.apply(Dispose.scala:81)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at os.package$.<clinit>(package.scala:19)
        ... 25 common frames omitted
ERROR c.d.c.ServerRunner - Unexpected error while running server: null
INFO  c.d.c.ServerRunner - Exception causing error is:
java.lang.ExceptionInInitializerError: null
        at ammonite.Main$.apply$default$5(Main.scala:68)
        at com.digitalasset.canton.console.HeadlessConsole$.createDefaultAmmoniteOptions(HeadlessConsole.scala:59)
        at com.digitalasset.canton.console.HeadlessConsole$.$anonfun$apply$1(HeadlessConsole.scala:46)
        at better.files.Dispose.apply(Dispose.scala:81)
        at com.digitalasset.canton.console.HeadlessConsole$.apply(HeadlessConsole.scala:45)
        at com.digitalasset.canton.ConsoleScriptRunner$.run(Runner.scala:169)
        at com.digitalasset.canton.ServerRunner.startWithBootstrap$1(Runner.scala:51)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3(Runner.scala:53)
        at com.digitalasset.canton.ServerRunner.$anonfun$run$3$adapted(Runner.scala:53)
        at scala.Option.fold(Option.scala:263)
        at com.digitalasset.canton.ServerRunner.run(Runner.scala:53)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:145)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:28)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:28)
        at com.digitalasset.canton.CantonEnterpriseApp.main(CantonEnterpriseApp.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at os.package$.<clinit>(package.scala:19)
        ... 25 common frames omitted

With this different exception it looks like if fails somewhere after writing to the folder.

Sorry about suggesting .../temp instead of /tmp. My bad.

Okay, this library is trying to convert the current working directory (pwd) to an absolute path and somehow not producing an absolute path. Could you show me echo $PWD just out of interest?

This is coming from the portion trying to run the bootstrap script so lets instead try to load the console now we have the temp dir sorted out but without loading our fancy repl (the --no-tty flag):

bin/canton daemon --no-tty -Djava.io.tmpdir=/home/app/tmp -c examples/01-simple-topology/simple-topology.conf --log-file-name=$PWD/tmp/canton.log --debug

@Matheus

Please check the documentation on how to run scripts remotely. It’s quite easy:
https://www.canton.io/docs/dev/user-manual/usermanual/console.html#remote-administration

On the other note: Right now, there are three issues:
(1) ammonite caching between process restarts
(2) ammonite struggling to determine pwd
(3) ammonite requiring ability to create temp files

The “requirement failed: ? is not an absolute path” is related to (2). With the version you have, you can currently not work around that issue. Here, the following java code seems to produce an invalid result:

java.nio.file.Paths.get(".").toAbsolutePath

I’ve modified Canton (not released yet) to make (1) and (2) manually configurable and allow in-memory caching for (1) and manually configure what the pwd is for (2). I’ll point you to the “snapshot release” once I merged the code.

In order to fix (3) we need to make sure that

java.io.File.createTempFile()

works on your setup, which is what @davidpadbury tried to suggest.

Therefore, my suggestion now is:
(1) please test from your side that temp file creation works.
(2) we’ll let you know where the daily snapshot versions of the community edition are so that you can test out the fix.

Cheers,
Ratko

Hi guys,

@davidpadbury so this looks more promising.

So firstly echo $PWD seems to work fine:

sh-4.4$ echo $PWD
/home/app

Secondly, running with --no-tty gives us much more feedback:

INFO  c.d.c.CantonEnterpriseApp$ - Starting Canton version 0.27.0
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment - Creating ForkJoinPool with parallelism = 2 (instead of 1) to avoid starvation.
INFO  a.e.s.Slf4jLogger - Slf4jLogger started
INFO  c.d.c.e.EnterpriseEnvironment - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.e.EnterpriseEnvironment tid:de2ea15e0626895cd8bc5babf15ac3f6 - Automatically starting all instances
INFO  c.d.c.p.EnterpriseParticipantNodeFactory$$anon$1:participant=participant1 - Starting admin-api services on AdminServerConfig(127.0.0.1,6012,None,Some(KeepAliveConfig(40s,20s)),10485760)
INFO  c.d.c.p.EnterpriseParticipantNodeFactory$$anon$1:participant=participant1 - Node is not initialized yet. Performing automated default initialization.
INFO  c.d.c.p.EnterpriseParticipantNodeFactory$$anon$1:participant=participant1 - Initializing node with id NodeId(participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,7fc7fbf8-be14-4c03-8e82-84c9548238c5)
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2702a61d824f122cae276fca452fb6fb - Applied identity transaction NamespaceDelegation(
  namespace = 1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,
  target = SigningPublicKey(id = 1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d, format = Tink, scheme = Ed25519),
  isRootDelegation = true
) at 2021-09-11T12:36:19.063309Z
INFO  c.d.c.p.p.PruningProcessorImpl:participant=participant1 - Pruning status: ParticipantPruningStatus()
INFO  c.d.c.p.l.a.CantonLedgerApiServerWrapper$Config:participant=participant1 - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.c.p.l.a.CantonLedgerApiServerWrapper$Config:participant=participant1 - Deriving 1 as number of threads from 'sys.runtime.availableProcessors()'. Please use '-Dscala.concurrent.context.numThreads' to override.
INFO  c.d.p.s.d.HikariConnection - Attempting to connect to the database (attempt 1/600), context: {participant: "participant1"}
INFO  c.z.h.HikariDataSource - daml.index.db.connection.migrations - Starting...
INFO  c.z.h.HikariDataSource - daml.index.db.connection.migrations - Start completed.
INFO  c.d.p.s.FlywayMigrationsCanton - Ensuring Flyway migration has either not started or there are no pending migrations..., context: {participant: "participant1"}
INFO  o.f.c.i.l.VersionPrinter - Flyway Community Edition 7.8.1 by Redgate
INFO  o.f.c.i.d.b.DatabaseType - Database: jdbc:h2:mem:ledger_api_participant11220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d_a4604687697c4edab06b0b8e07e58ec7 (H2 1.4)
INFO  c.d.p.s.FlywayMigrationsCanton - Running Flyway migration on empty database with 33 migrations pending..., context: {participant: "participant1"}
INFO  o.f.c.i.l.VersionPrinter - Flyway Community Edition 7.8.1 by Redgate
INFO  o.f.c.i.c.DbValidate - Successfully validated 33 migrations (execution time 00:00.014s)
INFO  o.f.c.i.s.JdbcTableSchemaHistory - Creating Schema History table "PUBLIC"."flyway_schema_history" ...
INFO  o.f.c.i.c.DbMigrate - Current version of schema "PUBLIC": << Empty Schema >>
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "1 - Init"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "2 - Command deduplication"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "3 - Contract Divulgence"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "4 - Contract Divulgence"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "5.0 - Extract Event Data"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "5.1 - Populate Event Data"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "5.2 - Extract Event Data"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "6 - Disclosures index"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "7 - Add configuration"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "8 - Party entries"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "9 - Package entries"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "10 - Loosen transaction check"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "11 - Create command completions table"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "12 - Command deduplication"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "13 - Events new schema"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "14 - Stable offsets"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "15 - Remove maximum record time"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "16 - Delete checkpoints"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "17 - Stable offsets archival"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "18 - Contracts new schema"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "19 - Events table fixes"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "20 - Drop old schema"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "21 - Event witnesses single table"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "22 - Add witnesses to participant events"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "23 - Parties is local"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "24 - event sequential id"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "25 - drop participant id"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "26 - add participant id to parameters"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "27 - Participant-pruning"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "28 - multiple submitters"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "29 - explicit compression"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "30 - offset as text"
INFO  o.f.c.i.c.DbMigrate - Migrating schema "PUBLIC" to version "99 - Contract keys canton"
INFO  o.f.c.i.c.DbMigrate - Successfully applied 33 migrations to schema "PUBLIC", now at version v99 (execution time 00:00.620s)
INFO  c.d.p.s.FlywayMigrationsCanton - Flyway schema migration finished successfully, applying org.flywaydb.core.api.output.MigrateResult@7cf6cd1 steps on empty database., context: {participant: "participant1"}
INFO  c.z.h.HikariDataSource - daml.index.db.connection.migrations - Shutdown initiated...
INFO  c.z.h.HikariDataSource - daml.index.db.connection.migrations - Shutdown completed.
INFO  c.d.p.i.RecoveringIndexer - Starting Indexer Server, context: {participant: "participant1"}
INFO  c.d.p.s.d.HikariConnection - Attempting to connect to the database (attempt 1/600), context: {participant: "participant1"}
INFO  c.z.h.HikariDataSource - daml.index.db.connection.indexer - Starting...
INFO  c.z.h.HikariDataSource - daml.index.db.connection.indexer - Start completed.
INFO  c.d.p.i.JdbcIndexerCanton$Factory - Initializing ledger with ID: participant1, context: {participant: "participant1"}
INFO  c.d.p.i.RecoveringIndexer - Started Indexer Server, context: {participant: "participant1"}
INFO  c.d.p.s.d.JdbcLedgerDaoCanton - Storing configuration entry, context: {participant: "participant1", submissionId: "TimeModel config"}
INFO  c.d.p.s.d.HikariConnection - Attempting to connect to the database (attempt 1/600), context: {participant: "participant1"}
INFO  c.z.h.HikariDataSource - daml.index.db.connection.api-server - Starting...
INFO  c.z.h.HikariDataSource - daml.index.db.connection.api-server - Start completed.
INFO  c.d.p.i.ReadOnlySqlLedger - Found existing ledger with ID: participant1, context: {participant: "participant1"}
INFO  c.d.p.a.ApiServicesCanton - Daml-LF Engine supports LF versions: 1.14, context: {participant: "participant1"}
INFO  c.d.p.a.s.LedgerConfigProvider - Initial ledger configuration lookup found configuration Configuration(1,LedgerTimeModel(PT0S,PT8760H,PT8760H),PT24H) at Absolute(000000000000000001). Looking for new ledger configurations from this offset., context: {participant: "participant1"}
INFO  c.d.p.a.LedgerApiServer - Listening on 127.0.0.1:6011 over plain text., context: {participant: "participant1"}
INFO  c.d.p.a.StandaloneApiServerCanton - Initialized API server version 1.16.0 with ledger-id = participant1, port = 6011, dar file = List(), context: {participant: "participant1"}
INFO  c.d.p.s.d.JdbcLedgerDaoCanton - Storing package entry, context: {participant: "participant1"}
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2f713e9acba764f0ff37bd8df95bce76 - Applied identity transaction VettedPackages(
  participant = participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,
  packages = Seq(
    65921e553a35...,
    57b5c520512c...,
    cb0552debf21...,
    3f4deaf145a1...,
    86828b984346...,
    f20de1e4e37b...,
    76bf0fd12bd9...,
    d58cf9939847...,
    40f452260bef...,
    e491352788e5...,
    6839a6d3d430...,
    518032f41fd0...,
    852d8e3a8ccf...,
    bfcd37bd6b84...,
    cc348d369011...,
    057eed1fd48c...,
    d14e08374fc7...,
    c1f1f0055879...,
    6c2c0667393c...,
    e22bce619ae2...,
    8a7806365bbd...,
    97b883cd8a2b...,
    733e38d36a27...,
    99a2705ed38c...
  )
) at 2021-09-11T12:36:23.340667Z
INFO  c.d.p.a.s.ApiLedgerIdentityService - Received request for ledger identity: GetLedgerIdentityRequest(), context: {participant: "participant1"}
INFO  c.d.p.a.s.ApiCommandCompletionService - Received request for completion subscription 0: CompletionStreamRequest(participant1,admin-ping,Set(participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d),Some(Absolute(000000000000000002))), context: {participant: "participant1", parties: ["participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d"], offset: "000000000000000002"}
INFO  c.d.p.a.s.ApiLedgerIdentityService - Received request for ledger identity: GetLedgerIdentityRequest(), context: {participant: "participant1"}
INFO  c.d.p.a.s.t.ApiTransactionService - Received request for transactions., context: {participant: "participant1", startExclusive: "000000000000000002", endInclusive: null, filters: {participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d: "all-templates"}, verbose: false, ledgerId: "participant1"}
INFO  c.d.p.a.s.ApiCommandCompletionService - Received request for completion subscription 1: CompletionStreamRequest(participant1,admin-dar-distribution,Set(participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d),Some(Absolute(000000000000000002))), context: {participant: "participant1", parties: ["participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d"], offset: "000000000000000002"}
INFO  c.d.p.a.s.t.ApiTransactionService - Received request for transactions., context: {participant: "participant1", startExclusive: "000000000000000002", endInclusive: null, filters: {participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d: "all-templates"}, verbose: false, ledgerId: "participant1"}
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2702a61d824f122cae276fca452fb6fb - Applied identity transaction OwnerToKeyMapping(
  owner = PAR::participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,
  key = SigningPublicKey(id = 122043951d8c04df595c2531b9653cef30544d30fc8fe508abf8889f84270ff01585, format = Tink, scheme = Ed25519)
) at 2021-09-11T12:36:23.915756Z
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2702a61d824f122cae276fca452fb6fb - Applied identity transaction OwnerToKeyMapping(
  owner = PAR::participant1::1220e9bba026c7e607ac2cddb652675a3b61c16fff743eb64b01de8cc9ccb9efa09d,
  key = EncryptionPublicKey(id = 122081ce8ee7ba7384975e718f89189b141738998a00b9803fcd6ebda45f079cb4ac, format = Tink, scheme = ECIES-P256_HMAC256_AES128-GCM)
) at 2021-09-11T12:36:23.922569Z
INFO  c.d.c.p.i.ParticipantIdentityManager:participant=participant1 tid:2702a61d824f122cae276fca452fb6fb - Applied identity transaction SignedLegalIdentityClaim(signature = Signature(signature = 306602310084..., signedBy = 12208d7e6a9a31f2e089dc029cf9bf7a58a81ad6aad0d9332a9e085e09c4c592c659), claim = ...) at 2021-09-11T12:36:24.196814Z
INFO  c.d.c.p.s.CantonSyncService:participant=participant1 tid:de2ea15e0626895cd8bc5babf15ac3f6 - Reconnecting to domains List()
INFO  c.d.c.p.s.CantonSyncService:participant=participant1 tid:de2ea15e0626895cd8bc5babf15ac3f6 - Successfully re-connected to domains
INFO  c.d.c.e.EnterpriseEnvironment tid:de2ea15e0626895cd8bc5babf15ac3f6 - Successfully started all nodes
INFO  c.d.c.ServerRunner - Canton started

@Ratko_Veprek thanks for the link, I’ll take a look in the docs.

I’ll also do a test with java.io.File.createTempFile() using a separate java project and post the result here.

Thanks both for the great support!

Cheers,
Matheus

Hello gentlemen,

So, although running a remote console locally was no problem, it has proven to be quite the struggle on our current setup due to the restrictions of our test environment.

Is there a ETA on the fix for the bootstrap script? I believe running that would take us most of the way we need to go.

Thanks in advance!

Best,
Matheus

Hi Matheus!

Here is a daily snapshot build that has an improvement with the ammonite caches:
https://www.canton.io/releases/canton-community-20210923.tar.gz

You have two additional configuration options:

canton.parameters.console.cache-dir = 
canton.parameters.console.working-dir = 

By default, these parameters are inferred. But if you define them we should at least get around the “?” path issue.

Can you then run

bin/canton daemon --no-tty -Djava.io.tmpdir=/home/app/tmp -c examples/01-simple-topology/simple-topology.conf --log-file-name=$PWD/tmp/canton.log --debug --bootstrap examples/01-simple-topology/simple-ping.canton

One of the last log lines should contain

INFO  c.d.c.p.p.TransactionProcessor:participant=participant2/domain=mydomain tid:.. - Finalizing Transaction request at RequestId(2021-09-24T10:00:14.931475Z) with event Some(TransactionAccepted(

If you see that then script execution works.

If that worked, then run the same command above again, but without daemon. Let’s hope you’ll get a console.

Best,
Ratko

1 Like

Hi Ratko,

Thanks for the response. So this is what I get now trying the 0.28.0 snapshot:

sh-4.4$ bin/canton daemon --no-tty -Djava.io.tmpdir=/home/app/tmp -c newconf.conf --log-file-name=$PWD/tmp/canton.log --debug
INFO  c.d.c.CantonCommunityApp$ - Starting Canton version 0.28.0-SNAPSHOT
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: ? is not an absolute path
        at scala.Predef$.require(Predef.scala:338)
        at os.Path.<init>(Path.scala:430)
        at os.Path$.apply(Path.scala:388)
        at ammonite.main.Defaults$.ammoniteHome(Defaults.scala:32)
        at com.digitalasset.canton.console.AmmoniteConsoleConfig$.defaultCacheDir(AmmoniteConsoleConfig.scala:30)
        at com.digitalasset.canton.console.AmmoniteConsoleConfig$.apply$default$1(AmmoniteConsoleConfig.scala:22)
        at com.digitalasset.canton.config.CantonParameters$.apply$default$8(CantonConfig.scala:256)
        at com.digitalasset.canton.config.CantonCommunityConfig$.apply$default$6(CantonCommunityConfig.scala:31)
        at com.digitalasset.canton.config.CantonCommunityConfig$anon$lazy$macro$194$1.inst$macro$138$lzycompute(CantonCommunityConfig.scala:67)
        at com.digitalasset.canton.config.CantonCommunityConfig$anon$lazy$macro$194$1.inst$macro$138(CantonCommunityConfig.scala:67)
        at com.digitalasset.canton.config.CantonCommunityConfig$.cantonCommunityConfigReader$lzycompute(CantonCommunityConfig.scala:67)
        at com.digitalasset.canton.config.CantonCommunityConfig$.cantonCommunityConfigReader(CantonCommunityConfig.scala:61)
        at com.digitalasset.canton.config.CantonCommunityConfig$.load(CantonCommunityConfig.scala:82)
        at com.digitalasset.canton.CantonCommunityApp$.loadConfig(CantonCommunityApp.scala:17)
        at com.digitalasset.canton.CantonAppDriver.delayedEndpoint$com$digitalasset$canton$CantonAppDriver$1(CantonAppDriver.scala:95)
        at com.digitalasset.canton.CantonAppDriver$delayedInit$body.apply(CantonAppDriver.scala:29)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at com.digitalasset.canton.CantonAppDriver.main(CantonAppDriver.scala:29)
        at com.digitalasset.canton.CantonCommunityApp.main(CantonCommunityApp.scala)

I tried both variants below in the config file:

canton {
  parameters {
    manual-start = yes
    console.cache-dir = /home/app/tmp
    console.working-dir = /home/app/tmp
  }
  ....
canton {
  parameters {
    manual-start = yes
    console.cache-dir = "/home/app/tmp"
    console.working-dir = "/home/app/tmp"
  }
  ....

But the result is the same, maybe there’s another path it’s failing to determine?

Cheers,
Matheus

Can you try to additionally start with

./bin/canton -Duser.home=/home/app ... 

It seems like

System.getProperty("user.home")

yields

?
1 Like

That did the trick :slight_smile: seems to be up and running, removing daemon made the interactive console appear.

Thanks a lot for the great support!

Can we expect these changes to be in the next Canton Enterprise release?

Yes. Enterprise is a highly enhanced version of the community release to with parallel transaction processing (much faster), high availability, ledger pruning. So generally, enterprise supports everything that community supports, but not the other way around.

1 Like