Hey @lashenhurst, thanks for the interesting question.
Most Daml components (excluding the compiler) are written in Scala and run inside the JVM. I will try and explain a little bit about how the JVM works at runtime. Forgive me if you know all this already.
The Scala code is compiled to Java bytecode, which is then interpreted by the JVM runtime when you launch the component. This bytecode is not directly compiled to machine code; instead, the runtime runs it using an interpreter for a while. This is slower but allows it to collect information about how the code works. After a function is called enough times (typically 100,000 calls, but it’s also time-dependent and configurable), the JVM will compile the bytecode to native machine code to improve the performance, using the data it gathered earlier to decide how best to optimize it. All this probably explains why you see a large response time at first, which decreases once the JVM has optimized the code.
Now let’s talk about memory. The JVM uses a form of garbage collection: effectively, once memory is no longer referenced, it doesn’t deallocate it straight away. Instead it waits until it gets fairly close to the limit (configured at startup using the
-Xmx flag) and then collects all the unused memory at once, typically also compacting the “live” data so that large areas can be allocated more effectively. This involves pausing the program for brief periods of time so that there are no race conditions while the garbage collector moves objects around in memory. I expect these pauses are what you’re seeing during your spikes.
To verify whether it’s the garbage collector which causes the spikes, I suggest enabling garbage collection logging. For Java SE 11 and newer, you can do this by adding the
--verbose:gc flag, which I think works on older versions as well.
Once you’ve verified that the garbage collector is causing the spikes, I suggest reading the Garbage Collector Tuning Guide published by Oracle to dig into how you can reduce these pauses.
I’m not sure how you’re running these components, so if you need help getting these parameters to the right process, just yell. You might have some luck setting the
JAVA_OPTS environment variable if there is no way to modify the arguments passed to the
java process on the command line directly.
If any of this doesn’t explain the spikes you’re seeing, please let us know so we can try and get to the bottom of things together.