How to micro-benchmark daml code

Questions
1

What options are there to micro-benchmark a particular piece of daml code? I found a description in this article, is this still the best way to do it?

2

If you have access to the enterprise edition of the SDK, I’d recommend the Daml Profiler. If that’s not an option, the method described in the blog is probably still the best.

1 Like
3

Please keep in mind that benchmarking and profiling answer different questions and using a profiler for benchmarking can often lead to the wrong conclusions:

Benchmarking tries to answer the question “how fast is X” usually as an absolute duration. (You might then compare those durations relative to other implementations). The durations you get here will be very close to what X will take in your production system.

Profiling on the other hand, tries to answer the question “how big is the impact of different parts of X on its total runtime”. There are two caveats here:

  1. While there are different ways to implement a profiler, all of them have some overhead (the exact overhead can vary drastically depending on the approach). So the absolute values are usually not useful and you should use it only as relative values.
  2. The overhead is not always proportional to the original execution time. Certain operations have higher overhead (a common example are function calls) while other operations might have no overhead. That means while the percentages of your execution time are still useful, relative comparisons of two different implementations can easily be very misleading and the difference might be due to different overhead rather than one implementation being faster than the other.

The workflow you usually want to follow is iterating through this:

  1. Setup a benchmark that tests the code you care about. Often it’s also useful to setup an acceptance criteria so you don’t spend time optimizing past a certain point that isn’t relevant.
  2. Assuming you don’t yet meet your target, use the profiler to find the areas that are most likely to lead to significant improvements.
  3. Implement an improvement.
  4. Validate that your improvement really payed off using the benchmark not the profiler.