I’m aware there a ticket to document these, but in the meantime is there any place in the source where I can look these up?
1 Like
Researching I found that I can use jconsole as a tool to explore the reported metrics visually. That’ll do in the meantime, but if anyone has a better solution to collect all metrics exposed, please let me know.
The definitions of all metrics has recently been centralized to make documenting them easier 
Have a look at Metrics.scala.
- The metrics are structured hierarchically as scala
objects. - Start with
object damlas the root and go down the object hierarchy, which also reflects the metric name. The metricdaml.index.db.store_transactioncan be found at
object daml {
object index {
object db {
val storeTransaction = ... ("store_transaction")
}
}
}
- Each metric has potentially multiple data points (e.g. for
Timers there are mean, p50, p90, p95, count, …)
4 Likes
I’ve now used jconsole to pull out a comprehensive list of metrics that the sandbox exposes. Here they are for reference:
counters
attributes
Count
names
daml.kvutils.committer.config.accepts
daml.kvutils.committer.config.rejections
daml.kvutils.committer.exceptions
daml.kvutils.committer.processing
gauges
attributes
Value
Number
names
daml.index.db.connection.api-server.pool.ActiveConnections
daml.index.db.connection.api-server.pool.IdleConnections
daml.index.db.connection.api-server.pool.MaxConnections
daml.index.db.connection.api-server.pool.MinConnections
daml.index.db.connection.api-server.pool.PendingConnections
daml.index.db.connection.api-server.pool.TotalConnections
daml.index.db.connection.indexer.pool.ActiveConnections
daml.index.db.connection.indexer.pool.IdleConnections
daml.index.db.connection.indexer.pool.MaxConnections
daml.index.db.connection.indexer.pool.MinConnections
daml.index.db.connection.indexer.pool.PendingConnections
daml.index.db.connection.indexer.pool.TotalConnections
daml.indexer.current_record_time_lag
daml.indexer.last_received_offset
daml.indexer.last_received_record_time
daml.kvutils.committer.last.entry_id
daml.kvutils.committer.last.exception
daml.kvutils.committer.last.participant_id
daml.kvutils.committer.last.record_time
daml.kvutils.submission.validator.state_value_cache.size
daml.kvutils.submission.validator.state_value_cache.weight
jvm.attributes.name
jvm.attributes.uptime
jvm.attributes.vendor
jvm.class_loader.loaded
jvm.class_loader.unloaded
jvm.garbage_collector.G1-Old-Generation.count
jvm.garbage_collector.G1-Old-Generation.time
jvm.garbage_collector.G1-Young-Generation.count
jvm.garbage_collector.G1-Young-Generation.time
jvm.memory_usage.heap.committed
jvm.memory_usage.heap.init
jvm.memory_usage.heap.max
jvm.memory_usage.heap.usage
jvm.memory_usage.heap.used
jvm.memory_usage.non-heap.committed
jvm.memory_usage.non-heap.init
jvm.memory_usage.non-heap.max
jvm.memory_usage.non-heap.usage
jvm.memory_usage.non-heap.used
jvm.memory_usage.pools.CodeHeap-'non-nmethods'.committed
jvm.memory_usage.pools.CodeHeap-'non-nmethods'.init
jvm.memory_usage.pools.CodeHeap-'non-nmethods'.max
jvm.memory_usage.pools.CodeHeap-'non-nmethods'.usage
jvm.memory_usage.pools.CodeHeap-'non-nmethods'.used
jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.committed
jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.init
jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.max
jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.usage
jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.used
jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.committed
jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.init
jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.max
jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.usage
jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.used
jvm.memory_usage.pools.Compressed-Class-Space.committed
jvm.memory_usage.pools.Compressed-Class-Space.init
jvm.memory_usage.pools.Compressed-Class-Space.max
jvm.memory_usage.pools.Compressed-Class-Space.usage
jvm.memory_usage.pools.Compressed-Class-Space.used
jvm.memory_usage.pools.G1-Eden-Space.committed
jvm.memory_usage.pools.G1-Eden-Space.init
jvm.memory_usage.pools.G1-Eden-Space.max
jvm.memory_usage.pools.G1-Eden-Space.usage
jvm.memory_usage.pools.G1-Eden-Space.used
jvm.memory_usage.pools.G1-Eden-Space.used-after-gc
jvm.memory_usage.pools.G1-Old-Gen.committed
jvm.memory_usage.pools.G1-Old-Gen.init
jvm.memory_usage.pools.G1-Old-Gen.max
jvm.memory_usage.pools.G1-Old-Gen.usage
jvm.memory_usage.pools.G1-Old-Gen.used
jvm.memory_usage.pools.G1-Old-Gen.used-after-gc
jvm.memory_usage.pools.G1-Survivor-Space.committed
jvm.memory_usage.pools.G1-Survivor-Space.init
jvm.memory_usage.pools.G1-Survivor-Space.max
jvm.memory_usage.pools.G1-Survivor-Space.usage
jvm.memory_usage.pools.G1-Survivor-Space.used
jvm.memory_usage.pools.G1-Survivor-Space.used-after-gc
jvm.memory_usage.pools.Metaspace.committed
jvm.memory_usage.pools.Metaspace.init
jvm.memory_usage.pools.Metaspace.max
jvm.memory_usage.pools.Metaspace.usage
jvm.memory_usage.pools.Metaspace.used
jvm.memory_usage.total.committed
jvm.memory_usage.total.init
jvm.memory_usage.total.max
jvm.memory_usage.total.used
jvm.thread_states.blocked.count
jvm.thread_states.count
jvm.thread_states.daemon.count
jvm.thread_states.deadlock.count
jvm.thread_states.deadlocks
jvm.thread_states.new.count
jvm.thread_states.runnable.count
jvm.thread_states.terminated.count
jvm.thread_states.timed_waiting.count
jvm.thread_states.waiting.count
histograms
attributes
Min
Max
Count
Mean
StdDev
75thPercentile
95thPercentile
98thPercentile
99thPercentile
999thPercentile
50thPercentile
SnapshotSize
names
daml.index.db.connection.api-server.pool.ConnectionCreation
daml.index.db.connection.api-server.pool.Usage
daml.index.db.connection.indexer.pool.ConnectionCreation
daml.index.db.connection.indexer.pool.Usage
meters
attributes
Count
RateUnit
OneMinuteRate
FiveMinuteRate
FifteenMinuteRate
MeanRate
names
daml.index.db.connection.api-server.pool.ConnectionTimeoutRate
daml.index.db.connection.indexer.pool.ConnectionTimeoutRate
timers
attributes
Min
Max
DurationUnit
Mean
StdDev
75thPercentile
95thPercentile
98thPercentile
99thPercentile
999thPercentile
50thPercentile
Count
RateUnit
OneMinuteRate
FiveMinuteRate
FifteenMinuteRate
MeanRate
names
daml.index.db.all.exec
daml.index.db.all.wait
daml.index.db.connection.api-server.pool.Wait
daml.index.db.connection.indexer.pool.Wait
daml.index.db.deduplicate_command
daml.index.db.get_initial_ledger_end.exec
daml.index.db.get_initial_ledger_end.wait
daml.index.db.get_ledger_end.exec
daml.index.db.get_ledger_end.wait
daml.index.db.get_ledger_id.exec
daml.index.db.get_ledger_id.wait
daml.index.db.get_lf_archive
daml.index.db.get_parties
daml.index.db.initialize_ledger_parameters.exec
daml.index.db.initialize_ledger_parameters.wait
daml.index.db.list_known_parties
daml.index.db.list_lf_packages
daml.index.db.load_configuration_entries.exec
daml.index.db.load_configuration_entries.wait
daml.index.db.lookup_active_contract
daml.index.db.lookup_configuration.exec
daml.index.db.lookup_configuration.wait
daml.index.db.lookup_key
daml.index.db.lookup_ledger_configuration
daml.index.db.lookup_ledger_end
daml.index.db.lookup_ledger_id
daml.index.db.lookup_maximum_ledger_time
daml.index.db.lookup_transaction
daml.index.db.remove_expired_deduplication_data
daml.index.db.remove_expired_deduplication_data.exec
daml.index.db.remove_expired_deduplication_data.wait
daml.index.db.stop_deduplicating_command
daml.index.db.store_configuration_entry
daml.index.db.store_configuration_entry.exec
daml.index.db.store_configuration_entry.wait
daml.index.db.store_initial_state
daml.index.db.store_ledger_entry
daml.index.db.store_package_entry
daml.index.db.store_party_entry
daml.index.db.store_rejection
daml.index.deduplicate_command
daml.index.get_lf_archive
daml.index.get_lf_package
daml.index.get_parties
daml.index.list_known_parties
daml.index.list_lf_packages
daml.index.lookup_contract
daml.index.lookup_flat_transaction_by_id
daml.index.lookup_key
daml.index.lookup_ledger_configuration
daml.index.lookup_maximum_ledger_time
daml.index.lookup_transaction_tree_by_id
daml.index.publish_configuration
daml.index.publish_party_allocation
daml.index.publish_transaction
daml.index.remove_expired_deduplication_data
daml.index.stop_deduplicating_command
daml.index.upload_packages
daml.indexer.processed_state_updates
daml.kvutils.committer.config.run_timer
daml.kvutils.committer.config.step_timers.authorize_submission
daml.kvutils.committer.config.step_timers.build_log_entry
daml.kvutils.committer.config.step_timers.check_ttl
daml.kvutils.committer.config.step_timers.deduplicate_submission
daml.kvutils.committer.config.step_timers.validate_submission
daml.kvutils.committer.run_timer
daml.kvutils.reader.open_envelope
daml.kvutils.reader.parse_updates
daml.kvutils.submission.conversion.archives_to_submission
daml.kvutils.submission.conversion.configuration_to_submission
daml.kvutils.submission.conversion.party_to_submission
daml.kvutils.submission.conversion.transaction_outputs
daml.kvutils.submission.conversion.transaction_to_submission
daml.kvutils.submission.validator.acquire_transaction_lock
daml.kvutils.submission.validator.commit_submission
daml.kvutils.submission.validator.failed_to_acquire_transaction
daml.kvutils.submission.validator.open_envelope
daml.kvutils.submission.validator.process_submission
daml.kvutils.submission.validator.release_transaction_lock
daml.kvutils.submission.validator.transform_submission
daml.kvutils.submission.validator.validate_submission
daml.kvutils.writer.commit
daml.ledger.database.queries.insert_record_into_log
daml.ledger.database.queries.select_from_log
daml.ledger.database.queries.select_latest_log_entry_id
daml.ledger.database.queries.select_state_values_by_keys
daml.ledger.database.queries.truncate
daml.ledger.database.queries.update_or_retrieve_ledger_id
daml.ledger.database.queries.update_state
daml.ledger.database.transactions.commit.acquire_connection
daml.ledger.database.transactions.commit.run
daml.ledger.database.transactions.read_head.acquire_connection
daml.ledger.database.transactions.read_head.run
daml.ledger.database.transactions.read_log.acquire_connection
daml.ledger.database.transactions.read_log.run
daml.ledger.database.transactions.retrieve_ledger_id.acquire_connection
daml.ledger.database.transactions.retrieve_ledger_id.run
daml.ledger.log.append
daml.ledger.log.read
daml.ledger.state.read
daml.ledger.state.write
daml.services.index.configuration_entries
daml.services.index.current_ledger_end
daml.services.index.deduplicate_command
daml.services.index.get_active_contracts
daml.services.index.get_completions
daml.services.index.get_ledger_configuration
daml.services.index.get_ledger_id
daml.services.index.get_lf_archive
daml.services.index.get_lf_package
daml.services.index.get_participant_id
daml.services.index.get_parties
daml.services.index.get_transaction_by_id
daml.services.index.get_transaction_tree_by_id
daml.services.index.list_known_parties
daml.services.index.list_lf_packages
daml.services.index.lookup_active_contract
daml.services.index.lookup_configuration
daml.services.index.lookup_contract_key
daml.services.index.lookup_maximum_ledger_time
daml.services.index.package_entries
daml.services.index.party_entries
daml.services.index.stop_deduplicating_command
daml.services.index.transaction_trees
daml.services.index.transactions
daml.services.read.get_ledger_initial_conditions
daml.services.read.state_updates
daml.services.write.allocate_party
daml.services.write.submit_configuration
daml.services.write.submit_transaction
daml.services.write.upload_packages
Happy monitoring!
4 Likes
Keep in mind that this would only show metrics that you actually “triggered” during a run. So if you had taken this snapshot right after starting the sandbox, you would have seen probably way fewer entries.
Oh, right… Might have to revisit then 
Are these metrics ledger depended? It will be great if they can be added into DAML doco with a bit descriptions.
Also is there any metrics related to ledger API client connection? e.g. number of connected ledger API client, session etc?
All the metrics that do not depend on the underlying ledger implementation (i.e. those for the participant node) have been documented as part of the ticket linked in the first post of this thread. For the latest version at the time of writing (1.5) you can find them here.
1 Like