Where can I find a list of all JMX metrics exposed by the sandbox and JSON API service?

I’m aware there a ticket to document these, but in the meantime is there any place in the source where I can look these up?

1 Like

Researching I found that I can use jconsole as a tool to explore the reported metrics visually. That’ll do in the meantime, but if anyone has a better solution to collect all metrics exposed, please let me know.

The definitions of all metrics has recently been centralized to make documenting them easier :wink:
Have a look at Metrics.scala.

  • The metrics are structured hierarchically as scala objects.
  • Start with object daml as the root and go down the object hierarchy, which also reflects the metric name. The metric daml.index.db.store_transaction can be found at
object daml { 
  object index {
     object db {
        val storeTransaction = ... ("store_transaction")
     }
  }
}
  • Each metric has potentially multiple data points (e.g. for Timers there are mean, p50, p90, p95, count, …)
4 Likes

I’ve now used jconsole to pull out a comprehensive list of metrics that the sandbox exposes. Here they are for reference:

counters
  attributes
    Count
  names
    daml.kvutils.committer.config.accepts
    daml.kvutils.committer.config.rejections
    daml.kvutils.committer.exceptions
    daml.kvutils.committer.processing

gauges
  attributes
    Value
    Number
  names
    daml.index.db.connection.api-server.pool.ActiveConnections
    daml.index.db.connection.api-server.pool.IdleConnections
    daml.index.db.connection.api-server.pool.MaxConnections
    daml.index.db.connection.api-server.pool.MinConnections
    daml.index.db.connection.api-server.pool.PendingConnections
    daml.index.db.connection.api-server.pool.TotalConnections
    daml.index.db.connection.indexer.pool.ActiveConnections
    daml.index.db.connection.indexer.pool.IdleConnections
    daml.index.db.connection.indexer.pool.MaxConnections
    daml.index.db.connection.indexer.pool.MinConnections
    daml.index.db.connection.indexer.pool.PendingConnections
    daml.index.db.connection.indexer.pool.TotalConnections
    daml.indexer.current_record_time_lag
    daml.indexer.last_received_offset
    daml.indexer.last_received_record_time
    daml.kvutils.committer.last.entry_id
    daml.kvutils.committer.last.exception
    daml.kvutils.committer.last.participant_id
    daml.kvutils.committer.last.record_time
    daml.kvutils.submission.validator.state_value_cache.size
    daml.kvutils.submission.validator.state_value_cache.weight
    jvm.attributes.name
    jvm.attributes.uptime
    jvm.attributes.vendor
    jvm.class_loader.loaded
    jvm.class_loader.unloaded
    jvm.garbage_collector.G1-Old-Generation.count
    jvm.garbage_collector.G1-Old-Generation.time
    jvm.garbage_collector.G1-Young-Generation.count
    jvm.garbage_collector.G1-Young-Generation.time
    jvm.memory_usage.heap.committed
    jvm.memory_usage.heap.init
    jvm.memory_usage.heap.max
    jvm.memory_usage.heap.usage
    jvm.memory_usage.heap.used
    jvm.memory_usage.non-heap.committed
    jvm.memory_usage.non-heap.init
    jvm.memory_usage.non-heap.max
    jvm.memory_usage.non-heap.usage
    jvm.memory_usage.non-heap.used
    jvm.memory_usage.pools.CodeHeap-'non-nmethods'.committed
    jvm.memory_usage.pools.CodeHeap-'non-nmethods'.init
    jvm.memory_usage.pools.CodeHeap-'non-nmethods'.max
    jvm.memory_usage.pools.CodeHeap-'non-nmethods'.usage
    jvm.memory_usage.pools.CodeHeap-'non-nmethods'.used
    jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.committed
    jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.init
    jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.max
    jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.usage
    jvm.memory_usage.pools.CodeHeap-'non-profiled-nmethods'.used
    jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.committed
    jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.init
    jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.max
    jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.usage
    jvm.memory_usage.pools.CodeHeap-'profiled-nmethods'.used
    jvm.memory_usage.pools.Compressed-Class-Space.committed
    jvm.memory_usage.pools.Compressed-Class-Space.init
    jvm.memory_usage.pools.Compressed-Class-Space.max
    jvm.memory_usage.pools.Compressed-Class-Space.usage
    jvm.memory_usage.pools.Compressed-Class-Space.used
    jvm.memory_usage.pools.G1-Eden-Space.committed
    jvm.memory_usage.pools.G1-Eden-Space.init
    jvm.memory_usage.pools.G1-Eden-Space.max
    jvm.memory_usage.pools.G1-Eden-Space.usage
    jvm.memory_usage.pools.G1-Eden-Space.used
    jvm.memory_usage.pools.G1-Eden-Space.used-after-gc
    jvm.memory_usage.pools.G1-Old-Gen.committed
    jvm.memory_usage.pools.G1-Old-Gen.init
    jvm.memory_usage.pools.G1-Old-Gen.max
    jvm.memory_usage.pools.G1-Old-Gen.usage
    jvm.memory_usage.pools.G1-Old-Gen.used
    jvm.memory_usage.pools.G1-Old-Gen.used-after-gc
    jvm.memory_usage.pools.G1-Survivor-Space.committed
    jvm.memory_usage.pools.G1-Survivor-Space.init
    jvm.memory_usage.pools.G1-Survivor-Space.max
    jvm.memory_usage.pools.G1-Survivor-Space.usage
    jvm.memory_usage.pools.G1-Survivor-Space.used
    jvm.memory_usage.pools.G1-Survivor-Space.used-after-gc
    jvm.memory_usage.pools.Metaspace.committed
    jvm.memory_usage.pools.Metaspace.init
    jvm.memory_usage.pools.Metaspace.max
    jvm.memory_usage.pools.Metaspace.usage
    jvm.memory_usage.pools.Metaspace.used
    jvm.memory_usage.total.committed
    jvm.memory_usage.total.init
    jvm.memory_usage.total.max
    jvm.memory_usage.total.used
    jvm.thread_states.blocked.count
    jvm.thread_states.count
    jvm.thread_states.daemon.count
    jvm.thread_states.deadlock.count
    jvm.thread_states.deadlocks
    jvm.thread_states.new.count
    jvm.thread_states.runnable.count
    jvm.thread_states.terminated.count
    jvm.thread_states.timed_waiting.count
    jvm.thread_states.waiting.count

histograms
  attributes
    Min
    Max
    Count
    Mean
    StdDev
    75thPercentile
    95thPercentile
    98thPercentile
    99thPercentile
    999thPercentile
    50thPercentile
    SnapshotSize
  names
    daml.index.db.connection.api-server.pool.ConnectionCreation
    daml.index.db.connection.api-server.pool.Usage
    daml.index.db.connection.indexer.pool.ConnectionCreation
    daml.index.db.connection.indexer.pool.Usage

meters
  attributes
    Count
    RateUnit
    OneMinuteRate
    FiveMinuteRate
    FifteenMinuteRate
    MeanRate
  names
    daml.index.db.connection.api-server.pool.ConnectionTimeoutRate
    daml.index.db.connection.indexer.pool.ConnectionTimeoutRate

timers
  attributes
    Min
    Max
    DurationUnit
    Mean
    StdDev
    75thPercentile
    95thPercentile
    98thPercentile
    99thPercentile
    999thPercentile
    50thPercentile
    Count
    RateUnit
    OneMinuteRate
    FiveMinuteRate
    FifteenMinuteRate
    MeanRate
  names
    daml.index.db.all.exec
    daml.index.db.all.wait
    daml.index.db.connection.api-server.pool.Wait
    daml.index.db.connection.indexer.pool.Wait
    daml.index.db.deduplicate_command
    daml.index.db.get_initial_ledger_end.exec
    daml.index.db.get_initial_ledger_end.wait
    daml.index.db.get_ledger_end.exec
    daml.index.db.get_ledger_end.wait
    daml.index.db.get_ledger_id.exec
    daml.index.db.get_ledger_id.wait
    daml.index.db.get_lf_archive
    daml.index.db.get_parties
    daml.index.db.initialize_ledger_parameters.exec
    daml.index.db.initialize_ledger_parameters.wait
    daml.index.db.list_known_parties
    daml.index.db.list_lf_packages
    daml.index.db.load_configuration_entries.exec
    daml.index.db.load_configuration_entries.wait
    daml.index.db.lookup_active_contract
    daml.index.db.lookup_configuration.exec
    daml.index.db.lookup_configuration.wait
    daml.index.db.lookup_key
    daml.index.db.lookup_ledger_configuration
    daml.index.db.lookup_ledger_end
    daml.index.db.lookup_ledger_id
    daml.index.db.lookup_maximum_ledger_time
    daml.index.db.lookup_transaction
    daml.index.db.remove_expired_deduplication_data
    daml.index.db.remove_expired_deduplication_data.exec
    daml.index.db.remove_expired_deduplication_data.wait
    daml.index.db.stop_deduplicating_command
    daml.index.db.store_configuration_entry
    daml.index.db.store_configuration_entry.exec
    daml.index.db.store_configuration_entry.wait
    daml.index.db.store_initial_state
    daml.index.db.store_ledger_entry
    daml.index.db.store_package_entry
    daml.index.db.store_party_entry
    daml.index.db.store_rejection
    daml.index.deduplicate_command
    daml.index.get_lf_archive
    daml.index.get_lf_package
    daml.index.get_parties
    daml.index.list_known_parties
    daml.index.list_lf_packages
    daml.index.lookup_contract
    daml.index.lookup_flat_transaction_by_id
    daml.index.lookup_key
    daml.index.lookup_ledger_configuration
    daml.index.lookup_maximum_ledger_time
    daml.index.lookup_transaction_tree_by_id
    daml.index.publish_configuration
    daml.index.publish_party_allocation
    daml.index.publish_transaction
    daml.index.remove_expired_deduplication_data
    daml.index.stop_deduplicating_command
    daml.index.upload_packages
    daml.indexer.processed_state_updates
    daml.kvutils.committer.config.run_timer
    daml.kvutils.committer.config.step_timers.authorize_submission
    daml.kvutils.committer.config.step_timers.build_log_entry
    daml.kvutils.committer.config.step_timers.check_ttl
    daml.kvutils.committer.config.step_timers.deduplicate_submission
    daml.kvutils.committer.config.step_timers.validate_submission
    daml.kvutils.committer.run_timer
    daml.kvutils.reader.open_envelope
    daml.kvutils.reader.parse_updates
    daml.kvutils.submission.conversion.archives_to_submission
    daml.kvutils.submission.conversion.configuration_to_submission
    daml.kvutils.submission.conversion.party_to_submission
    daml.kvutils.submission.conversion.transaction_outputs
    daml.kvutils.submission.conversion.transaction_to_submission
    daml.kvutils.submission.validator.acquire_transaction_lock
    daml.kvutils.submission.validator.commit_submission
    daml.kvutils.submission.validator.failed_to_acquire_transaction
    daml.kvutils.submission.validator.open_envelope
    daml.kvutils.submission.validator.process_submission
    daml.kvutils.submission.validator.release_transaction_lock
    daml.kvutils.submission.validator.transform_submission
    daml.kvutils.submission.validator.validate_submission
    daml.kvutils.writer.commit
    daml.ledger.database.queries.insert_record_into_log
    daml.ledger.database.queries.select_from_log
    daml.ledger.database.queries.select_latest_log_entry_id
    daml.ledger.database.queries.select_state_values_by_keys
    daml.ledger.database.queries.truncate
    daml.ledger.database.queries.update_or_retrieve_ledger_id
    daml.ledger.database.queries.update_state
    daml.ledger.database.transactions.commit.acquire_connection
    daml.ledger.database.transactions.commit.run
    daml.ledger.database.transactions.read_head.acquire_connection
    daml.ledger.database.transactions.read_head.run
    daml.ledger.database.transactions.read_log.acquire_connection
    daml.ledger.database.transactions.read_log.run
    daml.ledger.database.transactions.retrieve_ledger_id.acquire_connection
    daml.ledger.database.transactions.retrieve_ledger_id.run
    daml.ledger.log.append
    daml.ledger.log.read
    daml.ledger.state.read
    daml.ledger.state.write
    daml.services.index.configuration_entries
    daml.services.index.current_ledger_end
    daml.services.index.deduplicate_command
    daml.services.index.get_active_contracts
    daml.services.index.get_completions
    daml.services.index.get_ledger_configuration
    daml.services.index.get_ledger_id
    daml.services.index.get_lf_archive
    daml.services.index.get_lf_package
    daml.services.index.get_participant_id
    daml.services.index.get_parties
    daml.services.index.get_transaction_by_id
    daml.services.index.get_transaction_tree_by_id
    daml.services.index.list_known_parties
    daml.services.index.list_lf_packages
    daml.services.index.lookup_active_contract
    daml.services.index.lookup_configuration
    daml.services.index.lookup_contract_key
    daml.services.index.lookup_maximum_ledger_time
    daml.services.index.package_entries
    daml.services.index.party_entries
    daml.services.index.stop_deduplicating_command
    daml.services.index.transaction_trees
    daml.services.index.transactions
    daml.services.read.get_ledger_initial_conditions
    daml.services.read.state_updates
    daml.services.write.allocate_party
    daml.services.write.submit_configuration
    daml.services.write.submit_transaction
    daml.services.write.upload_packages

Happy monitoring!

4 Likes

Keep in mind that this would only show metrics that you actually “triggered” during a run. So if you had taken this snapshot right after starting the sandbox, you would have seen probably way fewer entries.

Oh, right… Might have to revisit then :pensive:

Are these metrics ledger depended? It will be great if they can be added into DAML doco with a bit descriptions.

Also is there any metrics related to ledger API client connection? e.g. number of connected ledger API client, session etc?