Thanks @oliverse and Cristina for the offline session.
I’d like to summarise the issues that I have encountered and the canton configs that need to be set for pruning relatively new data on ledger for testing or other purposes.
Below are example short-term pruning configs provided by @oliverse
1. pruning Participant (canton config)
# ensure that the participants don't need to keep history for more than 5m for duplicate request detection
# default was 7 days, so for short term pruning, this config need to be override
canton.participants.<participant node name>.init.ledger-api.max-deduplication-duration = <a time span shorter than (pruning timestamp - tx popluated timestamp)>
# for example
canton.participants.sandbox.init.ledger-api.max-deduplication-duration = 5m
2. pruning Sequencer and Mediator (canton config)
# to be able to prune sequencers, have each sequencer client request a time proof every 5 minutes
# note that if there is a risk that any of participants does not produce domain traffic within 10 minutes,
# ensure that they specify time-tracker.min-observation-duration of 5 minutes when connecting to the domain
# so that participants also request regular time proofs and don't block sequencer pruning.
canton.domains.<domain name>.time-tracker.min-observation-duration = 5m
# for example
canton.domains.local.time-tracker.min-observation-duration = 5m
# have the sequencer clients from all nodes acknowledge more frequently to ensure that the sequencer can prune.
canton.participants.<participant name>.sequencer-client.acknowledgement-interval = 5m
canton.domains.<domain name>.sequencer-client.acknowledgement-interval = 5m
# cause the database sequencer to perform frequent checkpoints, so that lack of checkpoints doesn't prevent pruning.
canton.domains.<domain name>.sequencer.reader.checkpoint-interval = 5m
# cut off unauthenticated participants after 5 minutes as they only use unauthenticated requests during onboarding;
# otherwise the sequencer would only cut them off after 24 hours by default which would prevent pruning for the first day.
canton.domains.<domain name>.sequencer.pruning.unauthenticated-member-retention = 5m
Note that sequencer pruning config cannot bet set without below canton domain config setting
storage.type = postgres and
sequencer.type = database
3. possible errors encountered
Exception in thread "main" io.grpc.StatusRuntimeException: FAILED_PRECONDITION: OFFSET_OUT_OF_RANGE(9,d8a6eb30): prune_up_to needs to be before ledger end <ledger end offset>
This error indicates that the pruned offset we set was too large and it had to be smaller than a certain offset.
I bumped into this error when I tried to use the end/last offset on ledger when trying to prune.
According to @oliverse this is an existing issue and there’s a fix in progress.
Participant cannot prune at specified offset due to max deduplication duration of <your dedup time in canton by setting canton.participants.<participant node name>.init.ledger-api.max-deduplication-duration>
The above error indicates that you are pruning transaction that is still within the dedup duration, and thus you are not able to prune them.
During my testing, I have shortened the config settings to 1 second and pruned after 1 second has long passed. Yet, the same error still occurred. As it turns out, the participant “time/clock” will not tick if there’s no events. The solution for my testing setup was to simply create some more transaction/event after the target pruned transaction when dedup time has elapsed.
Participant cannot prune at specified offset due to no suitable offset for domain <your domain name>
This error pops out once in a while, during my testing and was somehow resolved after below steps
- put domain onto postgres database
- set the participant dedup time config to a lower value (as described above)
- create other transaction after the intended prune offset when dedup time elapsed
GENERIC_CONFIG_ERROR(8,0): Cannot convert configuration to a config of class com.digitalasset.canton.config.CantonEnterpriseConfig. Failures are:
- (simple-topology.conf: 39) Key not found: 'type'.
When trying to set sequencer pruning config as described above, this error will occur if we have domain storage setup in memory. Moving the domain onto database can solve the issue.
PARTICIPANT_PRUNED_DATA_ACCESSED(9,8b9b8a46): Transactions request from to <your request end offset> precedes pruned offset <last pruned offset>
Above indicates that we are trying to read pruned transaction by giving the wrong offset range in while requesting transaction stream. If happens to me when I was trying to read transaction stream from Ledger Begin with below code
new FiltersByParty(Map.of(someParty, NoFilter.instance)), false);
The proper way of getting transaction stream from a pruned ledger would be, read ACS for snapshot and subscribe to transaction from the offset returned by the read-ACS call (basically the offset that ACS state ends at)