Pruning progress monitoring and process interruptions

Questions
,
1

  1. Are Log files available that show progress whilst pruning runs? If not, what is the recommended way to monitor the progress of pruning?

  2. What happens if the pruning process is interrupted/killed before it ends due to some reason? Do we just start it again and it will simply resume from where it left off?

1 Like
2

Hi @Mr_Mannoroth

As far as I know there are no logs produced that you should watch out for. The recommended way to monitor pruning is documented here: Ledger Pruning — Daml SDK 2.6.3 documentation

I’m not sure there is a sense of resuming where it left off, the next run should just remove every event that is fulfilling the criteria. It is highly recommended that you use automatic pruning.

Kind Regards,
Mate

1 Like
3

As Mate points out, the max-event-age metric available on participants, sequencers, and mediators is the primary gauge on whether the pruning schedule provided “enough time” to keep up with pruning. This metric shows the age in hours of the oldest (unpruned) event and also serves as the signal whether any sort of errors prevent pruning from proceeding.

For question 1, I just ran one of our pruning integration tests to see what errors any log mining you put in place might be interested in filtering for. You could look for log entries containing both “PruningScheduler” (matching pruning on participants, sequencers, and mediators) and “With Error”.

$ grep "PruningScheduler.*With Error" log/canton_test.log 
2023-05-02 10:19:58,715 [ScheduledParticipantPruningTestPostgres-env-execution-context-57] INFO  c.d.c.p.p.ParticipantPruningScheduler:ScheduledParticipantPruningTestPostgres/participant=participant1 tid:6e6fdcfe0a6a7382361bfd0327e293a5 - With Error(message = FAILED_PRECONDITION: UNSAFE_TO_PRUNE(9,ae0ff81a): Participant cannot prune at specified offset due to max deduplication duration of 1s, backoff = 1s) scheduled next 1s
2023-05-02 10:20:00,013 [ScheduledParticipantPruningTestPostgres-env-execution-context-288] INFO  c.d.c.p.p.ParticipantPruningScheduler:ScheduledParticipantPruningTestPostgres/participant=participant1 tid:d8e546aa0fe67d33784237a096568ee5 - With Error(message = FAILED_PRECONDITION: OFFSET_OUT_OF_RANGE(9,98fdfde9): prune_up_to needs to be before ledger end 000000000000000005, backoff = 1s) scheduled next 1s

To question 2.: Automatic pruning retries pruning as long as the current time falls in the pruning schedule resuming with the “next batch”, i.e. typically with the same batch attempted before the error.

Thanks,
– Oliver

2 Likes
4

Perfect - tysm @Mate_Varga and @oliverse!

1 Like