Database backup and disaster recovery

Questions
1

How can I back up and restore my validator and participant databases for disaster recovery? How do I use it to recover?

2

To create backup you can follow the documentation[1]
Regularly back up both the participant and validator databases within your DB instance. Make sure that participant DB backup especially is up to date . This helps ensure proper recovery order and avoids data mismatches.

For recovery, the exact steps required depends on your database but as an example, here is a detailed way of recovering from a PostgreSQL database backup.

Restoration Steps:

  1. Scale Down Services:
    Stop or scale down all nodes (participant and validator) that use the databases to prevent new writes during restoration.
  2. Restore Databases:
  • Use pg_restore or your preferred backup tool to restore the participant database first, then the validator database.
  1. Check Data Consistency:
  • Confirm that the participant DB is at least as recent as the validator DB. If the validator DB is ahead, it may cause recovery issues.
  1. Don’t restore Node Identities:
  • For a total recovery, restoring the databases is enough. However, in case of issue with you DB backup, make sure to always have your have node identity backups. you can use it to restore your node as well, especially if you are redeploying nodes from scratch.
  1. Bring Services Back Online:
    Start the participant and validator nodes and monitor logs for any errors or warnings.
  2. Test Recovery:
  • Validate that both nodes are operational and synchronized.
  • Test application functionality to ensure full recovery.

Best Practices:

  • Automate and schedule regular backups for both databases and node identities.
  • Document and periodically test your disaster recovery process in a production-like environment.

By following these steps, you can maximize your chances of a successful recovery after a disaster.
Make sure to check the official documentation for Disaster recovery[1:1]

  1. This link is for dev environment documentation, make sure to check the documentation according to your env ↩︎ ↩︎

2 Likes