Barman 3.11 and PostgreSQL 17: A Powerful Duo Delivering Seamless Enterprise-Grade Backup Strategies
This blog was co-authored by David Wagoner and Tim Boutin.
Introduction
With the release of PostgreSQL17 arriving in fall 2024, the database community is abuzz over a pivotal new feature: incremental backup. This enhancement holds the promise of significantly improving backup efficiency by enabling faster, smaller backups compared to traditional full backups alone, depending on the amount of data changed between backups. Barman 3.11 introduced a metric called “Resources saving” that shows an estimate of resources saved by taking incremental backups, as compared to the full cluster size (full backup).
This blog delves into why this feature is a game-changer and how it fits into a comprehensive backup strategy.
How Incremental Backup Delivers Greater Flexibility
Incremental backups are a cornerstone of effective database backup strategies. They provide flexibility and efficiency by only backing up the data that has changed since the last backup. This approach can save time and storage space, making it an attractive option for many organizations. The incremental backup feature in PostgreSQL 17 is poised to provide users with greater flexibility in their backup routines. For instance, a common backup schedule might include one full backup weekly, daily incremental backups, and continuous write-ahead log backups. This combination offers a mix of backup types, protection, and point-in-time recoverability for many scenarios.
Cutting Recovery Time by 95% with Incremental Backups
In early testing, supported by EDB, we’ve seen a significant improvement in recovery times for PostgreSQL, dropping from 78 minutes to just 4 minutes in one test, a reduction in recovery time by almost 95%. The key to this drastic reduction lies in using incremental backups.
Here’s how it works:
- Full Backup vs. Incremental Backup: A full backup copies everything in the database, while an incremental backup only captures the changes made since the last backup.
- Recovery Process: By taking regular full backups and adding incremental backups in between, you can speed up the recovery process significantly. Instead of replaying all the data changes (WAL) from the last full backup, you only need to apply the smaller, incremental changes.
This approach shows that using incremental backups not only saves time but also makes the recovery process much more efficient.
Enhancing the Typical Backup Schedule
A well-rounded backup schedule is crucial for maintaining data integrity and availability. Typically, a weekly full backup is complemented by daily incremental backups and continuous archive log backups. This setup ensures that data can be restored to any point in time during the backup retention period. Additionally, each database must have a clearly defined and tested RTO (Recovery Time Objective) and RPO (Recovery Point Objective) to ensure the backup regimen meets user expectations. The introduction of incremental backups in PostgreSQL 17 enhances the flexibility and efficiency of Postgres backup and recovery capabilities, catering to diverse organizational requirements.
Key Concepts
Understanding the importance of RTO and RPO is essential for any backup strategy. RTO refers to the time it takes to restore the database after a failure, while RPO indicates the maximum tolerable period during which data might be lost due to an incident. The new incremental backup feature in PostgreSQL 17 supports these objectives by offering more efficient backup options and, most importantly, potentially faster restores.
Feature Breakdown
The incremental backup feature combines several new features that together enable the creation and restoration of incremental backups. While these new features are powerful and exciting, they are still considered "primitives"—the fundamental building blocks of a larger backup strategy. Enterprises with critical PostgreSQL databases will need to either develop software to automate backups on a schedule, manage backup metadata, enforce retention policies, and simplify restore operations, or use a backup tool that offers these capabilities.
Automated Backup Tools
Barman (Backup and Recovery Manager) is a Postgres backup tool and an open-source project with contributors from around the globe. Initially developed by 2nd Quadrant and now maintained by EDB after the acquisition of 2nd Quadrant in 2020, Barman provides robust backup and recovery solutions for Postgres. Key features of Barman include support for multiple Postgres servers, easy configuration of backup retention policies, backup compression, WAL archiving and WAL streaming (RPO=0), and flexible restore options. EDB also offers 24/7 technical support for Barman, ensuring users have access to experienced Postgres professionals for their most critical applications.
Historical Context
Barman was the first PostgreSQL backup tool to offer incremental backups, using innovative techniques like Linux file hard links and rsync for file-level incremental backups. This was a significant enhancement in PostgreSQL backup capabilities at the time. The new incremental backup feature in PostgreSQL 17 extends earlier innovations by providing block-level incremental backup directly within the PostgreSQL core codebase.
Current Development
The Barman development team at EDB has been working diligently to add support for the new incremental backup feature set in PostgreSQL 17. By the time PostgreSQL 17 and EDB Postgres Advanced Server 17 arrive, the just-released Barman 3.11 version will already support these features, providing a seamless transition for users. You can read more about how Barman 3.11 supports incremental backups here.
Barman's enterprise-level features, including multi-server backup, backup compression, and comprehensive reporting, make it a valuable tool for managing Postgres backups efficiently.
Summary
The incremental backup feature in PostgreSQL 17 represents a significant leap forward, enhancing backup efficiency and flexibility for users.
In short, what kind of advantage you get out of incremental backup depends on how you use it. You can spend more time doing backups and use more storage to get faster recovery times in some scenarios -- and in other scenarios, you can accept greater recovery times to save on storage and time spent running backups. Postgres users get more control and flexibility with this new feature.
This feature, when combined with the support of Barman, provides a robust solution for managing PostgreSQL backups. We look forward to seeing the positive impact this will have on the PostgreSQL community and invite everyone to join us in celebrating this achievement. Thank you to all the contributors who made this possible.
Acknowledgments
The development of the incremental backup feature in PostgreSQL 17 is the result of the hard work and dedication of many individuals. Special thanks go to Robert Haas, Chief Database Scientist at EnterpriseDB, and the reviewers and testers who spent more than a year ensuring the feature's successful integration. Additionally, we extend our gratitude to Martin Marques, Director of Engineering for Database Tools and the talented EDB developers who contributed to supporting the new feature set in Barman. Their efforts have made a significant impact on PostgreSQL's backup and recovery capabilities.
How to Learn More
Learn more about what PostgreSQL 17 offers you from EDB’s leading Postgres experts. EDB Vice President and Postgres Evangelist Bruce Momjian and EDB CTO Marc Linster discuss their favorite features—including Incremental backup—in our "Exploring PostgreSQL 17: New Features & Enhancements" webinar. Register today for this on-demand webinar to learn more about leveraging new PostgreSQL 17 performance improvements to enhance database efficiency and speed.
Plus, stay tuned for more Barman 3.11 release news, as well as EDB analysis into the automation Barman delivers to your backup and recovery operations.