View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0000778||bareos-core||[All Projects] director||public||2017-02-08 21:44||2019-10-19 01:08|
|Fixed in Version|
|Summary||0000778: Always Incremental Consolidation is consolidating and deleting backup copies from a DR pool|
|Description||I have configured Bareos with Always incremental backups. I have also configured a copy schedule that runs daily and copies all of the daily backup jobs to a network drive in another location. As a simple example let's assume all even job IDs (1,3,5,7,9,...) are backups and all odd job IDs (2,4,6,8,10,...) are DR copies of those backups. On day 8 a consolidation job runs and consolidates, then purges job 1. When job 1 is purges, job 2 gets 'promoted' from copy to backup. On day 9 a consolidation job runs and consolidates jobs 2 and 3, then purges job 2. This continues daily.|
The end result is that each job gets consolidates twice, once as the backup and again the next day as the 'promoted' copy of the backup. After consolidation, the backups are purged and the DR copy is lost.
|Steps To Reproduce||Set up daily Always Incremental backups with 'Always Incremental Job Retention = days' and 'Always Incremental Keep Number = 3'. Set up a copy job to run daily to copy all backup jobs to a separate storage pool. Set up an Always Incremental Consolidate job to run daily.|
|Additional Information||In the Jobs table in the database you'll see the original backup job with Type B and the copies with Type C. After a consolidation of a backup job you'll see that job purged and it's associated copy job changed to Type B. The next days you'll see the consolidate job operate on the 'used to be a copy' job, since it is now identified as a backup. Then the job is removed from the database. So the two issues identified are 1) The same backup set is always consolidated twice on consecutive days (or whatever the consolidation interval is). The DR copies, that may be set up for a longer retention period, are purged by consolidation.|
|Tags||No tags attached.|
For anybody else that may be facing this issue, I wrote a shell script that I run in an admin job after running consolidate that changes the newly 'promoted' backups to copies. Note in the script below that the pool where the copies reside is named 'FileCopy'. Here's the content ...
# grab the database credentials from existing configuration files
catalogFile=`find /etc/bareos/bareos-dir.d/catalog/ -type f`
dbUser=`grep dbuser $catalogFile | grep -o '".*"' | sed 's/"//g'`
dbPwd=`grep dbpassword $catalogFile | grep -o '".*"' | sed 's/"//g'`
# Make sure all DR-Copy jobs that are in the FileCopy pool are properly set in the database as Copy (C) jobs.
/usr/bin/mysql bareos -u $dbUser -p$dbPwd -se "UPDATE Job J SET J.Type = 'C' WHERE J.Type <> 'C' AND EXISTS (SELECT 1 FROM Media M, JobMedia JM WHERE JM.JobId = J.JobId AND M.MediaId = JM.MediaID AND M.MediaType = 'FileCopy');"
I can confirm this issue still exists and is not documented as a limitation in 18.2.5
As far as can tell this directly conflicts with: https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html
This can be worked around as described in https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#virtual-full-jobs
"To make sure the longterm Level (Dir->Job) = VirtualFull is not taken as base for the next incrementals, the job type of the copied job is set to Type (Dir->Job) = Archive with the Run Script (Dir->Job)."
If you think the documentation needs a change to document this better, feel free to suggest something. Otherwise I'd like to close this issue.
|@arogge - The issue I have reported here is different. I do have VirtualFull backups configured for long term archiving and I do set those VirtualFull jobs to Archive via script as described in your reference link. The issue that I am describing is specific to Copy jobs. I prepare an offsite copy every night of each of the day's backups to allow me to perform an accurate recovery in the event of a complete site disaster. The problem that I've described is that when a primary AI job is consolidated and removed from the catalog, Bareos promotes the copy job to a primary job. On the next cycle that new 'primary' (used to be copy) job is consolidated again. Once that new 'primary' (used to be copy) job is consolidated it will get deleted from the catalog, making my offsite copies incomplete ;-0|
|2017-02-08 21:44||dpcushing||New Issue|
|2017-03-14 01:30||dpcushing||Note Added: 0002607|
|2019-09-15 04:38||brockp||Note Added: 0003571|
|2019-09-15 04:53||brockp||Note Edited: 0003571||View Revisions|
|2019-09-15 04:59||brockp||Note Edited: 0003571||View Revisions|
|2019-10-16 10:47||arogge||Note Added: 0003602|
|2019-10-16 10:47||arogge||Status||new => feedback|
|2019-10-19 01:08||dpcushing||Note Added: 0003609|
|2019-10-19 01:08||dpcushing||Status||feedback => new|