View Issue Details

IDProjectCategoryView StatusLast Update
0000778bareos-coredirectorpublic2023-09-04 17:14
Reporterdpcushing Assigned Tobruno-at-bareos  
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionno change required 
PlatformLinuxOSCentOSOS Version7
Product Version16.2.4 
Summary0000778: Always Incremental Consolidation is consolidating and deleting backup copies from a DR pool
DescriptionI have configured Bareos with Always incremental backups. I have also configured a copy schedule that runs daily and copies all of the daily backup jobs to a network drive in another location. As a simple example let's assume all even job IDs (1,3,5,7,9,...) are backups and all odd job IDs (2,4,6,8,10,...) are DR copies of those backups. On day 8 a consolidation job runs and consolidates, then purges job 1. When job 1 is purges, job 2 gets 'promoted' from copy to backup. On day 9 a consolidation job runs and consolidates jobs 2 and 3, then purges job 2. This continues daily.

The end result is that each job gets consolidates twice, once as the backup and again the next day as the 'promoted' copy of the backup. After consolidation, the backups are purged and the DR copy is lost.
Steps To ReproduceSet up daily Always Incremental backups with 'Always Incremental Job Retention = days' and 'Always Incremental Keep Number = 3'. Set up a copy job to run daily to copy all backup jobs to a separate storage pool. Set up an Always Incremental Consolidate job to run daily.
Additional InformationIn the Jobs table in the database you'll see the original backup job with Type B and the copies with Type C. After a consolidation of a backup job you'll see that job purged and it's associated copy job changed to Type B. The next days you'll see the consolidate job operate on the 'used to be a copy' job, since it is now identified as a backup. Then the job is removed from the database. So the two issues identified are 1) The same backup set is always consolidated twice on consecutive days (or whatever the consolidation interval is). The DR copies, that may be set up for a longer retention period, are purged by consolidation.
TagsNo tags attached.

Activities

dpcushing

dpcushing

2017-03-14 01:30

reporter   ~0002607

For anybody else that may be facing this issue, I wrote a shell script that I run in an admin job after running consolidate that changes the newly 'promoted' backups to copies. Note in the script below that the pool where the copies reside is named 'FileCopy'. Here's the content ...

#!/bin/bash
# grab the database credentials from existing configuration files
catalogFile=`find /etc/bareos/bareos-dir.d/catalog/ -type f`
dbUser=`grep dbuser $catalogFile | grep -o '".*"' | sed 's/"//g'`
dbPwd=`grep dbpassword $catalogFile | grep -o '".*"' | sed 's/"//g'`

# Make sure all DR-Copy jobs that are in the FileCopy pool are properly set in the database as Copy (C) jobs.
/usr/bin/mysql bareos -u $dbUser -p$dbPwd -se "UPDATE Job J SET J.Type = 'C' WHERE J.Type <> 'C' AND EXISTS (SELECT 1 FROM Media M, JobMedia JM WHERE JM.JobId = J.JobId AND M.MediaId = JM.MediaID AND M.MediaType = 'FileCopy');"
brockp

brockp

2019-09-15 04:38

reporter   ~0003571

Last edited: 2019-09-15 04:59

I can confirm this issue still exists and is not documented as a limitation in 18.2.5

As far as can tell this directly conflicts with: https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html

arogge

arogge

2019-10-16 10:47

manager   ~0003602

This can be worked around as described in https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#virtual-full-jobs

"To make sure the longterm Level (Dir->Job) = VirtualFull is not taken as base for the next incrementals, the job type of the copied job is set to Type (Dir->Job) = Archive with the Run Script (Dir->Job)."

If you think the documentation needs a change to document this better, feel free to suggest something. Otherwise I'd like to close this issue.
dpcushing

dpcushing

2019-10-19 01:08

reporter   ~0003609

@arogge - The issue I have reported here is different. I do have VirtualFull backups configured for long term archiving and I do set those VirtualFull jobs to Archive via script as described in your reference link. The issue that I am describing is specific to Copy jobs. I prepare an offsite copy every night of each of the day's backups to allow me to perform an accurate recovery in the event of a complete site disaster. The problem that I've described is that when a primary AI job is consolidated and removed from the catalog, Bareos promotes the copy job to a primary job. On the next cycle that new 'primary' (used to be copy) job is consolidated again. Once that new 'primary' (used to be copy) job is consolidated it will get deleted from the catalog, making my offsite copies incomplete ;-0
bruno-at-bareos

bruno-at-bareos

2023-09-04 17:14

manager   ~0005371

The term copy here is misleading. For your purpose, you want to do a copy but then this one needs to be similar to what virtual full is, an archive, so it never interfere with AI

Issue History

Date Modified Username Field Change
2017-02-08 21:44 dpcushing New Issue
2017-03-14 01:30 dpcushing Note Added: 0002607
2019-09-15 04:38 brockp Note Added: 0003571
2019-09-15 04:53 brockp Note Edited: 0003571
2019-09-15 04:59 brockp Note Edited: 0003571
2019-10-16 10:47 arogge Note Added: 0003602
2019-10-16 10:47 arogge Status new => feedback
2019-10-19 01:08 dpcushing Note Added: 0003609
2019-10-19 01:08 dpcushing Status feedback => new
2023-09-04 17:14 bruno-at-bareos Assigned To => bruno-at-bareos
2023-09-04 17:14 bruno-at-bareos Status new => closed
2023-09-04 17:14 bruno-at-bareos Resolution open => no change required
2023-09-04 17:14 bruno-at-bareos Note Added: 0005371