View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0000779||bareos-core||[All Projects] director||public||2017-02-08 22:33||2018-01-31 15:48|
|Target Version||Fixed in Version|
|Summary||0000779: Always Incremental Consolidate job does not prune the storage volumes|
|Description||Paragraph 23.3.3 in the Bareos manual has a note in the storage pool that says 'Don't prune jobs from the volume. Let this be done by Consolidate.' The consolidate job does purge the jobs from the database, but it does not prune the storage pool volume(s). Volumes with no reference to existing jobs are left in the in the Full or Used state until they are manually pruned.|
|Steps To Reproduce||Configure daily backups with Always Incremental and configure a daily consolidation job. None of the volumes from the AI related pools will be recycled.|
|Additional Information||I first discover this when my pool fill it unexpectedly to it's Maximum Volumes. You can query the database with the below and any volume that is returned and does not have a status of 'Purged' is a problem.|
SELECT m.MediaId, m.VolumeName, m.VolStatus, m.Recycle
FROM bareos.Media m
WHERE NOT EXISTS (SELECT 1 FROM JobMedia jm
WHERE jm.MediaId = m.MediaId)
|Tags||No tags attached.|
For anybody else that may be facing this issue, I wrote a shell script that I run in an admin job after running consolidate the prunes the unused volumes. Here's the content ...
# grab the database credentials from existing configuration files
catalogFile=`find /etc/bareos/bareos-dir.d/catalog/ -type f`
dbUser=`grep dbuser $catalogFile | grep -o '".*"' | sed 's/"//g'`
dbPwd=`grep dbpassword $catalogFile | grep -o '".*"' | sed 's/"//g'`
# Get a list of volumes no longer in use and submit them to the console for pruning.
# Query for a list of volumes
emptyVols=$(mysql bareos -u $dbUser -p$dbPwd -se "SELECT m.VolumeName FROM bareos.Media m where m.VolStatus not in ('Append','Purged') and not exists (select 1 from bareos.JobMedia jm where jm.MediaId=m.MediaId);")
# Submit volumes to bconsole for pruning
for volName in $emptyVols
/bin/bconsole << EOD
prune volume=$volName yes
|Does this mean that unnecessary old deleted data will remain in consolidated forever?|
@morfair - As noted, the jobs and associated contents are correctly purged from the database. So the database does not grow out of control and does not contain the 'old deleted data' forever.
Because the volumes are no pruned, they can never be recycled. So the problem is that your storage pools, both the incremental and the consolidated, will grow until you run out of space. The 'old deleted data' will remain in those volumes forever or until they are manually pruned. The script I provided resolves this issue by finding and pruning those non referenced volumes after the consolidation job is completed.
|@dpcushing - Sorry for some offtopic, but I do not understand how to remove jobs that was consolidated. "prune jobs" has now affect and I have many backuped files that no presend in FD older 1 month. I dont want keep this data and don't know how to set retention period for AI jobs.|
@morfair - This probably isn't appropriate for discussion under this bug report. I'd recommend that you search the bareos-users group to see if your question is answered there. If not, post a new question and there are lots of helpful users who can provide you with recommended solutions.
|2017-02-08 22:33||dpcushing||New Issue|
|2017-03-14 01:27||dpcushing||Note Added: 0002606|
|2018-01-31 12:18||morfair||Note Added: 0002894|
|2018-01-31 15:08||dpcushing||Note Added: 0002895|
|2018-01-31 15:12||morfair||Note Added: 0002896|
|2018-01-31 15:48||dpcushing||Note Added: 0002897|