Bareos Bug Tracker
Bareos Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000779bareos-core[All Projects] directorpublic2017-02-08 22:332018-01-31 15:48
Reporterdpcushing 
Assigned To 
PrioritynormalSeverityminorReproducibilityalways
StatusnewResolutionopen 
PlatformLinuxOSCentOSOS Version7
Product Version16.2.4 
Target VersionFixed in Version 
Summary0000779: Always Incremental Consolidate job does not prune the storage volumes
DescriptionParagraph 23.3.3 in the Bareos manual has a note in the storage pool that says 'Don't prune jobs from the volume. Let this be done by Consolidate.' The consolidate job does purge the jobs from the database, but it does not prune the storage pool volume(s). Volumes with no reference to existing jobs are left in the in the Full or Used state until they are manually pruned.
Steps To ReproduceConfigure daily backups with Always Incremental and configure a daily consolidation job. None of the volumes from the AI related pools will be recycled.
Additional InformationI first discover this when my pool fill it unexpectedly to it's Maximum Volumes. You can query the database with the below and any volume that is returned and does not have a status of 'Purged' is a problem.

SELECT m.MediaId, m.VolumeName, m.VolStatus, m.Recycle
FROM bareos.Media m
WHERE NOT EXISTS (SELECT 1 FROM JobMedia jm
 WHERE jm.MediaId = m.MediaId)
;
TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action
Attached Files

- Relationships

-  Notes
(0002606)
dpcushing (reporter)
2017-03-14 01:27

For anybody else that may be facing this issue, I wrote a shell script that I run in an admin job after running consolidate the prunes the unused volumes. Here's the content ...

#!/bin/bash
# grab the database credentials from existing configuration files
catalogFile=`find /etc/bareos/bareos-dir.d/catalog/ -type f`
dbUser=`grep dbuser $catalogFile | grep -o '".*"' | sed 's/"//g'`
dbPwd=`grep dbpassword $catalogFile | grep -o '".*"' | sed 's/"//g'`

# Get a list of volumes no longer in use and submit them to the console for pruning.
# Query for a list of volumes
emptyVols=$(mysql bareos -u $dbUser -p$dbPwd -se "SELECT m.VolumeName FROM bareos.Media m where m.VolStatus not in ('Append','Purged') and not exists (select 1 from bareos.JobMedia jm where jm.MediaId=m.MediaId);")
# Submit volumes to bconsole for pruning
for volName in $emptyVols
do
/bin/bconsole << EOD
prune volume=$volName yes
quit
EOD
done
(0002894)
morfair (reporter)
2018-01-31 12:18

Does this mean that unnecessary old deleted data will remain in consolidated forever?
(0002895)
dpcushing (reporter)
2018-01-31 15:08

@morfair - As noted, the jobs and associated contents are correctly purged from the database. So the database does not grow out of control and does not contain the 'old deleted data' forever.

Because the volumes are no pruned, they can never be recycled. So the problem is that your storage pools, both the incremental and the consolidated, will grow until you run out of space. The 'old deleted data' will remain in those volumes forever or until they are manually pruned. The script I provided resolves this issue by finding and pruning those non referenced volumes after the consolidation job is completed.
(0002896)
morfair (reporter)
2018-01-31 15:12

@dpcushing - Sorry for some offtopic, but I do not understand how to remove jobs that was consolidated. "prune jobs" has now affect and I have many backuped files that no presend in FD older 1 month. I dont want keep this data and don't know how to set retention period for AI jobs.
(0002897)
dpcushing (reporter)
2018-01-31 15:48

@morfair - This probably isn't appropriate for discussion under this bug report. I'd recommend that you search the bareos-users group to see if your question is answered there. If not, post a new question and there are lots of helpful users who can provide you with recommended solutions.

https://groups.google.com/forum/?fromgroups#!forum/bareos-users [^]

- Issue History
Date Modified Username Field Change
2017-02-08 22:33 dpcushing New Issue
2017-03-14 01:27 dpcushing Note Added: 0002606
2018-01-31 12:18 morfair Note Added: 0002894
2018-01-31 15:08 dpcushing Note Added: 0002895
2018-01-31 15:12 morfair Note Added: 0002896
2018-01-31 15:48 dpcushing Note Added: 0002897


Copyright © 2000 - 2018 MantisBT Team
Powered by Mantis Bugtracker