View Issue Details

IDProjectCategoryView StatusLast Update
0000585bareos-coredirectorpublic2016-01-11 14:54
Reporterurusha Assigned To 
Status acknowledgedResolutionopen 
PlatformLinuxOSUbuntuOS Version14.04
Product Version15.2.2 
Summary0000585: pool auto prune keep last jobs
DescriptionI have the next director configuration:

Client {
  Name = client1
  Address = client1
  Password = "..."
  File Retention = 5 years
  Job Retention = 5 years
Pool {
  Name = Full-1m-sd2
  Label Format = "Full-1m-sd2-"
  Volume Retention = 9 weeks
  Pool Type = Backup
  Recycle = no
  AutoPrune = yes
  Action On Purge = Truncate
  Maximum Volume Jobs = 1
Pool {
  Name = Diff-1m-sd2
  Label Format = "Diff-1m-sd2-"
  Volume Retention = 1 month 1 week
  Pool Type = Backup
  Recycle = no
  AutoPrune = yes
  Action On Purge = Truncate
  Maximum Volume Jobs = 1
Pool {
  Name = Incr-1w-sd2
  Label Format = "Incr-1w-sd2-"
  Volume Retention = 2 weeks
  Pool Type = Backup
  Recycle = no
  AutoPrune = yes
  Action On Purge = Truncate
  Maximum Volume Jobs = 1
JobDefs {
  Name = "DefaultJob-sd2"
  Type = Backup
  Schedule = "Daily-2105"
  Storage = sd2
  Priority = 10
  Write Bootstrap = "/var/lib/bareos/%c.bsr"
  Pool = Full-1m-sd2
  Full Backup Pool = Full-1m-sd2
  Differential Backup Pool = Diff-1m-sd2
  Incremental Backup Pool = Incr-1w-sd2
  Accurate = yes
  Max Full Interval = 5 weeks
  Reschedule On Error = yes
  Reschedule Interval = 1 hour
  Reschedule Times = 1
  Allow Duplicate Jobs = no
  Cancel Lower Level Duplicates = yes
  Cancel Queued Duplicates = yes

And the job, running every month, using these client, pools, jobdefs and file-based SD. So, every job has it's own volume that is purged if it's older than 9 weeks. Every job level uses its own pool.

The reason why I use this scheme is that I have not much disk space but require full backups per month for the last months, diff. backups per week for the last month and incr. backups per day for the last week. So, all incr. backups are purged after 2 weeks, and don't waste much space.

The problem is that an old volumes are purged at the start of the job - before the new volume is labeled. So, if all new jobs are failing for 9 weeks, on the next run ALL volumes with successful jobs will be purged, so there will be no full backup for the job at all.

To exclude such situation, I'm suggesting to implement a pool option like "auto prune keep last = N" (default N=0). This option modifies volume auto pruning algorithm:
  1. create the 'classic' list of volumes for auto-pruning (Media.MediaId).
  2. create the list of jobs from these volumes (Job.JobId,Job.Name).
  3. for every job (Job.Name) from the list(2) determine N most recent successful full-level backups (from current pool), determine volumes used by this backups and skip these volumes from the 'classic' list(1).
  4. run auto-pruning for resulting list of volumes.

With this algorithm all volumes used by N recent runs of every job name from this pool will not be purged. They could only be purged after client's job/file retention period expires.

This algorithm could be helpful with some other scheme, the main idea is keeping the last consistent full backup for the job.

Also, the logic could be extended to keep last N backups for every level, this would be useful when a single pool for all job levels is used:
auto prune keep last full = N
auto prune keep last diff = M
auto prune keep last incr = K

Hope this will be implemented someday.
Tagsauto prune, pool
bareos-master: impact
bareos-master: action
bareos-19.2: impact
bareos-19.2: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action




2015-12-15 11:10

reporter   ~0002037

Last edited: 2015-12-15 11:11

"And the job, running every month"
should be
"And the job, running every day"

Schedule {
  Name = "Daily-2105"
  Run = Level=Full 1st sat at 21:05
  Run = Level=Differential 2nd-5th sat at 21:05
  Run = Level=Incremental sun-fri at 21:05

Issue History

Date Modified Username Field Change
2015-12-15 11:00 urusha New Issue
2015-12-15 11:09 urusha Tag Attached: auto prune
2015-12-15 11:09 urusha Tag Attached: pool
2015-12-15 11:10 urusha Note Added: 0002037
2015-12-15 11:11 urusha Note Edited: 0002037
2016-01-11 14:54 pstorz Status new => acknowledged