View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001589bareos-coredirectorpublic2024-01-08 15:592024-01-08 17:23
Reporterraschu Assigned Tobruno-at-bareos  
PriorityurgentSeveritymajorReproducibilityalways
Status closedResolutionno change required 
Summary0001589: Critical - Always Incremental: Bareos purge volumes and destroy virtual full backups
DescriptionHi Bareos-Team,

I use Bareos version 23.0.1 and I set always incremental for some clients.
The example configuration (hostnames renamed):

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Job {
      Name = host1
      Client = host1
      JobDefs = host1
      Enabled = yes
      AlwaysIncremental = yes
      AlwaysIncrementalJobRetention = 30 days
      Accurate = yes
      Pool = aincr30d-bsd02
      Full Backup Pool = aicons30d-bsd02
    }
JobDefs {
      Name = host1
      Type = Backup
      Client = host1
      Fileset = host1
      Schedule = ai30d-4h00m
      Storage = bsd02
      Messages = host1
      Pool = Incremental
      Priority = 10
      Write Bootstrap = /var/lib/bareos/%c.bsr
    }
Pool {
  Name = aicons30d-bsd02 # 30 days always incremental consolidated
  Pool Type = Backup
  Volume Retention = 31 days # How long should the always incremental consolidated Backups be kept? + 1 day buffer
  Maximum Volume Bytes = 50 g # Limit Volume size to something reasonable
  Volume Use Duration = 23 hours # defines the time period that the Volume can be written
  Maximum Volume Jobs = 1 # nur ein Job pro Volume, damit fehlgeschlagene Jobs besser gelöscht werden können
  Label Format = "aicons30d-bsd02-" # Volumes label
  Storage = bsd02
}
Job {
    Name = "Consolidate"
    Type = "Consolidate"
    Accurate = "yes"
    JobDefs = "Consolidate"
}
JobDefs {
  Name = "Consolidate"
  Type = Backup
  Level = Incremental
  Client = bareos-fd
  FileSet = "SelfTest"
  Schedule = "Consolidate"
  Storage = local
  Messages = Standard
  Priority = 10
  Write Bootstrap = "/var/lib/bareos/%c.bsr"
  Pool = Incremental
  Full Backup Pool = Full # write Full Backups into "Full" Pool
  Differential Backup Pool = Differential # write Diff Backups into "Differential" Pool
  Incremental Backup Pool = Incremental # write Incr Backups into "Incremental" Pool
}
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

After a runtime for 30 days the consolidate job created a first virtual full backup for host1. Works good.

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
07-Jan 08:00 bareos-dir JobId 8093: BAREOS 23.0.1~pre7.606b211eb (19Dec23): 07-Jan-2024 08:00:02
  JobId: 8093
  Job: Consolidate.2024-01-07_08.00.00_38
  Scheduled time: 07-Jan-2024 08:00:00
  Start time: 07-Jan-2024 08:00:02
  End time: 07-Jan-2024 08:00:02
  Bareos binary info: Bareos community build (UNSUPPORTED): Get professional support from https://www.bareos.com
  Job triggered by: Scheduler
  Termination: Consolidate OK

07-Jan 08:00 bareos-dir JobId 8094: Start Virtual Backup JobId 8094, Job=host1.2024-01-07_08.00.02_39
07-Jan 08:00 bareos-dir JobId 8094: Bootstrap records written to /var/lib/bareos/bareos-dir.restore.1.bsr
07-Jan 08:00 bareos-dir JobId 8094: Consolidating JobIds 6589,6629 containing 165170 files
07-Jan 08:00 bareos-dir JobId 8094: Connected Storage daemon at storagehost:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
07-Jan 08:00 bareos-dir JobId 8094: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
07-Jan 08:00 bareos-dir JobId 8094: Using Device "bstore02-vd01" to read.
07-Jan 08:00 bareos-dir JobId 8094: There are no more Jobs associated with Volume "aicons30d-bsd02-1743". Marking it purged.
07-Jan 08:00 bareos-dir JobId 8094: All records pruned from Volume "aicons30d-bsd02-1743"; marking it "Purged"
07-Jan 08:00 bareos-dir JobId 8094: Recycled volume "aicons30d-bsd02-1743"
07-Jan 08:00 bareos-dir JobId 8094: Using Device "bstore02-vd02" to write.
07-Jan 08:00 bareos-sd JobId 8094: Recycled volume "aicons30d-bsd02-1743" on device "bstore02-vd02" (/bstore02), all previous data lost.
07-Jan 08:00 bareos-dir JobId 8094: Max Volume jobs=1 exceeded. Marking Volume "aicons30d-bsd02-1743" as Used.
07-Jan 08:00 bareos-sd JobId 8094: Ready to read from volume "aicons30d-bsd02-1739" on device "bstore02-vd01" (/bstore02).
07-Jan 08:00 bareos-sd JobId 8094: Forward spacing Volume "aicons30d-bsd02-1739" to file:block 0:239.
07-Jan 08:01 bareos-sd JobId 8094: End of Volume at file 2 on device "bstore02-vd01" (/bstore02), Volume "aicons30d-bsd02-1739"
07-Jan 08:01 bareos-sd JobId 8094: Ready to read from volume "aincr30d-bsd02-1784" on device "bstore02-vd01" (/bstore02).
07-Jan 08:01 bareos-sd JobId 8094: Forward spacing Volume "aincr30d-bsd02-1784" to file:block 0:237.
07-Jan 08:01 bareos-sd JobId 8094: End of Volume at file 0 on device "bstore02-vd01" (/bstore02), Volume "aincr30d-bsd02-1784"
07-Jan 08:01 bareos-sd JobId 8094: End of all volumes.
07-Jan 08:01 bareos-sd JobId 8094: Elapsed time=00:01:12, Transfer rate=156.3 M Bytes/second
07-Jan 08:01 bareos-sd JobId 8094: Releasing device "bstore02-vd02" (/bstore02).
07-Jan 08:01 bareos-sd JobId 8094: Releasing device "bstore02-vd01" (/bstore02).
07-Jan 08:01 bareos-dir JobId 8094: Insert of attributes batch table with 165170 entries start
07-Jan 08:01 bareos-dir JobId 8094: Insert of attributes batch table done
07-Jan 08:01 bareos-dir JobId 8094: Joblevel was set to joblevel of first consolidated job: Full
07-Jan 08:01 bareos-dir JobId 8094: Replicating deleted files from jobids 6589,6629 to jobid 8094
07-Jan 08:01 bareos-dir JobId 8094: Bareos bareos-dir 23.0.1~pre7.606b211eb (19Dec23):
  Build OS: Debian GNU/Linux 11 (bullseye)
  JobId: 8094
  Job: host1.2024-01-07_08.00.02_39
  Backup Level: Virtual Full
  Client: "host1" 23.0.0~pre1266.75ca8a396 (24Nov23) Red Hat Enterprise Linux Server release 7.9 (Maipo),redhat
  FileSet: "host1" 2023-12-01 04:00:02
  Pool: "aicons30d-bsd02" (From Job Pool's NextPool resource)
  Catalog: "MyCatalog" (From Client resource)
  Storage: "bsd02" (From Storage from Pool's NextPool resource)
  Scheduled time: 07-Jan-2024 08:00:02
  Start time: 08-Dez-2023 04:00:13
  End time: 08-Dez-2023 04:00:34
  Elapsed time: 21 secs
  Priority: 10
  Allow Mixed Priority: no
  SD Files Written: 165,170
  SD Bytes Written: 11,257,248,106 (11.25 GB)
  Rate: 536059.4 KB/s
  Volume name(s): aicons30d-bsd02-1743
  Volume Session Id: 31
  Volume Session Time: 1704452670
  Last Volume Bytes: 11,269,963,089 (11.26 GB)
  SD Errors: 0
  SD termination status: OK
  Accurate: yes
  Bareos binary info: Bareos community build (UNSUPPORTED): Get professional support from https://www.bareos.com
  Job triggered by: User
  Termination: Backup OK

07-Jan 08:01 bareos-dir JobId 8094: purged JobIds 6589,6629 as they were consolidated into Job 8094
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

Next day, Bareos run the consolidation again and find a other host (host2) for the first virtual full:

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
08-Jan 08:00 bareos-dir JobId 8140: BAREOS 23.0.1~pre7.606b211eb (19Dec23): 08-Jan-2024 08:00:02
  JobId: 8140
  Job: Consolidate.2024-01-08_08.00.00_28
  Scheduled time: 08-Jan-2024 08:00:00
  Start time: 08-Jan-2024 08:00:02
  End time: 08-Jan-2024 08:00:02
  Bareos binary info: Bareos community build (UNSUPPORTED): Get professional support from https://www.bareos.com
  Job triggered by: Scheduler
  Termination: Consolidate OK

08-Jan 08:00 bareos-dir JobId 8141: Start Virtual Backup JobId 8141, Job=host2.2024-01-08_08.00.02_29
08-Jan 08:01 bareos-dir JobId 8141: Bootstrap records written to /var/lib/bareos/bareos-dir.restore.2.bsr
08-Jan 08:01 bareos-dir JobId 8141: Consolidating JobIds 6592,6653 containing 3401249 files
08-Jan 08:01 bareos-dir JobId 8141: Connected Storage daemon at storagehost:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
08-Jan 08:01 bareos-dir JobId 8141: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
08-Jan 08:01 bareos-dir JobId 8141: Using Device "bstore02" to read.
08-Jan 08:01 bareos-dir JobId 8141: There are no more Jobs associated with Volume "aicons30d-bsd02-1739". Marking it purged.
08-Jan 08:01 bareos-dir JobId 8141: All records pruned from Volume "aicons30d-bsd02-1739"; marking it "Purged"
08-Jan 08:01 bareos-dir JobId 8141: Recycled volume "aicons30d-bsd02-1739"
08-Jan 08:01 bareos-dir JobId 8141: Using Device "bstore02-vd01" to write.
08-Jan 08:01 bareos-sd JobId 8141: Recycled volume "aicons30d-bsd02-1739" on device "bstore02-vd01" (/bstore02), all previous data lost.
08-Jan 08:01 bareos-dir JobId 8141: Max Volume jobs=1 exceeded. Marking Volume "aicons30d-bsd02-1739" as Used.
08-Jan 08:01 bareos-sd JobId 8141: Ready to read from volume "aicons30d-bsd02-1742" on device "bstore02" (/bstore02).
08-Jan 08:01 bareos-sd JobId 8141: Forward spacing Volume "aicons30d-bsd02-1742" to file:block 0:239.
08-Jan 08:07 bareos-sd JobId 8141: End of Volume at file 12 on device "bstore02" (/bstore02), Volume "aicons30d-bsd02-1742"
08-Jan 08:07 bareos-sd JobId 8141: Ready to read from volume "aicons30d-bsd02-1756" on device "bstore02" (/bstore02).
08-Jan 08:07 bareos-sd JobId 8141: Forward spacing Volume "aicons30d-bsd02-1756" to file:block 0:239.
08-Jan 08:07 bareos-sd JobId 8141: User defined maximum volume capacity 53,687,091,200 exceeded on device "bstore02-vd01" (/bstore02).
08-Jan 08:07 bareos-sd JobId 8141: End of medium on Volume "aicons30d-bsd02-1739" Bytes=53,686,042,747 Blocks=51,199 at 08-Jan-2024 08:07.
08-Jan 08:07 bareos-dir JobId 8141: Volume "aicons30d-bsd02-1743" has Volume Retention of 2678400 sec. and has 1 jobs that will be pruned
08-Jan 08:07 bareos-dir JobId 8141: Purging the following 1 JobIds: 8094
08-Jan 08:07 bareos-dir JobId 8141: There are no more Jobs associated with Volume "aicons30d-bsd02-1743". Marking it purged.
08-Jan 08:07 bareos-dir JobId 8141: All records pruned from Volume "aicons30d-bsd02-1743"; marking it
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

!!!HERE IS THE PROBLEM:

>>>
08-Jan 08:07 bareos-dir JobId 8141: Volume "aicons30d-bsd02-1743" has Volume Retention of 2678400 sec. and has 1 jobs that will be pruned
08-Jan 08:07 bareos-dir JobId 8141: Purging the following 1 JobIds: 8094
08-Jan 08:07 bareos-dir JobId 8141: There are no more Jobs associated with Volume "aicons30d-bsd02-1743". Marking it purged.
08-Jan 08:07 bareos-dir JobId 8141: All records pruned from Volume "aicons30d-bsd02-1743"; marking it
<<<

"aicons30d-bsd02-1743" with JobId 8094 still contains the backup from host1 and destroy them with this operation!

IN CONSEQUENCE the next virtual full from host1 will crashed and the backup is lost:

>>>
08-Jan 09:22 bareos-dir JobId 8142: Start Virtual Backup JobId 8142, Job=host1.2024-01-08_08.00.02_30
08-Jan 09:22 bareos-dir JobId 8142: Error: JobId 8094 is not present in the catalog
08-Jan 09:22 bareos-dir JobId 8142: Fatal error: Jobs missing from catalog. Cannot continue.
08-Jan 09:22 bareos-dir JobId 8142: Replicating deleted files from jobids 8094,6676 to jobid 8142
08-Jan 09:22 bareos-dir JobId 8142: Error: Bareos bareos-dir 23.0.1~pre7.606b211eb (19Dec23):
<<<

Is this a configuration failure or a bareos bug?

Thank you very much!

Bye Ralf
TagsNo tags attached.

Activities

bruno-at-bareos

bruno-at-bareos

2024-01-08 16:38

manager   ~0005672

Bareos is just doing what your configuration allow: Volume expired to early compared to your needs.

Volume 1743
Job being on it is tagged with the oldest job timestamp (not the rewrite of 7 January)
  Start time: 08-Dez-2023 04:00:13
  End time: 08-Dez-2023 04:00:34

So after 30 days it expires.

As mentioned into the documentation, it is highly recommended that pool retention is far over maximum virtual full. so only consolidation will expire jobid.
raschu

raschu

2024-01-08 17:12

reporter   ~0005674

Hi Bruno, thanks.

I assumed that the last written date would be checked :-(.


Bye Ralf
bruno-at-bareos

bruno-at-bareos

2024-01-08 17:22

manager   ~0005675

There's no need to assume, just read and apply documentation ;-)

At the end of this section there's a prominent WARNING
https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#storages-and-pools
bruno-at-bareos

bruno-at-bareos

2024-01-08 17:23

manager   ~0005676

Bareos works as documented. Pay attention to warning section, we place them for a good reason ;-)

Issue History

Date Modified Username Field Change
2024-01-08 15:59 raschu New Issue
2024-01-08 16:38 bruno-at-bareos Note Added: 0005672
2024-01-08 17:12 raschu Note Added: 0005674
2024-01-08 17:22 bruno-at-bareos Note Added: 0005675
2024-01-08 17:23 bruno-at-bareos Assigned To => bruno-at-bareos
2024-01-08 17:23 bruno-at-bareos Status new => closed
2024-01-08 17:23 bruno-at-bareos Resolution open => no change required
2024-01-08 17:23 bruno-at-bareos Note Added: 0005676