View Issue Details

IDProjectCategoryView StatusLast Update
0001115bareos-core[All Projects] directorpublic2019-11-26 12:34
ReporterbozoniusAssigned To 
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
PlatformLinuxOSUbuntuOS Version14.04
Product Version16.2.4 
Fixed in Version 
Summary0001115: changing onefs value in fileset should force a full backup
DescriptionThe documentation in the FileSet Resource section of bareos documents (as of 21 September 2019) states, in the second paragraph:

"Any change to the list of the included files will cause Bareos to automatically create a new FileSet (defined by the name and an MD5 checksum of the Include/Exclude contents). Each time a new FileSet is created, Bareos will ensure that the next backup is always a Full save."

This says very specifically that a change to the list of included files will force a full backup; merely changing an option value such as onefs does not do so.

I mounted a filesystem in the area to be backed up, changing (actually, adding) "onefs = no" to the fileset. I tried several times to get the files in the mounted file system to be backed up, with no luck. After CAREFULLY reading your documentation, I realized that (probably) a change to the disposition of the onefs option does not apply to the quoted paragraph.

In my situation at least, mounting the filesystem within the fileset definition's area introduced new files to be backed up. I am not sure how this might affect other users, nor do I know if any other options should force a full backup. I am cautious not to conjecture too much on an ideal solution, as I am not aware of how every last installation is configured or what results users expect. (I certainly expected the next backup to pull in the files on the mounted file system.)
Steps To ReproduceMount a file system containing some files in an area defined to be backed up in a fileset.

Change or add the option "onefs = no" to the fileset (where the options section did not specify it previously).

Run the same backup(s) which use that fileset, ensuring they are set to be run next as incremental.

Check to see if the files on the mounted file system are included in the backup.
TagsNo tags attached.
bareos-master: impactyes
bareos-master: action
bareos-19.2: impactyes
bareos-19.2: action
bareos-18.2: impactyes
bareos-18.2: actionnone
bareos-17.2: impactyes
bareos-17.2: actionnone
bareos-16.2: impactyes
bareos-16.2: actionnone
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action

Relationships

related to 0001143 confirmed Update a Job to Accurate instead of Full backup on FileSet changes. 

Activities

joergs

joergs

2019-11-11 19:16

administrator   ~0003620

Well, this should not happen.

Can you please verify, that your fileset changes are properly detected?
Use the following bconsole command to get all known versions of the fileset:

.sql query="select * from fileset where fileset='<YOURFILESETNAME>'"
bozonius

bozonius

2019-11-11 20:43

reporter   ~0003624

Last edited: 2019-11-11 20:57

View 4 revisions

I found the fileset in question. It was updated on 9/21/2019 to include another path and OneFS set to no. The added path was in the excluded section, which might explain why there was no full backup?

One other note: Despite the database showing the additional path in the exclude section in the most recent (9/21/2019) row of FileSet for that fileset, the configuration file for the fileset now has no reference to the excluded path. I'm not sure that this is harmful, but I thought I should mention it.

joergs

joergs

2019-11-12 15:36

administrator   ~0003630

I'm not sure, if I understood this correctly.

The fileset entry is written to the database, when a job did run which uses the fileset. Has there been a job using the new fileset?
Please also compare it to the output of the bconsole command "show fileset='<YOURFILESETNAME>", just in case, the new configuration is not loaded for any reason.
joergs

joergs

2019-11-12 15:46

administrator   ~0003631

Forget about my last question.
I verified that the behavior you describe still exists in the current Bareos master branch.

A change to the file list do upgrade the backup level to Full, a change in the options does not upgrade to a Full backup.
embareossed

embareossed

2019-11-12 20:45

reporter   ~0003632

[I just saw your last post, but I'll include this anyway, for reference. You may delete this note if desired.]

The last row for the fileset in question looks like:

| 30 | SystemConfig | B9/TG7tfvx+q7++Yw/th+C | 2019-09-21 03:00:01 | FileSet {
  Name = "SystemConfig"
  Include {
    Options {
      OneFS = No
      Compression = GZIP9
    }
    File = "/etc"
    File = "/root"
    File = "/var"
  }
  Exclude {
    File = "/var/tmp"
    File = "/var/run"
    File = "/var/lock"
    File = "/var/cache"
    File = "/root/tmp"
    File = "/var/lib/nothing"
  }
}

This resource record is part of 5 backups every night and has been since before this update was made to the config file:

FileSet {
  Name = "SystemConfig"
  Include {
    Options {
      One FS = no
      Compression = GZIP9
    }
    File = "/etc"
    File = "/root"
    File = "/var"
  }
  Exclude {
    File = "/var/tmp"
    File = "/var/run"
    File = "/var/lock"
    File = "/var/cache"
    File = "/root/tmp"
  }
}

The timestamp on this file is:

-rw-rw-r-- 1 bareos bareos 302 Sep 21 03:16 SystemConfig.conf

Also, I DO backup the /etc/bareos tree every night. I only keep about a month's worth of backups. Dates and times are sync'd to an ntp server.
embareossed

embareossed

2019-11-12 20:47

reporter   ~0003633

So is this the intended behavior by design and not a bug? Or is it a defect after all?
embareossed

embareossed

2019-11-12 21:19

reporter   ~0003634

I'm now realizing that this issue/bug might be mis-labeled. I think the real issue here is that the Exclude { } was modified. As you can see, the database thinks the files to be excluded would also mean /var/lib/nothing is to be excluded. But I removed that one exclusion, but the database continues to think /var/lib/nothing should be excluded. (In this case, no problem arises because obviously this was only a test case.) Once the newer definition of the fileset was used once (and it has been, many times since that change) I would expect the database to now have an updated definition as well, but it doesn't.

Excluded files probably need to be considered in backup calculations for the same reason that the included ones are. Here, I gave an example of where possibly new files might be added to subsequent backups, but the database might ignore the change and continue to exclude files and directories under /var/lib/nothing. I haven't tested this, but it does seem inconsistent at very least.

As to the change to OneFS, that may have been the change I made, probably earlier, that precipitated filing the current issue. Being this was months ago, I have forgotten. Sorry for any confusion.
joergs

joergs

2019-11-26 12:34

administrator   ~0003640

I discussed this issue with the Bareos Developer team. The conclusion is, that it works a designed. However, most people will admit, that a redesign might be a good idea.

In my tests I came to the conclusion, that changes to the "File" directives (in Include as well as in Exclude) will upgrade the job to Full, while all other options to not result in an upgrade.

I tried to clarify this fact in a change to the documentation, see https://github.com/bareos/bareos/pull/341

There have long been a discussion, if an upgrade to Accurate wouldn't be preferable to an Upgrade to Full. I created to Feature Request for this, together with hints about the current limitation, see https://bugs.bareos.org/view.php?id=1143

With this, I'm closing the bug report and hope future work in https://bugs.bareos.org/view.php?id=1143 will improve that behavior.

Issue History

Date Modified Username Field Change
2019-09-21 12:58 bozonius New Issue
2019-11-11 19:16 joergs Status new => feedback
2019-11-11 19:16 joergs Note Added: 0003620
2019-11-11 20:43 bozonius Note Added: 0003624
2019-11-11 20:43 bozonius Status feedback => new
2019-11-11 20:51 bozonius Note Edited: 0003624 View Revisions
2019-11-11 20:56 bozonius Note Edited: 0003624 View Revisions
2019-11-11 20:57 bozonius Note Edited: 0003624 View Revisions
2019-11-12 15:36 joergs Assigned To => joergs
2019-11-12 15:36 joergs Status new => feedback
2019-11-12 15:36 joergs Note Added: 0003630
2019-11-12 15:46 joergs Status feedback => confirmed
2019-11-12 15:46 joergs Note Added: 0003631
2019-11-12 20:45 embareossed Note Added: 0003632
2019-11-12 20:47 embareossed Note Added: 0003633
2019-11-12 21:19 embareossed Note Added: 0003634
2019-11-26 12:10 joergs Relationship added related to 0001143
2019-11-26 12:34 joergs Assigned To joergs =>
2019-11-26 12:34 joergs Status confirmed => closed
2019-11-26 12:34 joergs Resolution open => fixed
2019-11-26 12:34 joergs bareos-master: impact => yes
2019-11-26 12:34 joergs bareos-19.2: impact => yes
2019-11-26 12:34 joergs bareos-18.2: impact => yes
2019-11-26 12:34 joergs bareos-18.2: action => none
2019-11-26 12:34 joergs bareos-17.2: impact => yes
2019-11-26 12:34 joergs bareos-17.2: action => none
2019-11-26 12:34 joergs bareos-16.2: impact => yes
2019-11-26 12:34 joergs bareos-16.2: action => none
2019-11-26 12:34 joergs Note Added: 0003640