View Issue Details

IDProjectCategoryView StatusLast Update
0001194bareos-core[All Projects] file daemonpublic2020-06-08 10:46
ReporterIntAssigned Toarogge 
PriorityhighSeveritymajorReproducibilityalways
Status assignedResolutionreopened 
Platformx86OSWindowsOS Version2016
Product Version19.2.7 
Fixed in Version 
Summary0001194: Incremental backup does backup all files instead of changed files
DescriptionOn 30.1.2020 the incremental backup on my Windows Server 2016 (64bit) suddenly did not backup the changed files anymore but did backup up all files instead. This happened ever since except one time on 4.2.2020 where it did a correct incremental backup.
This change of behaviour does not correlate with Windows updates - the last time I installed Windows updates on the server was on 17.1.2020.
Also the Bareos server was not changed.
I am using the Accurate option for my backups.

in this bug https://bugs.bareos.org/view.php?id=907
I found the hint to use only MD5 hash for the accurate option so I tried this and changed my fileset configuration to "accurate = 5"
but it did not change the behaviour. The incremental backup is still backing up all files.

Backups of clients with Windows 10 still work correctly, so I assume that issues lies with the file daemon on Windows Server 2016.

Any ideas what this could be caused by?
Or any hint where to debug?
TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-19.2: impact
bareos-19.2: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action

Relationships

related to 0000907 acknowledged Incremental Job backups same files 

Activities

arogge

arogge

2020-02-13 09:37

developer   ~0003805

First of all: please don't run 17.2 if you can help it.

As you have accurate enabled, it could be that the accurate flags are not transferred to the client correctly.

- can you successfully open the restore browser in bconsole using the restore command and then selecting option 5, and the client in question?
- how many files are "all files"?
Int

Int

2020-02-13 12:05

reporter   ~0003809

Thank you for the quick feedback!

Upgrading from 17.2 is on my todo list but I didn't have the time to tackle that so far.
I can restore successfully for this client with bconsole.
All files are 8,786,819 files at the moment. The number of files didn't increase drastically, it was always about this amount for this client.

It would be great if 17.2 would work again, this would give me time to plan and execute the upgrade to 19.2 in a relaxed manner.
If you think an upgrade to 19.2 is the only solution for this problem I will have to reorganize my tasks and give this highest priority.
Int

Int

2020-02-18 09:08

reporter   ~0003821

Since I didn't get any further feedback about how to proceed, I decided myself and worked over the weekend to update Bareos to the latest version 19.2.6

So far (for the last two nights) the incremental and differential backups work correctly again, so the update seems to have solved the issue.

But I still don't understand why Bareos changed it's behaviour all of a sudden in the first place...
arogge

arogge

2020-02-18 10:32

developer   ~0003822

I don't understand it either - but I'm also not eager to dig into old code to find an issue that has been fixed in the meantime.
While 19.2 has its own issues (and we're working on that) it is probably a much better choice than 17.2.

Is your problem fixed now? Can I close the issue?
Int

Int

2020-02-18 10:43

reporter   ~0003823

Let me monitor the backup behaviour until end of the week just to be sure ...
I will give you feedback on Friday.

Thanks!
arogge

arogge

2020-02-18 15:42

developer   ~0003829

As the problem doesn't seem to occur in 19.2 anymore, I'll close the issue.
You can just reopen it if the issue resurfaces. Thank you!
Int

Int

2020-06-04 10:15

reporter   ~0003998

I have to reopen this issue.
After upgrading to version 19.2.6 and then to version 19.2.7 (to fix issue 1192) a month ago this issue did not happen again until two days ago.
Now I see the same behaviour with version 19.2.7 as described before - the Incremental backup on my Windows Server 2016 (64bit) does backup all files instead of changed files.
arogge

arogge

2020-06-04 10:55

developer   ~0003999

There are two things that I could imagine happened:
1. the accurate filelist generated by the director and sent to the fd was truncated or empty
2. the previous incremental marked all files as deleted

As 2.) is easier to check, I'd like to rule that out first. Run a restore command to restore up to the latest incremental using "restore client before specified time" and take a look at the restore filetree that is generated. Do you see all files that you'd expect, or are (all) files missing?
If all files are there, then you're probably having an issue related to 1.), if files are missing then probably one of your previous incrementals went wrong.

Thank you!
Int

Int

2020-06-05 11:21

reporter   ~0004002

I tested your option 2 and got this error from bconsole:
 Query failed: DROP TABLE temp1 : ERR=ERROR: table "temp1" does not exist
 For one or more of the JobIds selected, no files were found, so file selection is not possible.
 Most likely your retention policy pruned the files.

This pointed me in the right direction. The partition of the Postgres database was low on disk space and therefore the table "temp1" could not be created.
After increasing the disk space for the Postgres database the incremental backup is working correctly again.

thank you very much!

Maybe the error logging of Bareos could be improved to log an error to the messages of the backup job when the creation of the accurate filelist fails?
arogge

arogge

2020-06-08 10:46

developer   ~0004003

I'm glad you could fix the issue.

I'm not sure there is a simple way to detect that situation. After all, if the database hits an anomalous condition (like a filled up filesystem) it can fails in any way it sees fit.
But I'll see if we can do something to improve the situation.

Issue History

Date Modified Username Field Change
2020-02-13 08:53 Int New Issue
2020-02-13 09:37 arogge Assigned To => arogge
2020-02-13 09:37 arogge Status new => feedback
2020-02-13 09:37 arogge Note Added: 0003805
2020-02-13 09:37 arogge Relationship added related to 0000907
2020-02-13 12:05 Int Note Added: 0003809
2020-02-13 12:05 Int Status feedback => assigned
2020-02-18 09:08 Int Note Added: 0003821
2020-02-18 10:32 arogge Note Added: 0003822
2020-02-18 10:43 Int Note Added: 0003823
2020-02-18 15:42 arogge Status assigned => resolved
2020-02-18 15:42 arogge Resolution open => unable to reproduce
2020-02-18 15:42 arogge Note Added: 0003829
2020-06-04 10:15 Int Status resolved => new
2020-06-04 10:15 Int Resolution unable to reproduce => reopened
2020-06-04 10:15 Int Note Added: 0003998
2020-06-04 10:20 arogge Product Version 17.2.4 => 19.2.7
2020-06-04 10:55 arogge Note Added: 0003999
2020-06-04 10:56 arogge Status new => feedback
2020-06-05 11:21 Int Note Added: 0004002
2020-06-05 11:21 Int Status feedback => assigned
2020-06-08 10:46 arogge Note Added: 0004003