View Issue Details

IDProjectCategoryView StatusLast Update
0001571bareos-coreGeneralpublic2023-12-07 10:34
Reporterhostedpower Assigned Tobruno-at-bareos  
PrioritylowSeverityfeatureReproducibilityalways
Status resolvedResolutionno change required 
Summary0001571: Differential backups for postgresql new plugin
DescriptionHi,


We used the former postgresql plugin from Bareos, and never had this message during backup:

Differential backups are not supported! Only Full and Incremental


This is effectively during a Differential backup. We never had such messages with the old plugin, so we're wondering why we have them now with the Bareos version 23 plugin

Also we wonder, is this hard to support. Isn't the differential simply adding all incrementals into one? We checked old postgresql plugin backups and we have the feeling this also actually just worked correctly.
TagsNo tags attached.

Activities

bruno-at-bareos

bruno-at-bareos

2023-11-23 13:35

manager   ~0005525

After checking how the old plugin behave and especially that to be able to do differential, you will need to have all wal archive present in the right order since last full and we can't warranty that especially as the plugin is not in charge to manage the archive repository.

There's no internal build-in of differential backup in PostgreSQL. The plugin being based on PITR Wal archiving methodology there's no real needs to have a differential level. Full + n number of incremental will allow the restoration to the desired point in time.

As it is specifically linked to the cluster, it also forbidden the use of feature like Virtual Full.
hostedpower

hostedpower

2023-11-23 13:47

reporter   ~0005526

So you're not simply adding incremental backups together? Since that should work up to the last full otherwise? Or do I oversee something here? :)
bruno-at-bareos

bruno-at-bareos

2023-11-23 14:22

manager   ~0005527

Well we provide enough effort I guess to explain what's going on with the plugin in the documentation
https://docs.bareos.org/master/TasksAndConcepts/Plugins.html#postgresql-plugin

And plugin backup are different from traditional file backup.
hostedpower

hostedpower

2023-11-23 16:28

reporter   ~0005528

Well if it wouldn't work, I have to believe you. I was just under the impression that the incremental backups just contain wall data. I checked on the old plugin, and there the differential backups are running fine. If I look into those backups, the differential backups just contain all the backups from the incremental ones. It's very easy, all files seem to be simply present.

So I must be overlooking something, because apparentely, having the exact same wal files as in the incremental backups wouldn't be sufficient? :)
image.png (99,946 bytes)   
image.png (99,946 bytes)   
bruno-at-bareos

bruno-at-bareos

2023-11-27 10:24

manager   ~0005532

I can't state anything from the screenshot, as the content greatly depend of how checkbox option are checked.
Maybe here I just see the result of full+incremental job merged :-)
hostedpower

hostedpower

2023-11-27 13:11

reporter   ~0005534

No, this was of a single job (unchecked the boxes to merge).

In any case, since it are just wal files that are kept with incremental, wouldn't a differential just work? Or am I missing something in the puzzle?

The old jobs (previous plugin with differential as well) seem to work when we test, but maybe we didn't test the right case or something.
bruno-at-bareos

bruno-at-bareos

2023-12-05 16:10

manager   ~0005570

Just a word about this case. The old plugin blindly believe in the fact that all archived wal are there since the full and backup those. If one is missing, the restore will not be possible while the backup would have status "OK".

We decided for the new one to not let user believe the backup was ok when the chance it is not grow as time pass, that's why we disable the Differential mode.
hostedpower

hostedpower

2023-12-05 16:28

reporter   ~0005571

Also for the incremental we need to make sure we keep enough wall files or otherwise we would have problems? I don't see the big difference differential versus incremental :)
bruno-at-bareos

bruno-at-bareos

2023-12-05 16:50

manager   ~0005572

Yes this is not a completely a wrong assumption ;-)
Discovered afterwards.
hostedpower

hostedpower

2023-12-05 17:32

reporter   ~0005573

Maybe it would be a good idea to allow differential again, warn in the notes we need long enough wall files, and longterm some safety mechanism would be even better hahaha :)

We do a differential each day , so for us it's really nice to keep the differentials working, it's a great advantage since we take 24 incrementals per day, it's crazy to keep these all for a week long.
bruno-at-bareos

bruno-at-bareos

2023-12-07 10:33

manager   ~0005576

So the final statement on our side (of course for the moment, with money or PR everything can be changed :-))
We will not publish the plugin with level D allowed.

But as you know it is quite simple to hack it. So for your own usage you may want to change the M_FATAL to something else line 217
https://github.com/bareos/bareos/blob/1e7d73e668609f39a7caf4422710dfb58f1c0cd1/core/src/plugins/filed/python/postgresql/bareos-fd-postgresql.py#L217

At your own risk.
bruno-at-bareos

bruno-at-bareos

2023-12-07 10:34

manager   ~0005577

Differential support will not be enable for this plugin.
Workaround for people knowing the risk they took, can be proposed in a PR and merged afterward if acceptable.

Issue History

Date Modified Username Field Change
2023-11-23 12:17 hostedpower New Issue
2023-11-23 13:35 bruno-at-bareos Note Added: 0005525
2023-11-23 13:47 hostedpower Note Added: 0005526
2023-11-23 14:22 bruno-at-bareos Note Added: 0005527
2023-11-23 16:28 hostedpower Note Added: 0005528
2023-11-23 16:28 hostedpower File Added: image.png
2023-11-27 10:24 bruno-at-bareos Note Added: 0005532
2023-11-27 13:11 hostedpower Note Added: 0005534
2023-12-05 16:10 bruno-at-bareos Note Added: 0005570
2023-12-05 16:28 hostedpower Note Added: 0005571
2023-12-05 16:50 bruno-at-bareos Note Added: 0005572
2023-12-05 17:32 hostedpower Note Added: 0005573
2023-12-07 10:33 bruno-at-bareos Note Added: 0005576
2023-12-07 10:34 bruno-at-bareos Assigned To => bruno-at-bareos
2023-12-07 10:34 bruno-at-bareos Status new => resolved
2023-12-07 10:34 bruno-at-bareos Resolution open => no change required
2023-12-07 10:34 bruno-at-bareos Note Added: 0005577