View Issue Details

IDProjectCategoryView StatusLast Update
0001161bareos-coredirectorpublic2023-03-22 10:06
Reporterembareossed Assigned Tobruno-at-bareos  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
PlatformLinuxOSDebianOS Version9
Product Version18.2.5 
Summary0001161: JobId 0: Prior failed job found in catalog. Upgrading to Differential.
DescriptionDifferential backups always produce this error. It is confusing on two counts: 1) Scheduled differential backups should not generate this message since the backup is scheduled to be differential, and 2) Message does not indicate which backup is being (falsely) "upgraded." Since I have 5 differential jobs every week (except once a month when full backups are run), there are exactly 5 of these messages, so apparently such message is generated for each differential job.
Steps To ReproduceSchedule differential jobs. One such message will be generated for each job.

NB: My backup schedule is identical for all backup jobs. I have it set to perform monthly full backups, then differentials once a week, and incrementals all other days.
Additional InformationI am simply following the model offered in the documentation, so I assumed that this was a valid, workable approach.

These messages show up in separate emails from the backup emails.
TagsNo tags attached.

Activities

bruno-at-bareos

bruno-at-bareos

2021-12-21 15:30

manager   ~0004399

Are you still using those components ?
Debian 9 is out of maintenance, and Bareos 18 will get out of support, soon with the release of 21 ?

To help debugging it would be nice to add at least the joblog and the job definition
in bconsole
show job=YOURJOB

list joblog jobid=9999

Thanks
embareossed

embareossed

2021-12-24 17:40

reporter   ~0004413

I am migrating to devuan chimaera (debian buster) with bareos 20. I am not seeing this message in this version. Apparently it has been fixed, so it would be OK to close it.
embareossed

embareossed

2021-12-26 22:37

reporter   ~0004417

I spoke too soon. This morning, I was greeted to these messages again from jobs run on bareos 20. However, I will be upgrading my bareos 20 director (and storage daemon) -- ASAP -- to bareos 21. So it remains to be seen if this problem is corrected in bareos 21.

As to job definition/job log, that is precisely the problem: I have many jobs, and not all of them display this behavior. I get these emails for some subset of the entire set of jobs on the schedule. I wish I were getting the jobid=9999 (in your example) since this would give me more information. Instead, it gives me "jobid=0" rather than an actual job number.

If this remains an issue in bareos 21, I will dig deeper to try to figure this out, maybe trying to run various backup jobs in bareos 21 repeatedly until such an error email comes.
embareossed

embareossed

2021-12-26 22:40

reporter   ~0004418

One note aside. I see that sometimes a connection failure message also appears in the email. In recent cases, a job is failing because my wireless is down so I can't successfully backup my (wireless) laptop. So it could be related. If so, it would be better if the "prior failed job" message came with the associated jobid.
embareossed

embareossed

2021-12-27 02:30

reporter   ~0004419

Upgraded to bareos 21. More bad news on this issue: It still sends these emails. I think I can tell which job or jobs are causing these to be generated, but that's only because I just reset my bareos 21 catalog. This is not a deterministic way of finding the cause reliably, of course.

Also note that these are intermittent and somewhat random-seeming. Some days I will have successful runs of all of my backups, yet a few of them will spout these messages. And, as I said in my last update here, occasionally I also see other messages in the same emails about connectivity or other issues.

As far as finding a solution to this... may I suggest the following? In the source code, place a trap for jobid == 0. Since that should never happen (right?), I'd think an assertion at that point would be appropriate. I realize there might be several suspect places where the message is generated, but if you are using a common routine throughout the source for the email messaging, it shouldn't be hard to implement a stack traceback to find the specific caller.

As for me, I will see what I can do to run down the source of this problem, including combing over my job defs looking for problems. I was hoping that the director's --test option would detect anything that doesn't look right.
bruno-at-bareos

bruno-at-bareos

2021-12-28 09:50

manager   ~0004429

The information about the failing jobs would still be really appreciate.

One probable cause could be a differential looking for a full jobs on which is has to be based, and that full job get pruned too earlier.
embareossed

embareossed

2021-12-28 18:27

reporter   ~0004431

The summary line is the ENTIRE contents of the email message. There is no further info, per se. No indication as to which job generated the message, etc. And there are no failing jobs necessarily. Most days, all of my jobs run to completion successfully and yet I still see these messages.
bruno-at-bareos

bruno-at-bareos

2022-01-04 14:28

manager   ~0004450

Hello, I'm sorry, but I feel completely confused by your previous notes.

If I return to the subject of this report
0001161: JobId 0: Prior failed job found in catalog. Upgrading to Differential.
            Differential backups always produce this error.

I would like to have a joblog extract like the following

```
list joblog jobid=8100
2021-12-07 01:30:00 qt-kt-dir JobId 8100: No prior Full backup Job record found.
2021-12-07 01:30:00 qt-kt-dir JobId 8100: No prior or suitable Full backup found in catalog. Doing FULL backup.
2021-12-07 01:30:00 qt-kt-dir JobId 8100: Start Backup JobId 8100, Job=ai_share.2021-12-07_01.30.00_07
```
When you are telling you don't know which job is giving the message, I can only show you that there's always a jobid on the line so you can always refer to which job it is.
Maybe with your extract we will be able to move forward.
embareossed

embareossed

2022-01-05 02:02

reporter   ~0004454

Last edited: 2022-01-05 02:03

Based on an email whose contents are only

JobId 0: Prior failed job found in catalog. Upgrading to Differential.

Please tell me how to proceed to give you the extract information you need. I am more than happy to do this if you will explain how to.

embareossed

embareossed

2022-01-06 08:22

reporter   ~0004457

Here is an example of the email I receive (this is typical).
daemon_message.eml (562 bytes)
embareossed

embareossed

2022-01-06 08:23

reporter   ~0004458

*list joblog jobid=0
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
jobid not found in db, access to job or client denied by ACL, or client not found in db
*
embareossed

embareossed

2022-01-06 08:26

reporter   ~0004459

I am not using ACLs, just to be clear. And I am sure that jobid 0 is not found in the database since it is not a legit job number.

As far as which client it is crowing about, there is no indication of it in the email (see example email above).
bruno-at-bareos

bruno-at-bareos

2022-01-10 13:29

manager   ~0004462

jobid 0 doesn't exist for obvious reason.

Having a look at source code, I realized you are not getting the standard upgrade message when no prior job exist. You got the return when failed jobs exists.

Globally the code run the following query and got a leat one element (there's a limit 1 that I remove)

SELECT Level FROM Job
WHERE JobStatus NOT IN ('T','W')
AND Type='D'
AND Level IN ('F','D')
AND Name='%s'
AND StartTime>'%s'
ORDER BY StartTime DESC

Maybe you will be able to run that query with replacing (job by job) the %s for name, and starttime ?
I've not enough element here to give you more precise sql. but the return should show records that are wired.
I'm interested to have this report here.

Thanks.
embareossed

embareossed

2022-01-10 17:58

reporter   ~0004463

Please tell me exactly where to find the name of the job. I have about a dozen jobs that are started at the same time by way of a schedule object.
bruno-at-bareos

bruno-at-bareos

2022-01-11 09:30

manager   ~0004464

At the beginning of the report you were talking about 5 differential, then use those name one by one.
embareossed

embareossed

2022-01-11 10:19

reporter   ~0004465

Last edited: 2022-01-11 10:22

The first order of business is to figure out which jobs (by jobname) is generating these messages. That is what I have been unable to do.

Please take another look at the email message I attached (https://bugs.bareos.org/view.php?id=1161#c4457), as an example. Tell me, based on that little information, how to proceed. I will be happy to explore whatever files and information is needed. I simply do not know where this information would be--I am not a bareos dev so I am not familiar with the code base to any great extent.

bruno-at-bareos

bruno-at-bareos

2022-01-11 10:25

manager   ~0004466

Sorry, you're confusing me again, I can't at your place tell what's the name of the job you are running.
>> Since I have 5 differential jobs every week
I had understood you know which job are running, was, or will. with a "status dir days=7" you will have an overview of the next week, then you should be able to pick the name of the next Differential.
embareossed

embareossed

2022-01-11 10:55

reporter   ~0004467

Last edited: 2022-01-11 10:58

All of the jobs are run together in the schedule. They run a full back first sunday of the month, differentials every other sunday, and incrementals daily.

BTW, since I opened this ticket, I have added more jobs. I run about a dozen every night.

Also note that I did reveal this in the "Steps to Reproduce," above (please see NB).

bruno-at-bareos

bruno-at-bareos

2022-01-11 16:49

manager   ~0004468

Hello I think you didn't understood (take the time to) what I was asking. I was doing this on my free time, which become sparse actually at least for the next two weeks.
Your step to reproduce give only working solution here, that's why we were trying to narrow down your problem.
Maybe you don't want to share the information like the job name publicly or whatever, but I try hardly to guide you with the right command to execute, I would be really pleased to see the result of those and move forward.

Your last comment let me thinking I'm just wasting my precious contribution free time.
embareossed

embareossed

2022-01-11 23:26

reporter   ~0004472

You still have not given me instructions on how to get the name of the job. There are numerous jobs (about a dozen, as I said) that kick off by the schedule. So it can be any of them. The emails do not indicate the name of the job either; again, see my sample email message, above (have you looked at it?).

I understand if you don't want to work on this, but could you pass this issue to someone else who might be able to help? Do not close it because it is an open issue. Thank you.
bruno-at-bareos

bruno-at-bareos

2023-03-21 17:23

manager   ~0004918

Could it be reproducible with bareos 22 and supported Debian (like 11) ?
embareossed

embareossed

2023-03-21 23:07

reporter   ~0004924

I have no idea because I do not have debian here.

I am surprised you are following up on this issue given you previously indicated you did not want to pursue this further.

However, I can tell you that I am no longer receiving these bizarre messages. I am currently running bareos 21.0.0, and this is on Devuan Chimaera (~ debian 11).
bruno-at-bareos

bruno-at-bareos

2023-03-22 10:06

manager   ~0004925

Fixed with recent code.

Issue History

Date Modified Username Field Change
2019-12-15 22:43 embareossed New Issue
2021-12-21 15:30 bruno-at-bareos Note Added: 0004399
2021-12-24 17:40 embareossed Note Added: 0004413
2021-12-26 22:37 embareossed Note Added: 0004417
2021-12-26 22:40 embareossed Note Added: 0004418
2021-12-27 02:30 embareossed Note Added: 0004419
2021-12-28 09:50 bruno-at-bareos Note Added: 0004429
2021-12-28 18:27 embareossed Note Added: 0004431
2022-01-04 14:28 bruno-at-bareos Note Added: 0004450
2022-01-05 02:02 embareossed Note Added: 0004454
2022-01-05 02:03 embareossed Note Edited: 0004454
2022-01-06 08:22 embareossed File Added: daemon_message.eml
2022-01-06 08:22 embareossed Note Added: 0004457
2022-01-06 08:23 embareossed Note Added: 0004458
2022-01-06 08:26 embareossed Note Added: 0004459
2022-01-10 13:29 bruno-at-bareos Note Added: 0004462
2022-01-10 17:58 embareossed Note Added: 0004463
2022-01-11 09:30 bruno-at-bareos Note Added: 0004464
2022-01-11 10:19 embareossed Note Added: 0004465
2022-01-11 10:21 embareossed Note Edited: 0004465
2022-01-11 10:22 embareossed Note Edited: 0004465
2022-01-11 10:25 bruno-at-bareos Note Added: 0004466
2022-01-11 10:55 embareossed Note Added: 0004467
2022-01-11 10:58 embareossed Note Edited: 0004467
2022-01-11 16:49 bruno-at-bareos Note Added: 0004468
2022-01-11 23:26 embareossed Note Added: 0004472
2023-03-21 17:23 bruno-at-bareos Note Added: 0004918
2023-03-21 23:07 embareossed Note Added: 0004924
2023-03-22 10:06 bruno-at-bareos Assigned To => bruno-at-bareos
2023-03-22 10:06 bruno-at-bareos Status new => closed
2023-03-22 10:06 bruno-at-bareos Resolution open => fixed
2023-03-22 10:06 bruno-at-bareos Note Added: 0004925