View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0001161||bareos-core||director||public||2019-12-15 22:43||2022-01-11 23:26|
|Summary||0001161: JobId 0: Prior failed job found in catalog. Upgrading to Differential.|
|Description||Differential backups always produce this error. It is confusing on two counts: 1) Scheduled differential backups should not generate this message since the backup is scheduled to be differential, and 2) Message does not indicate which backup is being (falsely) "upgraded." Since I have 5 differential jobs every week (except once a month when full backups are run), there are exactly 5 of these messages, so apparently such message is generated for each differential job.|
|Steps To Reproduce||Schedule differential jobs. One such message will be generated for each job.|
NB: My backup schedule is identical for all backup jobs. I have it set to perform monthly full backups, then differentials once a week, and incrementals all other days.
|Additional Information||I am simply following the model offered in the documentation, so I assumed that this was a valid, workable approach.|
These messages show up in separate emails from the backup emails.
|Tags||No tags attached.|
Are you still using those components ?
Debian 9 is out of maintenance, and Bareos 18 will get out of support, soon with the release of 21 ?
To help debugging it would be nice to add at least the joblog and the job definition
list joblog jobid=9999
|I am migrating to devuan chimaera (debian buster) with bareos 20. I am not seeing this message in this version. Apparently it has been fixed, so it would be OK to close it.|
I spoke too soon. This morning, I was greeted to these messages again from jobs run on bareos 20. However, I will be upgrading my bareos 20 director (and storage daemon) -- ASAP -- to bareos 21. So it remains to be seen if this problem is corrected in bareos 21.
As to job definition/job log, that is precisely the problem: I have many jobs, and not all of them display this behavior. I get these emails for some subset of the entire set of jobs on the schedule. I wish I were getting the jobid=9999 (in your example) since this would give me more information. Instead, it gives me "jobid=0" rather than an actual job number.
If this remains an issue in bareos 21, I will dig deeper to try to figure this out, maybe trying to run various backup jobs in bareos 21 repeatedly until such an error email comes.
|One note aside. I see that sometimes a connection failure message also appears in the email. In recent cases, a job is failing because my wireless is down so I can't successfully backup my (wireless) laptop. So it could be related. If so, it would be better if the "prior failed job" message came with the associated jobid.|
Upgraded to bareos 21. More bad news on this issue: It still sends these emails. I think I can tell which job or jobs are causing these to be generated, but that's only because I just reset my bareos 21 catalog. This is not a deterministic way of finding the cause reliably, of course.
Also note that these are intermittent and somewhat random-seeming. Some days I will have successful runs of all of my backups, yet a few of them will spout these messages. And, as I said in my last update here, occasionally I also see other messages in the same emails about connectivity or other issues.
As far as finding a solution to this... may I suggest the following? In the source code, place a trap for jobid == 0. Since that should never happen (right?), I'd think an assertion at that point would be appropriate. I realize there might be several suspect places where the message is generated, but if you are using a common routine throughout the source for the email messaging, it shouldn't be hard to implement a stack traceback to find the specific caller.
As for me, I will see what I can do to run down the source of this problem, including combing over my job defs looking for problems. I was hoping that the director's --test option would detect anything that doesn't look right.
The information about the failing jobs would still be really appreciate.
One probable cause could be a differential looking for a full jobs on which is has to be based, and that full job get pruned too earlier.
|The summary line is the ENTIRE contents of the email message. There is no further info, per se. No indication as to which job generated the message, etc. And there are no failing jobs necessarily. Most days, all of my jobs run to completion successfully and yet I still see these messages.|
Hello, I'm sorry, but I feel completely confused by your previous notes.
If I return to the subject of this report
0001161: JobId 0: Prior failed job found in catalog. Upgrading to Differential.
Differential backups always produce this error.
I would like to have a joblog extract like the following
list joblog jobid=8100
2021-12-07 01:30:00 qt-kt-dir JobId 8100: No prior Full backup Job record found.
2021-12-07 01:30:00 qt-kt-dir JobId 8100: No prior or suitable Full backup found in catalog. Doing FULL backup.
2021-12-07 01:30:00 qt-kt-dir JobId 8100: Start Backup JobId 8100, Job=ai_share.2021-12-07_01.30.00_07
When you are telling you don't know which job is giving the message, I can only show you that there's always a jobid on the line so you can always refer to which job it is.
Maybe with your extract we will be able to move forward.
Based on an email whose contents are only
JobId 0: Prior failed job found in catalog. Upgrading to Differential.
Please tell me how to proceed to give you the extract information you need. I am more than happy to do this if you will explain how to.
Here is an example of the email I receive (this is typical).
daemon_message.eml (562 bytes)
*list joblog jobid=0
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
jobid not found in db, access to job or client denied by ACL, or client not found in db
I am not using ACLs, just to be clear. And I am sure that jobid 0 is not found in the database since it is not a legit job number.
As far as which client it is crowing about, there is no indication of it in the email (see example email above).
jobid 0 doesn't exist for obvious reason.
Having a look at source code, I realized you are not getting the standard upgrade message when no prior job exist. You got the return when failed jobs exists.
Globally the code run the following query and got a leat one element (there's a limit 1 that I remove)
SELECT Level FROM Job
WHERE JobStatus NOT IN ('T','W')
AND Level IN ('F','D')
ORDER BY StartTime DESC
Maybe you will be able to run that query with replacing (job by job) the %s for name, and starttime ?
I've not enough element here to give you more precise sql. but the return should show records that are wired.
I'm interested to have this report here.
|Please tell me exactly where to find the name of the job. I have about a dozen jobs that are started at the same time by way of a schedule object.|
|At the beginning of the report you were talking about 5 differential, then use those name one by one.|
The first order of business is to figure out which jobs (by jobname) is generating these messages. That is what I have been unable to do.
Please take another look at the email message I attached (https://bugs.bareos.org/view.php?id=1161#c4457), as an example. Tell me, based on that little information, how to proceed. I will be happy to explore whatever files and information is needed. I simply do not know where this information would be--I am not a bareos dev so I am not familiar with the code base to any great extent.
Sorry, you're confusing me again, I can't at your place tell what's the name of the job you are running.
>> Since I have 5 differential jobs every week
I had understood you know which job are running, was, or will. with a "status dir days=7" you will have an overview of the next week, then you should be able to pick the name of the next Differential.
All of the jobs are run together in the schedule. They run a full back first sunday of the month, differentials every other sunday, and incrementals daily.
BTW, since I opened this ticket, I have added more jobs. I run about a dozen every night.
Also note that I did reveal this in the "Steps to Reproduce," above (please see NB).
Hello I think you didn't understood (take the time to) what I was asking. I was doing this on my free time, which become sparse actually at least for the next two weeks.
Your step to reproduce give only working solution here, that's why we were trying to narrow down your problem.
Maybe you don't want to share the information like the job name publicly or whatever, but I try hardly to guide you with the right command to execute, I would be really pleased to see the result of those and move forward.
Your last comment let me thinking I'm just wasting my precious contribution free time.
You still have not given me instructions on how to get the name of the job. There are numerous jobs (about a dozen, as I said) that kick off by the schedule. So it can be any of them. The emails do not indicate the name of the job either; again, see my sample email message, above (have you looked at it?).
I understand if you don't want to work on this, but could you pass this issue to someone else who might be able to help? Do not close it because it is an open issue. Thank you.
|2019-12-15 22:43||embareossed||New Issue|
|2021-12-21 15:30||bruno-at-bareos||Note Added: 0004399|
|2021-12-24 17:40||embareossed||Note Added: 0004413|
|2021-12-26 22:37||embareossed||Note Added: 0004417|
|2021-12-26 22:40||embareossed||Note Added: 0004418|
|2021-12-27 02:30||embareossed||Note Added: 0004419|
|2021-12-28 09:50||bruno-at-bareos||Note Added: 0004429|
|2021-12-28 18:27||embareossed||Note Added: 0004431|
|2022-01-04 14:28||bruno-at-bareos||Note Added: 0004450|
|2022-01-05 02:02||embareossed||Note Added: 0004454|
|2022-01-05 02:03||embareossed||Note Edited: 0004454|
|2022-01-06 08:22||embareossed||File Added: daemon_message.eml|
|2022-01-06 08:22||embareossed||Note Added: 0004457|
|2022-01-06 08:23||embareossed||Note Added: 0004458|
|2022-01-06 08:26||embareossed||Note Added: 0004459|
|2022-01-10 13:29||bruno-at-bareos||Note Added: 0004462|
|2022-01-10 17:58||embareossed||Note Added: 0004463|
|2022-01-11 09:30||bruno-at-bareos||Note Added: 0004464|
|2022-01-11 10:19||embareossed||Note Added: 0004465|
|2022-01-11 10:21||embareossed||Note Edited: 0004465|
|2022-01-11 10:22||embareossed||Note Edited: 0004465|
|2022-01-11 10:25||bruno-at-bareos||Note Added: 0004466|
|2022-01-11 10:55||embareossed||Note Added: 0004467|
|2022-01-11 10:58||embareossed||Note Edited: 0004467|
|2022-01-11 16:49||bruno-at-bareos||Note Added: 0004468|
|2022-01-11 23:26||embareossed||Note Added: 0004472|