 |
Hello, We encounter the same error randomly with the same product version (and OS version). As for medicmomcilo, we can send additional information and logs. Thank you, Mickaël |
|
|
So what backends are you using? A single tape or disk? (or some other backend). |
|
|
Hello, We use disks: several pools containing multiple media stored on disk. |
|
|
Hi, We are encountering the same issue. We are running CentOS 7.2.1511. uname -a reports: Linux 3.10.0-327.22.2.el7.x86_64 0000001 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux We also, are only using disk backends. Cheers, Ian |
|
|
Hi, We have multiple disk devices defined and that is all. However, we don't use autochanger device. Kind regards, Momcilo. |
|
|
Hi, Just upgraded to 16.2.4. We are still getting the same error on random jobs. What more information do you require to get this fixed? Cheers, Ian |
|
|
Confirmed, as several environments are affected by this problem, |
|
|
> What more information do you require to get this fixed? Well, a setup to reproduce the problem would be perfect. Not being able to reproduce the problem makes it hard to fix it. Maybe a debug trace might bring some insight. You can enable the debug log by the bconsole command: setdebug level=100 trace=1 dir Also director configuration and some information about the environment would be helpful. |
|
|
Hi, I'm also getting these errors, please see my comments on the users group, maybe the info provided helps. https://groups.google.com/forum/?fromgroups=#!topic/bareos-users/Nbv-_fRWxaY I have debugging enabled for dir and sd, but it's kind of difficult to identify the related sections, because there's no timestamp in the tracefiles. I assume the number after filename:xxx- is the jobid, and I can tell you that, one of my jobs hich failed had 0 bytes, and even though failed. I cannot find the jobid at all in the sd trace, but there are some lines in the dir trace. let me try to upload the traces. thanks, Robert |
|
|
|
|
|
failed-job.log (3,258 bytes) |
|
|
The xxx after the filename is the line number in the source code. |
|
|
I see, and after that comes the job-id, right? would you please remove the trace file from this ticket? thank you. |
|
|
hi, I'm also getting messages like: 16-Dec 00:37 muc1pro-backup-1-sd JobId 8102: Job tw1kc0624.2016-12-16_00.13.09_12 is waiting. Cannot find any appendable volumes. Please use the "label" command to create a new Volume for: Storage: "DiskDeviceIncr2" (/data/bareos-storage/Incr) Pool: DiskDaily Media type: FileIncr although the pool limit is 600 Volumes and there are currently only 310 in use, and plenty of diskspace. If I try to run a single job, which requires new volumes to be labeled, it seem to work, but when the nightly schedule runs ( 5 in parallel) randomly throws either "No volume name given" or "Cannot find any appendable volumes" cheers, Robert |
|
|
Hi joergs, Our setup is as follows: Dedicated physical server is a Dell PowerEdge R710: - 2 x Intel Xeon X5650 Quad Core with Hyper Threading enabled, - 64GB RAM, - 6 x 900GB SAS in Raid5 (50GB /, 4.45TB /backup) - additional 6TB SAS via iSCSI (Dell Equallogic SANs) - 4 x 1GbE NICs connected to iSCSI VLAN in team config (round robin). - 4 x 1GbE NICs connected to Prod VLAN in team config (round robin). Running CentOS 7.2.1511 Was running Bareos 15.2 up until 1 month ago. Now running Bareos 16.2.4 - Upgraded and rejigged config files to match the new setup. Currently using MariaDB, but in the process of migrating to PGSQL. External storage node running in Google Cloud for Full & Diff migrations. Problem has persisted across versions. However this morning, I have just noted that all hourly incrementals were failing. I restarted the bareos-dir service. Now the subsequent jobs are successful. Attaching Bareos configuration as backup.tar.gz Once the jobs start failing again, I will enable debugging and rerun a job. Cheers, Ian |
|
|
|
|
|
hi all, I took a look into the trace file of the director, and seems the the problem is in the volume selection. I have defined a storage with 5 different devices pointing to the same folder. the storage with max 5 jobs and each device with max 1 job. looking at the trace file, here's the SELECT count for each job to find the volume: JobID : SELECT COUONT 10472 :10 10473 :121 10474 :189 10475 :291 10476 :189 used this command: for i in `seq 2 6` ; do echo -n 1047$i : ; grep -- "-1047$i" muc1pro-backup-1-dir.trace | egrep "SELECT" | wc -l ; done for the first job after the last select I see: Find_next_vol MediaId=3735 MediaId=3735 means volume=Incr-3735 in all the other 4 jobs, after some selects, the same volume is found but then comes a "unwanted_volumes=Incr-3735" and continues to run other selects. after the last select I see : Find_next_vol MediaId=0 , but there's no volume with mediaid = 0 , and most likely therefore it throws a no volume name given... here you can see the selects and find_next_vol records in the trace: grep -- "-10472" muc1pro-backup-1-dir.trace | egrep --color=always "SELECT|Find_next_vol|unwanted" | less -r replace 10472 in the grep with other jobid's from above. attaching the trace file. see the attached affected jobid's above. bareos-15.2.3-trace-centos6.tar.gz Thanks, Robert |
|
|
|
|
|
Hi all, we hit the same issue from time to time. CentOS 6, Bareos 15.2.2-37.1 No trace, only logs: 01-Mär 17:38 bareos-dir JobId 439358: console command: run BeforeJob "prune files client=clientname yes" 01-Mär 17:38 bareos-dir JobId 439358: Start Backup JobId 439358, Job=jobname.2017-03-01_17.38.33_29 01-Mär 17:38 bareos-dir JobId 439358: Using Device "FileStorageName0" to write. 01-Mär 17:39 bareos-dir JobId 439358: Fatal error: Catalog error creating JobMedia record. sql_create.c:153 Update Media record UPDATE Media SET EndFile=0, EndBlock=0 WHERE MediaId=0 failed: ERR= 01-Mär 17:39 bareos-sd JobId 439358: Fatal error: Error creating JobMedia record: 1992 Create JobMedia error 01-Mär 17:39 clientname JobId 439358: Fatal error: dir_cmd.c:2400 Bad response to Append Data command. Wanted 3000 OK data 01-Mär 17:39 bareos-sd JobId 439358: Marking Volume "VOLUME41965" in Error in Catalog. 01-Mär 17:39 bareos-sd JobId 439358: Fatal error: askdir.c:357 NULL Volume name. This shouldn't happen!!! 01-Mär 17:39 bareos-sd JobId 439358: Fatal error: Job 439358 canceled. Regards, Christian |
|
|
Hi there. Sam issue, pretty sure it's because of multiple devices defined (needed to forever incremental). |
|
|
I think I managed to track down properly this bug. It seems to be introduced by a large fix for split brain problem[1]. Looks like there is a race condition for volumes on backup start. This causes a lot of volumes to be 'reserved' and marked as 'unwanted' for other backups. If a request is made for a volume that is 'unwanted' then a request should be repeated. However, only case where this is properly handled is if you have ran out of all volumes and in need to purge/recycle oldest. Only in such case it would retry on failure to get volume. I am attaching proposed patch which would retry on every failure. Patch is minimal and easy to review before you decide on implementing or not. This is tested in small environment and proved to work fine. Couldn't confirm in our production due to build limitations we have. [1] https://github.com/bareos/bareos/commit/fa7a54a3b19f2f003a076b36532e277d98aaa812 Kind regards, Momo. |
|
|
|
|
|
Fix committed to bareos bareos-16.2 branch with changesetid 7019. |
|
|
Fix committed to bareos bareos-15.2 branch with changesetid 7026. |
|
|
@Momo: thank you for your patch. We analysed this problem in a customer environment, and think the patch Philipp did make, solved the problem. It just did take longer, until we pushed it to github. |
- Anonymous
