View Issue Details

IDProjectCategoryView StatusLast Update
0000636bareos-corestorage daemonpublic2023-12-13 15:34
Reporteraxestr Assigned Tobruno-at-bareos  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionunable to reproduce 
PlatformLinuxOSCentOSOS Version6
Product Version15.2.2 
Summary0000636: Error on Copy Job / Maybe simmilar to 0000361
DescriptionFor an copy job, I become allways the error
 Fatal error: append.c:191 FI=51 from SD not positive or sequential=0
see additional information for full job log.
All other jobs and copy jobs are running fine.
Maybe, this is simmialar to 0000361?

How can I help to resolve this Issue?
Steps To ReproduceScheduler reproduces this error daily.
Additional Information30-Mar 01:40 home-dir JobId 9703: Copying using JobId=6613 Job=MsWs1-Users.2016-02-18_22.30.00_32
30-Mar 01:40 home-dir JobId 9703: Bootstrap records written to /var/lib/bareos/home-dir.restore.9.bsr
30-Mar 01:40 home-dir JobId 9703: Start Copying JobId 9703, Job=Copy-HomeLocal-Q-to-Backup1-Q.2016-03-30_01.40.03_21
30-Mar 01:40 home-dir JobId 9703: Using Device "Home-Store1" to read.
30-Mar 01:40 home-dir JobId 9704: Using Device "Backup1-Store1" to write.
30-Mar 01:40 backup1-sd JobId 9704: Volume "Backup1-Q-4" previously written, moving to end of data.
30-Mar 01:40 home-sd JobId 9703: Ready to read from volume "HomeLocal-Q-2" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
30-Mar 01:40 backup1-sd JobId 9704: Ready to append to end of Volume "Backup1-Q-4" size=6782764324
30-Mar 01:40 home-sd JobId 9703: Forward spacing Volume "HomeLocal-Q-2" to file:block 2:2081572941.
30-Mar 01:40 home-sd JobId 9703: End of Volume at file 2 on device "Home-Store1" (/home/system-shares/bareos-storage/store1), Volume "HomeLocal-Q-2"
30-Mar 01:40 home-sd JobId 9703: Ready to read from volume "HomeLocal-Q-3" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
30-Mar 01:40 backup1-sd JobId 9704: Fatal error: append.c:191 FI=51 from SD not positive or sequential=0
30-Mar 01:40 backup1-sd JobId 9704: Elapsed time=00:00:01, Transfer rate=0 Bytes/second
30-Mar 01:40 home-sd JobId 9703: Forward spacing Volume "HomeLocal-Q-3" to file:block 0:11067481.
30-Mar 01:40 home-sd JobId 9703: Error: bsock_tcp.c:422 Write error sending -1 bytes to Storage daemon:backup1.pmit.cc:9103: ERR=Connection reset by peer
30-Mar 01:40 home-sd JobId 9703: Fatal error: mac.c:537 Network send error to SD. ERR=Connection reset by peer
30-Mar 01:40 home-dir JobId 9703: Error: Bareos home-dir 15.2.2 (16Nov15):
  Build OS: x86_64-redhat-linux-gnu redhat CentOS release 6.6 (Final)
  Prev Backup JobId: 6613
  Prev Backup Job: MsWs1-Users.2016-02-18_22.30.00_32
  New Backup JobId: 9704
  Current JobId: 9703
  Current Job: Copy-HomeLocal-Q-to-Backup1-Q.2016-03-30_01.40.03_21
  Backup Level: Full
  Client: Dummy
  FileSet: "Dummy"
  Read Pool: "HomeLocal-Q" (From Job resource)
  Read Storage: "Home-Store1" (From Pool resource)
  Write Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Write Storage: "Backup1-Store1" (From Storage from Pool's NextPool resource)
  Next Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Catalog: "MyCatalog" (From Client resource)
  Start time: 30-Mar-2016 01:40:05
  End time: 30-Mar-2016 01:40:05
  Elapsed time: 0 secs
  Priority: 10
  SD Files Written: 2
  SD Bytes Written: 3,078 (3.078 KB)
  Rate: 0.0 KB/s
  Volume name(s):
  Volume Session Id: 50
  Volume Session Time: 1459228020
  Last Volume Bytes: 0 (0 B)
  SD Errors: 1
  SD termination status: Fatal Error
  Termination: *** Copying Error ***
TagsNo tags attached.

Activities

axestr

axestr

2016-04-08 07:35

reporter   ~0002230

Found a nasty workaround. The problem was the copy of a job which resits on two volumen (HomeLocal-Q-2 and HomeLocal-Q-3).
Deleted the job, now everything is fine. But deleting the job is not a clean solution ;-)
mvwieringen

mvwieringen

2016-05-09 19:13

developer   ~0002265

I think this is a corner case where there is not much on the first
volume (e.g. not even a full data record) and as such an ASSERT is triggered
that makes sure that the FI (FileIndex) is progressing. Probably a serious
difficult one to reproduce in a reliable way to be able to create a workaround.
As you deleted the Job we also have not really a way to get some higher debug
output to see if my hunch is right.
axestr

axestr

2016-05-10 06:21

reporter   ~0002266

If the needed Information is in the PostgreSQL database, I can get backups from this database.
Also, I have this information from BAT and following reports:

Error: Bareos home-dir 15.2.2 (16Nov15):
  Build OS: x86_64-redhat-linux-gnu redhat CentOS release 6.6 (Final)
  Prev Backup JobId: 6613
  Prev Backup Job: MsWs1-Users.2016-02-18_22.30.00_32
  New Backup JobId: 10355
  Current JobId: 10354
  Current Job: Copy-HomeLocal-Q-to-Backup1-Q.2016-04-07_01.40.02_26
  Backup Level: Full
  Client: Dummy
  FileSet: "Dummy"
  Read Pool: "HomeLocal-Q" (From Job resource)
  Read Storage: "Home-Store1" (From Pool resource)
  Write Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Write Storage: "Backup1-Store1" (From Storage from Pool's NextPool resource)
  Next Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Catalog: "MyCatalog" (From Client resource)
  Start time: 07-Apr-2016 01:40:05
  End time: 07-Apr-2016 01:40:05
  Elapsed time: 0 secs
  Priority: 10
  SD Files Written: 2
  SD Bytes Written: 3,078 (3.078 KB)
  Rate: 0.0 KB/s
  Volume name(s):
  Volume Session Id: 510
  Volume Session Time: 1459228020
  Last Volume Bytes: 0 (0 B)
  SD Errors: 1
  SD termination status: Fatal Error
  Termination: *** Copying Error ***




27-Mar 01:20 home-dir JobId 9457: Copying using JobId=6613 Job=MsWs1-Users.2016-02-18_22.30.00_32
27-Mar 01:20 home-dir JobId 9457: Bootstrap records written to /var/lib/bareos/home-dir.restore.4.bsr
27-Mar 01:20 home-dir JobId 9457: Start Copying JobId 9457, Job=Copy-HomeLocal-Q-to-Backup1-Q.2016-03-27_01.20.02_58
27-Mar 01:20 home-dir JobId 9457: Using Device "Home-Store1" to read.
27-Mar 01:20 home-dir JobId 9458: Using Device "Backup1-Store1" to write.
27-Mar 01:20 backup1-sd JobId 9458: Volume "Backup1-Q-4" previously written, moving to end of data.
27-Mar 01:20 home-sd JobId 9457: Ready to read from volume "HomeLocal-Q-2" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
27-Mar 01:20 backup1-sd JobId 9458: Ready to append to end of Volume "Backup1-Q-4" size=6143665867
27-Mar 01:20 home-sd JobId 9457: Forward spacing Volume "HomeLocal-Q-2" to file:block 2:2081572941.
27-Mar 01:20 home-sd JobId 9457: End of Volume at file 2 on device "Home-Store1" (/home/system-shares/bareos-storage/store1), Volume "HomeLocal-Q-2"
27-Mar 01:20 home-sd JobId 9457: Ready to read from volume "HomeLocal-Q-3" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
27-Mar 01:20 backup1-sd JobId 9458: Fatal error: append.c:191 FI=51 from SD not positive or sequential=0
27-Mar 01:20 backup1-sd JobId 9458: Elapsed time=00:00:01, Transfer rate=0 Bytes/second
27-Mar 01:20 home-sd JobId 9457: Forward spacing Volume "HomeLocal-Q-3" to file:block 0:11067481.
27-Mar 01:20 home-sd JobId 9457: Error: bsock_tcp.c:422 Write error sending -1 bytes to Storage daemon:backup1.pmit.cc:9103: ERR=Connection reset by peer
27-Mar 01:20 home-sd JobId 9457: Fatal error: mac.c:537 Network send error to SD. ERR=Connection reset by peer
27-Mar 01:20 home-dir JobId 9457: Error: Bareos home-dir 15.2.2 (16Nov15):
  Build OS: x86_64-redhat-linux-gnu redhat CentOS release 6.6 (Final)
  Prev Backup JobId: 6613
  Prev Backup Job: MsWs1-Users.2016-02-18_22.30.00_32
  New Backup JobId: 9458
  Current JobId: 9457
  Current Job: Copy-HomeLocal-Q-to-Backup1-Q.2016-03-27_01.20.02_58
  Backup Level: Full
  Client: Dummy
  FileSet: "Dummy"
  Read Pool: "HomeLocal-Q" (From Job resource)
  Read Storage: "Home-Store1" (From Pool resource)
  Write Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Write Storage: "Backup1-Store1" (From Storage from Pool's NextPool resource)
  Next Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Catalog: "MyCatalog" (From Client resource)
  Start time: 27-Mar-2016 01:20:03
  End time: 27-Mar-2016 01:20:03
  Elapsed time: 0 secs
  Priority: 10
  SD Files Written: 2
  SD Bytes Written: 3,078 (3.078 KB)
  Rate: 0.0 KB/s
  Volume name(s):
  Volume Session Id: 44
  Volume Session Time: 1458977449
  Last Volume Bytes: 0 (0 B)
  SD Errors: 1
  SD termination status: Fatal Error
  Termination: *** Copying Error ***



28-Mar 01:40 home-dir JobId 9533: Copying using JobId=6613 Job=MsWs1-Users.2016-02-18_22.30.00_32
28-Mar 01:40 home-dir JobId 9533: Bootstrap records written to /var/lib/bareos/home-dir.restore.3.bsr
28-Mar 01:40 home-dir JobId 9533: Start Copying JobId 9533, Job=Copy-HomeLocal-Q-to-Backup1-Q.2016-03-28_01.40.00_30
28-Mar 01:40 home-dir JobId 9533: Using Device "Home-Store1" to read.
28-Mar 01:40 home-dir JobId 9534: Using Device "Backup1-Store1" to write.
28-Mar 01:40 backup1-sd JobId 9534: Volume "Backup1-Q-4" previously written, moving to end of data.
28-Mar 01:40 home-sd JobId 9533: Ready to read from volume "HomeLocal-Q-2" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
28-Mar 01:40 backup1-sd JobId 9534: Ready to append to end of Volume "Backup1-Q-4" size=6592042035
28-Mar 01:40 home-sd JobId 9533: Forward spacing Volume "HomeLocal-Q-2" to file:block 2:2081572941.
28-Mar 01:40 home-sd JobId 9533: End of Volume at file 2 on device "Home-Store1" (/home/system-shares/bareos-storage/store1), Volume "HomeLocal-Q-2"
28-Mar 01:40 home-sd JobId 9533: Ready to read from volume "HomeLocal-Q-3" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
28-Mar 01:40 backup1-sd JobId 9534: Fatal error: append.c:191 FI=51 from SD not positive or sequential=0
28-Mar 01:40 backup1-sd JobId 9534: Elapsed time=00:00:01, Transfer rate=0 Bytes/second
28-Mar 01:40 home-sd JobId 9533: Forward spacing Volume "HomeLocal-Q-3" to file:block 0:11067481.
28-Mar 01:40 home-sd JobId 9533: Error: bsock_tcp.c:422 Write error sending -1 bytes to Storage daemon:backup1.pmit.cc:9103: ERR=Connection reset by peer
28-Mar 01:40 home-sd JobId 9533: Fatal error: mac.c:537 Network send error to SD. ERR=Connection reset by peer
28-Mar 01:40 home-dir JobId 9533: Error: Bareos home-dir 15.2.2 (16Nov15):
  Build OS: x86_64-redhat-linux-gnu redhat CentOS release 6.6 (Final)
  Prev Backup JobId: 6613
  Prev Backup Job: MsWs1-Users.2016-02-18_22.30.00_32
  New Backup JobId: 9534
  Current JobId: 9533
  Current Job: Copy-HomeLocal-Q-to-Backup1-Q.2016-03-28_01.40.00_30
  Backup Level: Full
  Client: Dummy
  FileSet: "Dummy"
  Read Pool: "HomeLocal-Q" (From Job resource)
  Read Storage: "Home-Store1" (From Pool resource)
  Write Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Write Storage: "Backup1-Store1" (From Storage from Pool's NextPool resource)
  Next Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Catalog: "MyCatalog" (From Client resource)
  Start time: 28-Mar-2016 01:40:32
  End time: 28-Mar-2016 01:40:33
  Elapsed time: 1 sec
  Priority: 10
  SD Files Written: 2
  SD Bytes Written: 3,078 (3.078 KB)
  Rate: 3.1 KB/s
  Volume name(s):
  Volume Session Id: 27
  Volume Session Time: 1459067007
  Last Volume Bytes: 0 (0 B)
  SD Errors: 1
  SD termination status: Fatal Error
  Termination: *** Copying Error ***



29-Mar 01:40 home-dir JobId 9607: Copying using JobId=6613 Job=MsWs1-Users.2016-02-18_22.30.00_32
29-Mar 01:40 home-dir JobId 9607: Bootstrap records written to /var/lib/bareos/home-dir.restore.1.bsr
29-Mar 01:40 home-dir JobId 9607: Start Copying JobId 9607, Job=Copy-HomeLocal-Q-to-Backup1-Q.2016-03-29_01.40.02_44
29-Mar 01:40 home-dir JobId 9607: Using Device "Home-Store1" to read.
29-Mar 01:40 home-dir JobId 9608: Using Device "Backup1-Store1" to write.
29-Mar 01:40 backup1-sd JobId 9608: Volume "Backup1-Q-4" previously written, moving to end of data.
29-Mar 01:40 home-sd JobId 9607: Ready to read from volume "HomeLocal-Q-2" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
29-Mar 01:40 backup1-sd JobId 9608: Ready to append to end of Volume "Backup1-Q-4" size=6592238917
29-Mar 01:40 home-sd JobId 9607: Forward spacing Volume "HomeLocal-Q-2" to file:block 2:2081572941.
29-Mar 01:40 home-sd JobId 9607: End of Volume at file 2 on device "Home-Store1" (/home/system-shares/bareos-storage/store1), Volume "HomeLocal-Q-2"
29-Mar 01:40 home-sd JobId 9607: Ready to read from volume "HomeLocal-Q-3" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
29-Mar 01:40 backup1-sd JobId 9608: Fatal error: append.c:191 FI=51 from SD not positive or sequential=0
29-Mar 01:40 backup1-sd JobId 9608: Elapsed time=00:00:01, Transfer rate=0 Bytes/second
29-Mar 01:40 home-sd JobId 9607: Forward spacing Volume "HomeLocal-Q-3" to file:block 0:11067481.
29-Mar 01:40 home-sd JobId 9607: Error: bsock_tcp.c:422 Write error sending -1 bytes to Storage daemon:backup1.pmit.cc:9103: ERR=Connection reset by peer
29-Mar 01:40 home-sd JobId 9607: Fatal error: mac.c:537 Network send error to SD. ERR=Connection reset by peer
29-Mar 01:40 home-dir JobId 9607: Error: Bareos home-dir 15.2.2 (16Nov15):
  Build OS: x86_64-redhat-linux-gnu redhat CentOS release 6.6 (Final)
  Prev Backup JobId: 6613
  Prev Backup Job: MsWs1-Users.2016-02-18_22.30.00_32
  New Backup JobId: 9608
  Current JobId: 9607
  Current Job: Copy-HomeLocal-Q-to-Backup1-Q.2016-03-29_01.40.02_44
  Backup Level: Full
  Client: Dummy
  FileSet: "Dummy"
  Read Pool: "HomeLocal-Q" (From Job resource)
  Read Storage: "Home-Store1" (From Pool resource)
  Write Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Write Storage: "Backup1-Store1" (From Storage from Pool's NextPool resource)
  Next Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Catalog: "MyCatalog" (From Client resource)
  Start time: 29-Mar-2016 01:40:05
  End time: 29-Mar-2016 01:40:05
  Elapsed time: 0 secs
  Priority: 10
  SD Files Written: 2
  SD Bytes Written: 3,078 (3.078 KB)
  Rate: 0.0 KB/s
  Volume name(s):
  Volume Session Id: 79
  Volume Session Time: 1459067007
  Last Volume Bytes: 0 (0 B)
  SD Errors: 1
  SD termination status: Fatal Error
  Termination: *** Copying Error ***



30-Mar 01:40 home-dir JobId 9703: Copying using JobId=6613 Job=MsWs1-Users.2016-02-18_22.30.00_32
30-Mar 01:40 home-dir JobId 9703: Bootstrap records written to /var/lib/bareos/home-dir.restore.9.bsr
30-Mar 01:40 home-dir JobId 9703: Start Copying JobId 9703, Job=Copy-HomeLocal-Q-to-Backup1-Q.2016-03-30_01.40.03_21
30-Mar 01:40 home-dir JobId 9703: Using Device "Home-Store1" to read.
30-Mar 01:40 home-dir JobId 9704: Using Device "Backup1-Store1" to write.
30-Mar 01:40 backup1-sd JobId 9704: Volume "Backup1-Q-4" previously written, moving to end of data.
30-Mar 01:40 home-sd JobId 9703: Ready to read from volume "HomeLocal-Q-2" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
30-Mar 01:40 backup1-sd JobId 9704: Ready to append to end of Volume "Backup1-Q-4" size=6782764324
30-Mar 01:40 home-sd JobId 9703: Forward spacing Volume "HomeLocal-Q-2" to file:block 2:2081572941.
30-Mar 01:40 home-sd JobId 9703: End of Volume at file 2 on device "Home-Store1" (/home/system-shares/bareos-storage/store1), Volume "HomeLocal-Q-2"
30-Mar 01:40 home-sd JobId 9703: Ready to read from volume "HomeLocal-Q-3" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
30-Mar 01:40 backup1-sd JobId 9704: Fatal error: append.c:191 FI=51 from SD not positive or sequential=0
30-Mar 01:40 backup1-sd JobId 9704: Elapsed time=00:00:01, Transfer rate=0 Bytes/second
30-Mar 01:40 home-sd JobId 9703: Forward spacing Volume "HomeLocal-Q-3" to file:block 0:11067481.
30-Mar 01:40 home-sd JobId 9703: Error: bsock_tcp.c:422 Write error sending -1 bytes to Storage daemon:backup1.pmit.cc:9103: ERR=Connection reset by peer
30-Mar 01:40 home-sd JobId 9703: Fatal error: mac.c:537 Network send error to SD. ERR=Connection reset by peer
30-Mar 01:40 home-dir JobId 9703: Error: Bareos home-dir 15.2.2 (16Nov15):
  Build OS: x86_64-redhat-linux-gnu redhat CentOS release 6.6 (Final)
  Prev Backup JobId: 6613
  Prev Backup Job: MsWs1-Users.2016-02-18_22.30.00_32
  New Backup JobId: 9704
  Current JobId: 9703
  Current Job: Copy-HomeLocal-Q-to-Backup1-Q.2016-03-30_01.40.03_21
  Backup Level: Full
  Client: Dummy
  FileSet: "Dummy"
  Read Pool: "HomeLocal-Q" (From Job resource)
  Read Storage: "Home-Store1" (From Pool resource)
  Write Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Write Storage: "Backup1-Store1" (From Storage from Pool's NextPool resource)
  Next Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Catalog: "MyCatalog" (From Client resource)
  Start time: 30-Mar-2016 01:40:05
  End time: 30-Mar-2016 01:40:05
  Elapsed time: 0 secs
  Priority: 10
  SD Files Written: 2
  SD Bytes Written: 3,078 (3.078 KB)
  Rate: 0.0 KB/s
  Volume name(s):
  Volume Session Id: 50
  Volume Session Time: 1459228020
  Last Volume Bytes: 0 (0 B)
  SD Errors: 1
  SD termination status: Fatal Error
  Termination: *** Copying Error ***



07-Apr 01:40 home-dir JobId 10354: Copying using JobId=6613 Job=MsWs1-Users.2016-02-18_22.30.00_32
07-Apr 01:40 home-dir JobId 10354: Bootstrap records written to /var/lib/bareos/home-dir.restore.170.bsr
07-Apr 01:40 home-dir JobId 10354: Start Copying JobId 10354, Job=Copy-HomeLocal-Q-to-Backup1-Q.2016-04-07_01.40.02_26
07-Apr 01:40 home-dir JobId 10354: Using Device "Home-Store1" to read.
07-Apr 01:40 home-dir JobId 10355: Using Device "Backup1-Store1" to write.
07-Apr 01:40 backup1-sd JobId 10355: Volume "Backup1-Q-0" previously written, moving to end of data.
07-Apr 01:40 home-sd JobId 10354: Ready to read from volume "HomeLocal-Q-2" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
07-Apr 01:40 backup1-sd JobId 10355: Ready to append to end of Volume "Backup1-Q-0" size=3490511342
07-Apr 01:40 home-sd JobId 10354: Forward spacing Volume "HomeLocal-Q-2" to file:block 2:2081572941.
07-Apr 01:40 home-sd JobId 10354: End of Volume at file 2 on device "Home-Store1" (/home/system-shares/bareos-storage/store1), Volume "HomeLocal-Q-2"
07-Apr 01:40 home-sd JobId 10354: Ready to read from volume "HomeLocal-Q-3" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
07-Apr 01:40 backup1-sd JobId 10355: Fatal error: append.c:191 FI=51 from SD not positive or sequential=0
07-Apr 01:40 backup1-sd JobId 10355: Elapsed time=00:00:01, Transfer rate=0 Bytes/second
07-Apr 01:40 home-sd JobId 10354: Forward spacing Volume "HomeLocal-Q-3" to file:block 0:11067481.
07-Apr 01:40 home-sd JobId 10354: Error: bsock_tcp.c:422 Write error sending -1 bytes to Storage daemon:backup1.pmit.cc:9103: ERR=Connection reset by peer
07-Apr 01:40 home-sd JobId 10354: Fatal error: mac.c:537 Network send error to SD. ERR=Connection reset by peer
07-Apr 01:40 home-dir JobId 10354: Error: Bareos home-dir 15.2.2 (16Nov15):
  Build OS: x86_64-redhat-linux-gnu redhat CentOS release 6.6 (Final)
  Prev Backup JobId: 6613
  Prev Backup Job: MsWs1-Users.2016-02-18_22.30.00_32
  New Backup JobId: 10355
  Current JobId: 10354
  Current Job: Copy-HomeLocal-Q-to-Backup1-Q.2016-04-07_01.40.02_26
  Backup Level: Full
  Client: Dummy
  FileSet: "Dummy"
  Read Pool: "HomeLocal-Q" (From Job resource)
  Read Storage: "Home-Store1" (From Pool resource)
  Write Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Write Storage: "Backup1-Store1" (From Storage from Pool's NextPool resource)
  Next Pool: "Backup1-Q" (From Job Pool's NextPool resource)
  Catalog: "MyCatalog" (From Client resource)
  Start time: 07-Apr-2016 01:40:05
  End time: 07-Apr-2016 01:40:05
  Elapsed time: 0 secs
  Priority: 10
  SD Files Written: 2
  SD Bytes Written: 3,078 (3.078 KB)
  Rate: 0.0 KB/s
  Volume name(s):
  Volume Session Id: 510
  Volume Session Time: 1459228020
  Last Volume Bytes: 0 (0 B)
  SD Errors: 1
  SD termination status: Fatal Error
  Termination: *** Copying Error ***
axestr

axestr

2017-11-04 08:37

reporter   ~0002811

Hi Bareos Team,

Same error pop's up again daily. What could I do/track to identify?

04-Nov 01:20 home-dir JobId 64023: Copying using JobId=62888 Job=Leela-Users.2017-10-22_22.30.00_57
04-Nov 01:20 home-dir JobId 64023: Bootstrap records written to /var/lib/bareos/home-dir.restore.108.bsr
04-Nov 01:20 home-dir JobId 64023: Start Copying JobId 64023, Job=Copy-HomeLocal-Y-to-Backup1-Y.2017-11-04_01.20.02_23
04-Nov 01:20 home-dir JobId 64023: Using Device "Home-Store1" to read.
04-Nov 01:20 home-dir JobId 64024: Using Device "Backup1-Store1" to write.
04-Nov 01:20 home-sd JobId 64023: Ready to read from volume "HomeLocal-Y-10" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
04-Nov 01:20 backup1-sd JobId 64024: Volume "Backup1-Y-11" previously written, moving to end of data.
04-Nov 01:20 backup1-sd JobId 64024: Ready to append to end of Volume "Backup1-Y-11" size=3495845162
04-Nov 01:20 home-sd JobId 64023: Forward spacing Volume "HomeLocal-Y-10" to file:block 2:2100121434.
04-Nov 01:20 home-sd JobId 64023: End of Volume at file 2 on device "Home-Store1" (/home/system-shares/bareos-storage/store1), Volume "HomeLocal-Y-10"
04-Nov 01:20 home-sd JobId 64023: Ready to read from volume "HomeLocal-Y-11" on device "Home-Store1" (/home/system-shares/bareos-storage/store1).
04-Nov 01:20 backup1-sd JobId 64024: Fatal error: append.c:191 FI=217 from SD not positive or sequential=0
04-Nov 01:20 home-sd JobId 64023: Forward spacing Volume "HomeLocal-Y-11" to file:block 0:21805270.
04-Nov 01:20 home-sd JobId 64023: Error: bsock_tcp.c:422 Write error sending 14208 bytes to Storage daemon:backup1.pmit.cc:9103: ERR=Connection reset by peer
04-Nov 01:20 home-sd JobId 64023: Fatal error: mac.c:326 Network send error to SD. ERR=Connection reset by peer
04-Nov 01:20 home-sd JobId 64023: Error: bsock_tcp.c:357 Socket has errors=1 on call to Storage daemon:backup1.pmit.cc:9103
04-Nov 01:20 backup1-sd JobId 64024: Elapsed time=00:00:01, Transfer rate=0 Bytes/second
04-Nov 01:20 home-dir JobId 64023: Error: Bareos home-dir 15.2.2 (16Nov15):
  Build OS: x86_64-redhat-linux-gnu redhat CentOS release 6.6 (Final)
  Prev Backup JobId: 62888
  Prev Backup Job: Leela-Users.2017-10-22_22.30.00_57
  New Backup JobId: 64024
  Current JobId: 64023
  Current Job: Copy-HomeLocal-Y-to-Backup1-Y.2017-11-04_01.20.02_23
  Backup Level: Full
  Client: Dummy
  FileSet: "Dummy"
  Read Pool: "HomeLocal-Y" (From Job resource)
  Read Storage: "Home-Store1" (From Pool resource)
  Write Pool: "Backup1-Y" (From Job Pool's NextPool resource)
  Write Storage: "Backup1-Store1" (From Storage from Pool's NextPool resource)
  Next Pool: "Backup1-Y" (From Job Pool's NextPool resource)
  Catalog: "MyCatalog" (From Client resource)
  Start time: 04-Nov-2017 01:20:04
  End time: 04-Nov-2017 01:20:05
  Elapsed time: 1 sec
  Priority: 10
  SD Files Written: 1
  SD Bytes Written: 3,606 (3.606 KB)
  Rate: 3.6 KB/s
  Volume name(s):
  Volume Session Id: 315
  Volume Session Time: 1509371965
  Last Volume Bytes: 0 (0 B)
  SD Errors: 2
  SD termination status: Fatal Error
  Termination: *** Copying Error ***

BR, Axel
IvanBayan

IvanBayan

2018-05-23 11:43

reporter   ~0003014

I have same problem.
Bareos version 17.2.4-9.1 installed from http://download.bareos.org/bareos/release/latest/xUbuntu_16.04.

First it looks like network problem, but when I've made dump I've found that TCP connection closed by storage without FIN flag, eventually it just started to sent RST flag.
In debug output from storage I've found next:
 23-May-2018 04:52:49.623885 ow-backup03-sd (850): message.c:858-6475 Enter dispatch_message type=3 msg=ow-backup03-sd JobId 6475: Fatal error: append.c:192 FI=5 from SD not positive or sequential=0
23-May-2018 04:52:49.623899 ow-backup03-sd (850): message.c:1129-6475 DIRECTOR for following msg: ow-backup03-sd JobId 6475: Fatal error: append.c:192 FI=5 from SD not positive or sequential=0

It only occurs when I try to run copy job.
IvanBayan

IvanBayan

2018-05-23 14:12

reporter   ~0003015

Copy job is not completely broken, it can't copy only jobs which share volume with another job:
*list jobs volume=ord-uploadcdr01-fd-mia-backup03_WV-CDRs_full-20180514-5851-486
+-------+----------------------------+--------------------+---------------------+------+-------+----------+-------------+-----------+
| jobid | name | client | starttime | type | level | jobfiles | jobbytes | jobstatus |
+-------+----------------------------+--------------------+---------------------+------+-------+----------+-------------+-----------+
| 5,926 | ow-cdrupload01_CDRs_daily | ow-cdrupload01-fd | 2018-05-15 21:00:02 | B | F | 108 | 809,354,467 | T |
| 5,927 | ord-uploadcdr01_CDRs_daily | ord-uploadcdr01-fd | 2018-05-15 21:00:03 | B | F | 70 | 334,296,463 | T |
+-------+----------------------------+--------------------+---------------------+------+-------+----------+-------------+-----------+
*list volumes jobid=5926
Jobid 5926 used 1 Volume(s): ord-uploadcdr01-fd-mia-backup03_WV-CDRs_full-20180514-5851-486
*list volumes jobid=5927
Jobid 5927 used 1 Volume(s): ord-uploadcdr01-fd-mia-backup03_WV-CDRs_full-20180514-5851-486

In webui job looks like it uses two volumes.
IvanBayan

IvanBayan

2018-05-23 14:13

reporter  

bareos_volumes.png (33,262 bytes)   
bareos_volumes.png (33,262 bytes)   
bruno-at-bareos

bruno-at-bareos

2023-12-13 15:34

manager   ~0005636

can't be reproduced with recent version. If you are such a case, please don't delete the job or volumes and reopen an new issue.

Issue History

Date Modified Username Field Change
2016-03-30 08:50 axestr New Issue
2016-04-08 07:35 axestr Note Added: 0002230
2016-05-09 19:13 mvwieringen Note Added: 0002265
2016-05-09 19:13 mvwieringen Status new => feedback
2016-05-10 06:21 axestr Note Added: 0002266
2016-05-10 06:21 axestr Status feedback => new
2017-11-04 08:37 axestr Note Added: 0002811
2018-05-23 11:43 IvanBayan Note Added: 0003014
2018-05-23 14:12 IvanBayan Note Added: 0003015
2018-05-23 14:13 IvanBayan File Added: bareos_volumes.png
2023-12-13 15:34 bruno-at-bareos Assigned To => bruno-at-bareos
2023-12-13 15:34 bruno-at-bareos Status new => closed
2023-12-13 15:34 bruno-at-bareos Resolution open => unable to reproduce
2023-12-13 15:34 bruno-at-bareos Note Added: 0005636