Bareos Bug Tracker
Bareos Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000639bareos-core[All Projects] storage daemonpublic2016-04-05 23:492017-08-18 10:42
Reporterareplogle 
Assigned To 
PrioritylowSeverityminorReproducibilityalways
StatusacknowledgedResolutionopen 
PlatformLinuxOSUbuntuOS Version14.04
Product Version15.2.2 
Target VersionFixed in Version 
Summary0000639: bareos unable to read AWS VTL iSCSI devices
DescriptionWhen using AWS Storage Gateway, trying to use bareos utilities to access the iSCSI devices exposed by Amazon's storage gateway results in failure on read attempts.


---<example>---
# mt -f /dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst rewind
# btape -c /etc/bareos/bareos-sd.conf /dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst
Tape block granularity is 1024 bytes.
btape: butil.c:274-0 Using device: "/dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst" for writing.
btape: btape.c:487-0 open device "Drive-1" (/dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst): OK
*test

=== Write, rewind, and re-read test ===

I'm going to write 10000 records and an EOF
then write 10000 records and an EOF, then rewind,
and re-read the data to verify that it is correct.

This is an *essential* feature ...

btape: btape.c:1172-0 Wrote 10000 blocks of 64412 bytes.
btape: btape.c:619-0 Wrote 1 EOF to "Drive-1" (/dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst)
btape: btape.c:1188-0 Wrote 10000 blocks of 64412 bytes.
btape: btape.c:619-0 Wrote 1 EOF to "Drive-1" (/dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst)
btape: btape.c:1230-0 Rewind OK.
05-Apr 17:39 btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "Drive-1" (/dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst). ERR=Input/output error.
btape: btape.c:1250-0 Read block 1 failed! ERR=Input/output error
--</example>--

Steps To ReproduceInstall bareos 15.2.2 on Ubuntu 14.04
Deploy AWS Gateway VM (There's a 60 day free trial if you need to reproduce) in your network.
Install lsscsi / open-iscsi packages
Set /etc/iscsi/iscsid.conf to automatic
Discover and register devices.
Run btape test.
Additional InformationUsing standard posix utilities like tar, I'm able to write and read from the AWS vtapes.
---<example>---
root@pbkupph01:~# mt -f /dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst rewind
root@pbkupph01:~# tar cvf /dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0 .
./
./.aliasrc
./.viminfo
./.bashrc
./.ssh/
./.ssh/known_hosts
./.bconsole_history
./.bash_history
./.profile
root@pbkupph01:~# mt -f /dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst rewind
root@pbkupph01:~# tar tvf /dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0
drwx------ root/root 0 2016-04-02 15:02 ./
-rw-r--r-- root/root 212 2016-04-02 15:02 ./.aliasrc
-rw------- root/root 14029 2016-04-02 15:02 ./.viminfo
-rw-r--r-- root/root 3216 2016-04-02 14:52 ./.bashrc
drwx------ root/root 0 2016-02-23 09:22 ./.ssh/
-rw-r--r-- root/root 444 2016-02-23 09:22 ./.ssh/known_hosts
-rw------- root/root 637 2016-04-02 14:50 ./.bconsole_history
-rw------- root/root 18424 2016-04-02 16:29 ./.bash_history
-rw-r--r-- root/root 140 2014-02-19 21:43 ./.profile
root@pbkupph01:~# mkdir tmp
root@pbkupph01:~# cd tmp
root@pbkupph01:~/tmp# mt -f /dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0-nst rewind
root@pbkupph01:~/tmp# tar xvf /dev/tape/by-path/ip-172.22.2.250:3260-iscsi-iqn.1997-05.com.amazon:sgw-5d977234-tapedrive-01-lun-0
./
./.aliasrc
./.viminfo
./.bashrc
./.ssh/
./.ssh/known_hosts
./.bconsole_history
./.bash_history
./.profile
root@pbkupph01:~/tmp# ls -la
total 64
drwx------ 3 root root 4096 Apr 2 15:02 .
drwx------ 4 root root 4096 Apr 5 17:43 ..
-rw-r--r-- 1 root root 212 Apr 2 15:02 .aliasrc
-rw------- 1 root root 18424 Apr 2 16:29 .bash_history
-rw-r--r-- 1 root root 3216 Apr 2 14:52 .bashrc
-rw------- 1 root root 637 Apr 2 14:50 .bconsole_history
-rw-r--r-- 1 root root 140 Feb 19 2014 .profile
drwx------ 2 root root 4096 Feb 23 09:22 .ssh
-rw------- 1 root root 14029 Apr 2 15:02 .viminfo
root@pbkupph01:~/tmp#
---</example>---
TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action
Attached Filestxt file icon bareos-sysdig.txt [^] (13,809 bytes) 2016-04-06 00:47 [Show Content]

- Relationships
has duplicate 0000743new Bareos Amazon VTL block.c:1003 Read error on fd=6 at file:blk 0:0 on device "Drive_01" (/dev/st3). ERR=Input/output error. 

-  Notes
(0002228)
areplogle (reporter)
2016-04-06 00:53

I've run another test with sysdig in another window (e.g. - sysdig "proc.name=btape" ) executing a write block, rewind and read block.

It appears that its failing to find libc.mo, then trying to get /etc/localtime to determine my TZ (I think).

Is it possible that the write test is writing data thats outside of the codepage range for the read to handle?

Reference: bareos-sysdig.txt
(0002233)
areplogle (reporter)
2016-04-08 22:11

Just a quick update. Same exact issue on Centos 7.2.1511 (core)

# uname -a
Linux pbkupph01 3.10.0-327.13.1.el7.x86_64 0000001 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
(0002246)
areplogle (reporter)
2016-04-24 18:43

For anyone else who's running into issue, for some reason Bacula 7.4 works fine. I've switched to that for now but am bummed since their plugin support isn't nearly as good or well documented.
(0002253)
areplogle (reporter)
2016-04-29 23:57

Had some time to circle back to this reviewing my steps. I realized that I was using the nst device which I thought was the cause of my issue (not being able to rewind), but even using the non-nst device I get the same issue.
(0002254)
areplogle (reporter)
2016-05-01 15:58

Since reproducing this would be a significant effort (have aws acct, deploy storage gateway to local vm env, setup new bareos install etc) I thought it might be easier if I provided the difference between the working block.c from bacula and comparison to the block.c from bareos.

Working:
https://github.com/prelegalwonder/bacula/blob/master/src/stored/block.c#L459 [^]

Input/Output Err:
https://github.com/bareos/bareos/blob/master/src/stored/block.c#L996 [^]

I have very little professional c development experience so forgive my ignorance if I'm just providing non-sense relations, but I also noticed that the bacula stored.h loads an additional vtape_dev.h/c and maybe there's some significant vtl support there thats allowing it to work in bacula?

Please let me know if there's anything I can do to assist.
(0002328)
sassyn (reporter)
2016-07-31 18:30

Hi,

Just had the same issue on ubuntu 14.04 with 15.2.2-37.1.

Did someone manage to solve this?

Is the upcoming 16.3.1 solve this?

Thank You
(0002349)
fbacchella (reporter)
2016-09-07 19:23

Severity minor ? bareos is broken on AWS. I will need to switch to bacula because of this bug.
(0002350)
fbacchella (reporter)
2016-09-07 19:24

I have the problem on RHEL 7.2 too.
(0002412)
maurizio.proietti (reporter)
2016-10-27 10:39

I have the problem on CentOS 7 too with tape library StorageTek L40.
With bacula 7.4 there is not this problem.
With all version of bareos btape fail.
I've tried with bareos 15 and 16 by yum and compiled by source code (from github).

this is the last messages from btape

btape: btape.c:1183-0 Wrote 10000 blocks of 64412 bytes.
btape (129): backends/generic_tape_device.c:368-0 === weof_dev="Drive-1" (/dev/nst0)
btape: btape.c:614-0 Wrote 1 EOF to "Drive-1" (/dev/nst0)
btape (950): record.c:353-0 Enter free_record.
btape (950): record.c:357-0 Data buf is freed.
btape (950): record.c:359-0 Leave free_record.
btape (400): backends/generic_tape_device.c:1115-0 rewind res=0 fd=3 "Drive-1" (/dev/nst0)
btape: btape.c:1225-0 Rewind OK.
btape (250): block.c:960-0 Full read in read_block_from_device() len=64512
btape (250): block.c:1000-0 Read device got: ERR=Errore di input/output
btape (250): sd_plugins.c:319-0 No bplugin_list: generate_plugin_event ignored.
btape (850): message.c:1557-0 Enter Jmsg type=4
btape (850): message.c:858-0 Enter dispatch_message type=4 msg=btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "Drive-1" (/dev/nst0). ERR=Errore di input/output.
btape (850): message.c:1138-0 STDOUT for following msg: btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "Drive-1" (/dev/nst0). ERR=Errore di input/output.
27-ott 10:36 btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "Drive-1" (/dev/nst0). ERR=Errore di input/output.
btape: btape.c:1245-0 Read block 1 failed! ERR=Errore di input/output
btape (950): record.c:353-0 Enter free_record.
btape (950): record.c:357-0 Data buf is freed.
btape (950): record.c:359-0 Leave free_record.

I will need to switch to bacula because of this bug.
(0002628)
damiano.bortolato (reporter)
2017-04-18 11:17
edited on: 2017-04-18 14:17

Same problem on CentOS Linux release 7.3.1611 bareos 16.2.4, generic linux tools working on AWS Storage Gateway and bareos-fd don't.

Error given by bconsole:

Connecting to Storage daemon VTL at bcksrv1.utv.cps:9103 ...
Sending label command for Volume "UTV2A548F" Slot 1 ...
block.c:286 Volume data error at 0:0! Wanted ID: "BB02", got "". Buffer discarded.
block.c:630 Write error at 0:0 on device "amazon-vtl-td0" (/dev/nst7). ERR=Input/output error.
block.c:644 Write error on fd=4 at file:blk 0:0 on device "amazon-vtl-td0" (/dev/nst7). ERR=Input/output error.
Backspace record at EOT failed. ERR=Input/output error
3912 Failed to label Volume: ERR=generic_tape_device.c:726 ioctl MTBSR error on "amazon-vtl-td0" (/dev/nst7). ERR=Input/output error.

Label command failed for Volume UTV2A548F.


I noticed also the following messages on the syslog:

Apr 18 11:01:03 bcksrv1 kernel: connection18:0: detected conn error (1020)
Apr 18 11:01:03 bcksrv1 kernel: st 19:0:0:0: [st7] Error e0000 (driver bt 0x0, host bt 0xe).
Apr 18 11:01:03 bcksrv1 iscsid: Kernel reported iSCSI connection 18:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Apr 18 11:01:05 bcksrv1 kernel: st 19:0:0:0: [st7] Sense Key : Unit Attention [current]
Apr 18 11:01:05 bcksrv1 kernel: st 19:0:0:0: [st7] Add. Sense: Power on, reset, or bus device reset occurred
Apr 18 11:01:05 bcksrv1 kernel: connection18:0: detected conn error (1020)
Apr 18 11:01:05 bcksrv1 kernel: st 19:0:0:0: [st7] Error e0000 (driver bt 0x0, host bt 0xe).
Apr 18 11:01:05 bcksrv1 iscsid: connection18:0 is operational after recovery (1 attempts)
Apr 18 11:01:05 bcksrv1 iscsid: Kernel reported iSCSI connection 18:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Apr 18 11:01:07 bcksrv1 kernel: st 19:0:0:0: [st7] Sense Key : Unit Attention [current]
Apr 18 11:01:07 bcksrv1 kernel: st 19:0:0:0: [st7] Add. Sense: Power on, reset, or bus device reset occurred
Apr 18 11:01:07 bcksrv1 iscsid: connection18:0 is operational after recovery (1 attempts)

Apparently it is due to a communication error which causes an iSCSI disconnection. I would not consider this a MINOR bug, I was switching from Bacula after reading a tutorial give by Luis Alberto Giménez at Open Source Backup Conference in 2015 where he said that everything was working fine with amazon AWS, well now it isn't, so it is a feature that has been broken. I will ask him which bareos version he was running and on what OS...





Tried with

    Backward Space File = no
    Backward Space Record = no

on device config file. Nothing chages:

Connecting to Storage daemon VTL at bcksrv1.utv.cps:9103 ...
Sending label command for Volume "UTV2A548F" Slot 1 ...
block.c:286 Volume data error at 0:0! Wanted ID: "BB02", got "". Buffer discarded.
block.c:630 Write error at 0:0 on device "amazon-vtl-td0" (/dev/nst7). ERR=Input/output error.
block.c:644 Write error on fd=4 at file:blk 0:0 on device "amazon-vtl-td0" (/dev/nst7). ERR=Input/output error.
Backspace record at EOT failed. ERR=Input/output error
3912 Failed to label Volume: ERR=generic_tape_device.c:726 ioctl MTBSR error on "amazon-vtl-td0" (/dev/nst7). ERR=Input/output error.

Label command failed for Volume UTV2A548F.

It is still trying to perform an MTBSR ioctl() call, and there it fails.


After disabling MTBSR btape test gives:

Tape block granularity is 1024 bytes.
btape: butil.c:274-0 Using device: "amazon-vtl-td0" for writing.
btape: btape.c:482-0 open device "amazon-vtl-td0" (/dev/nst7): OK
*test

=== Write, rewind, and re-read test ===

I'm going to write 10000 records and an EOF
then write 10000 records and an EOF, then rewind,
and re-read the data to verify that it is correct.

This is an *essential* feature ...

btape: btape.c:1167-0 Wrote 10000 blocks of 64412 bytes.
btape: btape.c:614-0 Wrote 1 EOF to "amazon-vtl-td0" (/dev/nst7)
btape: btape.c:1183-0 Wrote 10000 blocks of 64412 bytes.
btape: btape.c:614-0 Wrote 1 EOF to "amazon-vtl-td0" (/dev/nst7)
btape: btape.c:1225-0 Rewind OK.
18-Apr 14:16 btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "amazon-vtl-td0" (/dev/nst7). ERR=Input/output error.
btape: btape.c:1245-0 Read block 1 failed! ERR=Input/output error

(0002681)
guglez (reporter)
2017-07-11 15:18

Was it fixed?
(0002688)
samashton (reporter)
2017-07-20 03:20

Same problem on Redhat 7.3 (EC2 AMI), Bareos Version: 16.2.4 (01 July 2016)

Can use mt and tar as described but btape and bareos report ERR=Input/output error
btape result:

btape: btape.c:1167-0 Wrote 10000 blocks of 64412 bytes.
btape: btape.c:614-0 Wrote 1 EOF to "VTLDrive01" (/dev/tape/by-path/ip-172.31.0.30:3260-iscsi-iqn.1997-05.com.amazon:sgw-2d6f8044-tapedrive-01-lun-0-nst)
btape: btape.c:1183-0 Wrote 10000 blocks of 64412 bytes.
btape: btape.c:614-0 Wrote 1 EOF to "VTLDrive01" (/dev/tape/by-path/ip-172.31.0.30:3260-iscsi-iqn.1997-05.com.amazon:sgw-2d6f8044-tapedrive-01-lun-0-nst)
btape: btape.c:1225-0 Rewind OK.
19-Jul 21:05 btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "VTLDrive01" (/dev/tape/by-path/ip-172.31.0.30:3260-iscsi-iqn.1997-05.com.amazon:sgw-2d6f8044-tapedrive-01-lun-0-nst). ERR=Input/output error.
btape: btape.c:1245-0 Read block 1 failed! ERR=Input/output error

And Bareos:

3307 Issuing autochanger "unload slot 9, drive 0" command.
3304 Issuing autochanger "load slot 10, drive 0" command.
3305 Autochanger "load slot 10, drive 0", status is OK.
block.c:630 Write error at 0:0 on device "VTLDrive01" (/dev/tape/by-path/ip-172.31.0.30:3260-iscsi-iqn.1997-05.com.amazon:sgw-2d6f8044-tapedrive-01-lun-0-nst). ERR=Input/output error.
block.c:644 Write error on fd=4 at file:blk 0:0 on device "VTLDrive01" (/dev/tape/by-path/ip-172.31.0.30:3260-iscsi-iqn.1997-05.com.amazon:sgw-2d6f8044-tapedrive-01-lun-0-nst). ERR=Input/output error.
Backspace record at EOT failed. ERR=Input/output error
3912 Failed to label Volume: ERR=generic_tape_device.c:726 ioctl MTBSR error on "VTLDrive01" (/dev/tape/by-path/ip-172.31.0.30:3260-iscsi-iqn.1997-05.com.amazon:sgw-2d6f8044-tapedrive-01-lun-0-nst). ERR=Input/output error.

Basicall I can not integrate Baroeos with Amazon VTL Service
(0002703)
damiano.bortolato (reporter)
2017-08-18 10:42

No fixes yet (as far as I know).
I'm wondering this is considered a "minor" bug.

- Issue History
Date Modified Username Field Change
2016-04-05 23:49 areplogle New Issue
2016-04-06 00:47 areplogle File Added: bareos-sysdig.txt
2016-04-06 00:53 areplogle Note Added: 0002228
2016-04-08 22:11 areplogle Note Added: 0002233
2016-04-15 16:17 maik Status new => acknowledged
2016-04-15 16:20 maik Priority high => low
2016-04-15 16:20 maik Severity block => minor
2016-04-24 18:43 areplogle Note Added: 0002246
2016-04-29 23:57 areplogle Note Added: 0002253
2016-05-01 15:58 areplogle Note Added: 0002254
2016-07-31 18:30 sassyn Note Added: 0002328
2016-09-07 19:23 fbacchella Note Added: 0002349
2016-09-07 19:24 fbacchella Note Added: 0002350
2016-10-27 10:39 maurizio.proietti Note Added: 0002412
2016-12-22 12:48 joergs Relationship added has duplicate 0000743
2017-04-18 11:17 damiano.bortolato Note Added: 0002628
2017-04-18 11:50 damiano.bortolato Note Edited: 0002628 View Revisions
2017-04-18 14:17 damiano.bortolato Note Edited: 0002628 View Revisions
2017-07-11 15:18 guglez Note Added: 0002681
2017-07-20 03:20 samashton Note Added: 0002688
2017-08-18 10:42 damiano.bortolato Note Added: 0002703


Copyright © 2000 - 2017 MantisBT Team
Powered by Mantis Bugtracker