View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0000297 | bareos-core | General | public | 2014-05-14 02:57 | 2015-03-25 19:18 |
Reporter | stevec | Assigned To | |||
Priority | normal | Severity | minor | Reproducibility | always |
Status | closed | Resolution | fixed | ||
Platform | Linux | OS | Ubuntu | OS Version | 12.04 |
Product Version | 13.2.3 | ||||
Summary | 0000297: hardware encryption (LTO) not creating cachefile/hangs bconsole | ||||
Description | New install of bareos 13.2.3, against quantum LTO6 drive. Able to run btape test against drive with no problems. Able to do bscrypto; create pass phrase and can manually set drive with key. When issuing commands per readme.scsicrypto to label a tape with encrypt, does not create key cache file, and seems to hang bconsole (have to ctrl-c to exit). Volume is added to pool however. bareos is running as root on this system. | ||||
Steps To Reproduce | bareos 13.2.3 compiled from source (options below). bscrypto -g <keyfile> bscrypto -k <keyfile> modified bareos-sd.conf & bareos-dir.conf with Key Encryption Key = "<pass from bscrypto -k>" with bareos started; put blank tape in drive after issuing manual umount in bconsole mt-st -f /dev/sto rewind mt-st -f /dev/st0 weof inside bconsole issue: *autodisplay on *umount storage=LTO6 Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" Connecting to Storage daemon LTO6 at loki:9103 ... 3002 Device ""LTO6" (/dev/nst0)" unmounted. *label storage=LTO6 pool=BackupSetFA volume=FA0000 encrypt Generating new hardware encryption key Connecting to Storage daemon LTO6 at loki:9103 ... Sending label command for Volume "FA0000" Slot 0 ... 3000 OK label. VolBytes=1024 Volume="FA0000" Device="LTO6" (/dev/nst0) Catalog record for Volume "FA0000", Slot 0 successfully created. Requesting to mount LTO6 ... CatReq Job=*System* GetVolInfo VolName=FA0000 write=0 ------ This is where it hangs about 30% of the time. Regardless of it hanging, no crypto cache file is created, and encryption is not enabled on tape when backing up to it. | ||||
Additional Information | --------<bareos-sd.conf>------ Storage { Name = loki-sd HeartbeatInterval=295 Maximum Concurrent Jobs = 10 Pid Directory = "/var/run" Plugin Directory = /opt/bareos/lib } Director { Name = loki-dir Password = "xxxx" Key Encryption Key = "xxxxxx" } Director { Name = loki-mon Password = "xxxxxx" Monitor = yes } Device { Name = FileStorage Media Type = File Archive Device = /tmp LabelMedia = yes; # lets Bareos label unlabeled media Random Access = Yes; AutomaticMount = yes; # when device opened, read it RemovableMedia = no; AlwaysOpen = no; } Device { Name = LTO6 Drive Crypto Enabled = Yes Query Crypto Status = Yes Changer Device = /dev/sg45 Alert Command = "sh -c 'smartctl -H -l error %c'" AlwaysOpen = yes; Archive Device = /dev/nst0 AutomaticMount = yes; Offline On Unmount = yes; Volume Poll Interval = 600; Maximum Block Size = 4194304 Maximum File Size = 50G Maximum Job Spool Size = 800G Maximum Network Buffer Size = 262144 Maximum Spool Size = 6100G LabelMedia = No Media Type = LTO6 RandomAccess = no; RemovableMedia = yes; Spool Directory = /opt/bareos/var/spool } Messages { Name = Standard director = loki-dir = all } ---------<bareos-sd.conf>-------------- ---------<bareos-dir.conf (head)>------- Director { Name = loki-dir HeartbeatInterval = 295 Maximum Concurrent Jobs = 10 Messages = Daemon Password = "xxxx" QueryFile = "/opt/bareos/etc/query.sql" Key Encryption Key = "xxxx" Plugin Directory = /opt/bareos/lib } -------->bareos-dir.conf (header portion)<------------ | ||||
Tags | No tags attached. | ||||
--- compile / configure flags set ---- ./configure --prefix=/opt/bareos --enable-scsi-crypto --enable-batch-insert --with-sqlite3 --with-openssl --with-working-dir=/opt/bareos/var/working --enable-smartalloc --enable-readline --with-logdir=/opt/bareos/var/logs |
|
What hardware ? Single drive ? If so why changer device then in config ? Please capture the output from bareos-sd -f -d 200 and bareos-dir -f -d 200 and attach this to the bug report. We have this running fine on a autochanger setup. Some drives however don't have crypto by default and need a license key (USB in some cases) and when a drive is in a library that has key management it will also not allow you to use AME (Application Managed Encryption.) |
|
This is a single drive, external. I added the changer device in the config (this is a migration from Bacula 5.2) I found the smartctl command seemed to work better with the scsi generic device for the same drive (/dev/nst0 & /dev/sg45 are the same device). The drive does support crypto as shown: bscrypto -s ../etc/lto6-0.key /dev/sg45 ./bscrypto -e /dev/sg45 Drive encryption status: Encryption Mode: Encrypt Decryption Mode: Mixed Raw Decryption Mode Disabled (RDMD): Enabled Volume Contains Encrypted Logical Blocks (VCELB): Disabled Logical Block encryption parameters: Application Managed Key Associated Data (KAD) Descriptor: Normal key So manually this does work and the drive encryption light does turn on. Seems to be more of a problem from within bareos-sd or director not doing this correctly, does not create the cryptoc cache file. I have tried this manually with tar setting the key with bscrypto and then trying to read the tape both with the key sent to the drive and without. |
|
In essence the SD plugin uses the same command as bscrypto e.g. the logic is in the shared lib so if things work with bscrypto they should also work with the SD plugin. So its probably something with the interaction of the plugin and the SD/DIR when using a single drive without a debug output I really have nothing to go on and so I cannot get an idea in what area the problem might be. |
|
Ok, with -d200 set, blank tape entered, and the following issued from bconsole: *label storage=LTO6 pool=BackupSetFA volume=FA0000 encrypt output from sd debug below. no cryptoc file is created to store the keys which I would expect in the working directory. you see it calling scsicrypto-sd.c below with a rc=0 but it doesn't appear to be doing anything? ----- loki-sd: dir_cmd.c:332-0 Conn: Hello Director loki-dir calling loki-sd: dir_cmd.c:350-0 Got a DIR connection at 15-May-2014 04:31:58 loki-sd: scsicrypto-sd.c:168-0 scsicrypto-sd: newPlugin JobId=0 loki-sd: cram-md5.c:64-0 send: auth cram-md5 <1182879435.1400146318@loki-sd> ssl=0 loki-sd: cram-md5.c:123-0 cram-get received: auth cram-md5 <1319166053.1400146318@loki-dir> ssl=0 loki-sd: cram-md5.c:142-0 sending resp to challenge: 2m+2Un+lWg/GA4E7LE+MKB loki-sd: dir_cmd.c:239-0 Message channel init completed. loki-sd: dir_cmd.c:250-0 <dird: label LTO6 VolumeName=FA0000 PoolName=BackupSetFA MediaType=LTO6 Slot=0 drive=0 loki-sd: dir_cmd.c:268-0 Do command: label loki-sd: dir_cmd.c:793-0 Found device LTO6 loki-sd: dir_cmd.c:833-0 Found device LTO6 loki-sd: dir_cmd.c:655-0 Stole device "LTO6" (/dev/nst0) lock, writing label. loki-sd: dir_cmd.c:657-0 try_autoload_device - looking for volume_info loki-sd: autochanger.c:99-0 Device "LTO6" (/dev/nst0) is not an autochanger loki-sd: dev.c:348-0 Close fd for mode change. loki-sd: dev.c:358-0 open dev: type=2 dev_name="LTO6" (/dev/nst0) vol=FA0000 mode=OPEN_READ_WRITE loki-sd: dev.c:1861-0 Enter mount loki-sd: dev.c:409-0 Open dev: device is tape loki-sd: dev.c:424-0 Try open "LTO6" (/dev/nst0) mode=OPEN_READ_WRITE loki-sd: dev.c:446-0 Rewind after open loki-sd: dev.c:2367-0 In set_os_device_parameters loki-sd: dev.c:2389-0 MTSETDRVBUFFER loki-sd: dev.c:498-0 open dev: tape 4 opened loki-sd: dev.c:371-0 preserve=0x0 fd=4 loki-sd: label.c:72-0 Enter read_volume_label res=0 device="LTO6" (/dev/nst0) vol=FA0000 dev_Vol=*NULL* loki-sd: scsicrypto-sd.c:245-0 scsicrypto-sd: handlePluginEvent event 8 loki-sd: scsicrypto-sd.c:445-0 scsicrypto-sd: Not clearing crypto key because encryption is currently not enabled on drive loki-sd: label.c:136-0 Big if statement in read_volume_label loki-sd: label.c:141-0 Requested Volume "FA0000" on "LTO6" (/dev/nst0) is not a Bareos labeled Volume, because: ERR=block.c:1015 Read zero bytes at 0:0 on device "LTO6" (/dev/nst0). loki-sd: label.c:169-0 No volume label - bailing out loki-sd: label.c:276-0 return 3 loki-sd: label.c:331-0 write_volume_label() loki-sd: label.c:351-0 New VolName=FA0000 loki-sd: label.c:360-0 Label type=0 loki-sd: scsicrypto-sd.c:245-0 scsicrypto-sd: handlePluginEvent event 10 loki-sd: scsicrypto-sd.c:445-0 scsicrypto-sd: Not clearing crypto key because encryption is currently not enabled on drive loki-sd: label.c:654-0 Start create_volume_label() loki-sd: dev.c:1701-0 Clear volhdr vol= Volume Label: Id : Bacula 1.0 immortal VerNo : 11 VolName : FA0000 PrevVolName : VolFile : 0 LabelType : PRE_LABEL LabelSize : 0 PoolName : BackupSetFA MediaType : LTO6 PoolType : Backup HostName : loki Date label written: 15-May-2014 04:31 loki-sd: label.c:644-0 Created Vol label rec: FI=PRE_LABEL len=155 loki-sd: label.c:407-0 Wrote label of 155 bytes to "LTO6" (/dev/nst0) loki-sd: label.c:410-0 Call write_block_to_dev() loki-sd: label.c:419-0 Wrote block to device loki-sd: dev.c:1502-0 === weof_dev="LTO6" (/dev/nst0) Volume Label: Id : Bacula 1.0 immortal VerNo : 11 VolName : FA0000 PrevVolName : VolFile : 1 LabelType : PRE_LABEL LabelSize : 0 PoolName : BackupSetFA MediaType : LTO6 PoolType : Backup HostName : loki Date label written: 15-May-2014 04:31 loki-sd: label.c:429-0 Call reserve_volume loki-sd: vol_mgr.c:355-0 enter reserve_volume=FA0000 drive="LTO6" (/dev/nst0) loki-sd: vol_mgr.c:264-0 new Vol=FA0000 at 7f07d4003958 dev="LTO6" (/dev/nst0) loki-sd: vol_mgr.c:479-0 === set in_use. vol=FA0000 dev="LTO6" (/dev/nst0) loki-sd: vol_mgr.c:513-0 Inc walk_start use_count=2 volname=FA0000 loki-sd: vol_mgr.c:200-0 List end new volume: FA0000 in_use=1 swap=0 on device "LTO6" (/dev/nst0) loki-sd: vol_mgr.c:622-0 === clear in_use vol=FA0000 loki-sd: vol_mgr.c:638-0 === set not reserved vol=FA0000 num_writers=0 dev_reserved=0 dev="LTO6" (/dev/nst0) loki-sd: dir_cmd.c:250-0 <dird: mount LTO6 drive=0 loki-sd: dir_cmd.c:268-0 Do command: mount loki-sd: dir_cmd.c:858-0 ok=1 drive=0 slot=0 loki-sd: dir_cmd.c:793-0 Found device LTO6 loki-sd: dir_cmd.c:833-0 Found device LTO6 loki-sd: dir_cmd.c:865-0 mount cmd blocked=0 must_unload=0 loki-sd: dir_cmd.c:931-0 Not blocked changer=0 slot=0 loki-sd: scsicrypto-sd.c:218-0 scsicrypto-sd: freePlugin JobId=0 loki-sd: job.c:455-0 Start stored free_jcr loki-sd: job.c:567-0 End stored free_jcr --------------- |
|
Ok nothing special here, keep in mind that the label is not encrypted so things looks just fine. When you write the first Job to that tape it will request the key from the director and the fill the cache. When there is nothing written to a tape there is no need to have any key in the cache. Also keep in mind that the cache is only used when the SD cannot ask the DIR for the key. The reason the label is not encrypted has a valid reason as now we can at least determine that we are dealing with a Bareos tape. If you also encrypt the label you cannot determine a empty tape from a tape for which you have the wrong key and you could relabel a valid tape just because you have the wrong key. Also the label contains no data other then the meta information that its a Bareos tape. So first do a backup to the volume and then capture the output from the SD. Also please attach the logs as attachments to the bug and not as Note. |
|
Thanks, skipped over the file attach screen. Ok, attached debug output from all three and caught one of the problems which is the 'hang' at the console prompt when trying to label a tape with encrypt. I had FA0000 in the drive, did a umount against that, put in a new tape, did: mt-st -f /dev/st0 rewind mt-st -f /dev/st0 weof then in bconsole issued: label storage=LTO6 pool=BackupSetFA volume=FA0001 encrypt This then generated the following but hung bconsole (never got a prompt back) Generating new hardware encryption key Connecting to Storage daemon LTO6 at loki:9103 ... Sending label command for Volume "FA0001" Slot 0 ... 3000 OK label. VolBytes=1024 Volume="FA0001" Device="LTO6" (/dev/nst0) Catalog record for Volume "FA0001", Slot 0 successfully created. Requesting to mount LTO6 ... CatReq Job=*System* GetVolInfo VolName=FA0001 write=0 ----- When I kill bconsole and try to get back in to do anything with the storage daemon all commands hang like it has lost communication to the storage daemon. |
|
Ok clear the last line printed in the DIR shows a protocol message that should be answered by the DIR and not be printed to the user. That explains why it hangs. The fix is obvious use bget_dirmsg() instead of bnet_recv() that will see the protocol message and send the catalog info requested. This is needed as the catalog info contains the encryption key. The special case here is the automount something that is normally disabled on autochangers. |
|
0001-Fix-hardware-encryption-LTO-not-creating-cachefile-h.patch (1,895 bytes)
From 2638e0adde6c9e98334b404e5dd5ab391f552a03 Mon Sep 17 00:00:00 2001 From: Marco van Wieringen <marco.van.wieringen@bareos.com> Date: Mon, 19 May 2014 09:33:54 +0200 Subject: [PATCH] Fix hardware encryption (LTO) not creating cachefile/hangs bconsole Use bget_dirmsg() instead of bnet_recv() when automounting volume because as part of the mount request the stored can request catalog information for any plugin who listens to the bsdEventLabelVerified event. Fixes #297: hardware encryption (LTO) not creating cachefile/hangs bconsole --- src/dird/ua_label.c | 15 ++++++++++++++- 1 files changed, 14 insertions(+), 1 deletions(-) diff --git a/src/dird/ua_label.c b/src/dird/ua_label.c index c968e20..0f2326b 100644 --- a/src/dird/ua_label.c +++ b/src/dird/ua_label.c @@ -301,8 +301,19 @@ checkName: bash_spaces(dev_name); sd->fsend("mount %s drive=%d", dev_name, drive); unbash_spaces(dev_name); - while (bnet_recv(sd) >= 0) { + + /* + * We use bget_dirmsg here and not bnet_recv because as part of + * the mount request the stored can request catalog information for + * any plugin who listens to the bsdEventLabelVerified event. + * As we don't want to loose any non protocol data e.g. errors + * without a 3xxx prefix we set the allow_any_message of + * bget_dirmsg to true and as such is behaves like a normal + * bnet_recv for any non protocol messages. + */ + while (bget_dirmsg(sd, true) >= 0) { ua->send_msg("%s", sd->msg); + /* * Here we can get * 3001 OK mount. Device=xxx or @@ -319,9 +330,11 @@ checkName: } } } + if (print_reminder) { ua->info_msg(_("Do not forget to mount the drive!!!\n")); } + close_sd_bsock(ua); return 1; -- 1.7.3.2 |
|
Please apply the attached patch to your director (patch -p1 < 0001*) when you are in the toplevel of the source tree and compile a new director and test with that. This patch was generated against master but it also cleanly applies to bareos-13.2 and probably also against bareos-12.4 If this patch solves the problem (which I think it will) I will apply it to master and then in a couple of days to the other branches. |
|
Ok, just applied and tested the patch. This works, when labeling tapes now the director does not hang; the crytoc cache file is created and populated, and when writing the tapes I see that storage director/drive is getting keys. Thanks. |
|
Fix committed to bareos master branch with changesetid 1779. | |
Fix committed to bareos bareos-13.2 branch with changesetid 1791. | |
Fix committed to bareos bareos-12.4 branch with changesetid 1788. | |
Fix committed to bareos playground branch with changesetid 1798. | |
Fix committed to bareos2015 bareos-14.2 branch with changesetid 4779. | |
Due to the reimport of the Github repository to bugs.bareos.org, the status of some tickets have been changed. These tickets will be closed again. Sorry for the noise. |
|
bareos: master 421ddd09 2014-05-19 09:33 Ported: N/A Details Diff |
Fix hardware encryption (LTO) not creating cachefile/hangs bconsole Use bget_dirmsg() instead of bnet_recv() when automounting volume because as part of the mount request the stored can request catalog information for any plugin who listens to the bsdEventLabelVerified event. Fixes 0000297: hardware encryption (LTO) not creating cachefile/hangs bconsole |
Affected Issues 0000297 |
|
mod - src/dird/ua_label.c | Diff File | ||
bareos: bareos-12.4 9e32beab 2014-05-19 09:33 Ported: N/A Details Diff |
Fix hardware encryption (LTO) not creating cachefile/hangs bconsole Use bget_dirmsg() instead of bnet_recv() when automounting volume because as part of the mount request the stored can request catalog information for any plugin who listens to the bsdEventLabelVerified event. Fixes 0000297: hardware encryption (LTO) not creating cachefile/hangs bconsole |
Affected Issues 0000297 |
|
mod - src/dird/ua_label.c | Diff File | ||
bareos: bareos-13.2 83821c33 2014-05-19 09:33 Ported: N/A Details Diff |
Fix hardware encryption (LTO) not creating cachefile/hangs bconsole Use bget_dirmsg() instead of bnet_recv() when automounting volume because as part of the mount request the stored can request catalog information for any plugin who listens to the bsdEventLabelVerified event. Fixes 0000297: hardware encryption (LTO) not creating cachefile/hangs bconsole |
Affected Issues 0000297 |
|
mod - src/dird/ua_label.c | Diff File | ||
bareos: playground 89cf18f9 2014-05-19 09:33 Ported: N/A Details Diff |
Fix hardware encryption (LTO) not creating cachefile/hangs bconsole Use bget_dirmsg() instead of bnet_recv() when automounting volume because as part of the mount request the stored can request catalog information for any plugin who listens to the bsdEventLabelVerified event. Fixes 0000297: hardware encryption (LTO) not creating cachefile/hangs bconsole |
Affected Issues 0000297 |
|
mod - src/dird/ua_label.c | Diff File | ||
bareos2015: bareos-12.4 39f6398e 2014-05-19 11:33 Ported: N/A Details Diff |
Fix hardware encryption (LTO) not creating cachefile/hangs bconsole Use bget_dirmsg() instead of bnet_recv() when automounting volume because as part of the mount request the stored can request catalog information for any plugin who listens to the bsdEventLabelVerified event. Fixes 0000297: hardware encryption (LTO) not creating cachefile/hangs bconsole |
Affected Issues 0000297 |
|
mod - src/dird/ua_label.c | Diff File | ||
bareos2015: bareos-13.2 6ef64602 2014-05-19 11:33 Ported: N/A Details Diff |
Fix hardware encryption (LTO) not creating cachefile/hangs bconsole Use bget_dirmsg() instead of bnet_recv() when automounting volume because as part of the mount request the stored can request catalog information for any plugin who listens to the bsdEventLabelVerified event. Fixes 0000297: hardware encryption (LTO) not creating cachefile/hangs bconsole |
Affected Issues 0000297 |
|
mod - src/dird/ua_label.c | Diff File | ||
bareos2015: bareos-14.2 03bb94a3 2014-05-19 11:33 Ported: N/A Details Diff |
Fix hardware encryption (LTO) not creating cachefile/hangs bconsole Use bget_dirmsg() instead of bnet_recv() when automounting volume because as part of the mount request the stored can request catalog information for any plugin who listens to the bsdEventLabelVerified event. Fixes 0000297: hardware encryption (LTO) not creating cachefile/hangs bconsole |
Affected Issues 0000297 |
|
mod - src/dird/ua_label.c | Diff File |
Date Modified | Username | Field | Change |
---|---|---|---|
2014-05-14 02:57 | stevec | New Issue | |
2014-05-14 03:02 | stevec | Note Added: 0000857 | |
2014-05-14 20:04 | mvwieringen | Note Added: 0000858 | |
2014-05-14 20:04 | mvwieringen | Status | new => feedback |
2014-05-14 20:04 | mvwieringen | Description Updated | |
2014-05-14 20:04 | mvwieringen | Steps to Reproduce Updated | |
2014-05-14 20:04 | mvwieringen | Additional Information Updated | |
2014-05-14 23:42 | stevec | Note Added: 0000859 | |
2014-05-14 23:42 | stevec | Status | feedback => new |
2014-05-15 08:58 | mvwieringen | Note Added: 0000860 | |
2014-05-15 08:58 | mvwieringen | Assigned To | => mvwieringen |
2014-05-15 08:58 | mvwieringen | Status | new => feedback |
2014-05-15 11:39 | stevec | Note Added: 0000861 | |
2014-05-15 11:39 | stevec | Status | feedback => assigned |
2014-05-15 14:18 | mvwieringen | Note Added: 0000862 | |
2014-05-15 14:18 | mvwieringen | Status | assigned => feedback |
2014-05-18 12:27 | stevec | File Added: bareos-debug.zip | |
2014-05-18 12:29 | stevec | Note Added: 0000869 | |
2014-05-18 12:29 | stevec | Status | feedback => assigned |
2014-05-18 12:44 | stevec | Note Edited: 0000869 | |
2014-05-19 09:45 | mvwieringen | Note Added: 0000870 | |
2014-05-19 09:45 | mvwieringen | File Added: 0001-Fix-hardware-encryption-LTO-not-creating-cachefile-h.patch | |
2014-05-19 09:47 | mvwieringen | Note Added: 0000871 | |
2014-05-19 09:47 | mvwieringen | Status | assigned => feedback |
2014-05-20 12:50 | stevec | Note Added: 0000875 | |
2014-05-20 12:50 | stevec | Status | feedback => assigned |
2014-05-20 13:58 | mvwieringen | Changeset attached | => bareos master 421ddd09 |
2014-05-20 13:58 | mvwieringen | Note Added: 0000876 | |
2014-05-20 13:58 | mvwieringen | Status | assigned => resolved |
2014-05-20 13:58 | mvwieringen | Resolution | open => fixed |
2014-05-23 14:59 | mvwieringen | Changeset attached | => bareos bareos-13.2 83821c33 |
2014-05-23 14:59 | mvwieringen | Note Added: 0000878 | |
2014-05-23 15:00 | mvwieringen | Changeset attached | => bareos bareos-12.4 9e32beab |
2014-05-23 15:00 | mvwieringen | Note Added: 0000880 | |
2014-05-26 12:16 | mvwieringen | Changeset attached | => bareos playground 89cf18f9 |
2014-05-26 12:16 | mvwieringen | Note Added: 0000882 | |
2014-06-06 15:59 | mvwieringen | Status | resolved => closed |
2014-06-06 15:59 | mvwieringen | Assigned To | mvwieringen => |
2015-03-25 16:51 | mvwieringen | Changeset attached | => bareos2015 bareos-12.4 39f6398e |
2015-03-25 16:51 | mvwieringen | Changeset attached | => bareos2015 bareos-13.2 6ef64602 |
2015-03-25 16:51 | mvwieringen | Changeset attached | => bareos2015 bareos-14.2 03bb94a3 |
2015-03-25 16:51 | mvwieringen | Note Added: 0001349 | |
2015-03-25 16:51 | mvwieringen | Status | closed => resolved |
2015-03-25 19:18 | joergs | Note Added: 0001505 | |
2015-03-25 19:18 | joergs | Status | resolved => closed |