 |
Think for starters we need a debug output level 200 of the storage daemon to see what the plugin is doing and what not and why. e.g. bareos-sd -f -d 200 > logfile 2>&1 run as user bareos via su - bareos -s /bin/bash And probably the exact config of the tape device in the bareos-sd.conf |
|
|
|
|
|
|
|
|
As requested I attached the desired files (for security reasons we removed keys). |
|
|
Ok log shows that the plugin thinks it doesn't need to clear the encryption key because no encryption is enabled on the drive. Given you have the following in your config: Query Crypto Status = yes; It means that the plugin queries the actual encryption status of the drive in determining ic crypto is enabled. You could change Query Crypto Status = yes; to Query Crypto Status = no; then it doesn't query the drive but in our testing that always worked ok. I would be interested in the output of status storage=<storage_name> when the volume is loaded as that prints the encryption status of the drive and uses the same low level SCSI page so we can see what the drive encryption status is. |
|
|
20150317-bareos-status-storage.txt (2,917 bytes) |
|
|
I attached the output of status storage=TapeLTO5. The tape was still in the drive. I'm not sure what you want to see. But due the fact we set AlwaysOpen=no I assume that you do not see what you expected. I tried thereafter a "mount" and then the output looks basically like the one already attached except the end with the info about Used Volume status: ---- 8< -------- 8< -------- 8< -------- 8< -------- 8< ---- Used Volume status: WeeklyData-LTO5-0001 on device "TapeStorageLTO5" (/dev/nst0) Reader=0 writers=0 reserves=0 volinuse=0 ==== ==== ---- 8< -------- 8< -------- 8< -------- 8< -------- 8< ---- Even the change of the mentioned flag (Query Crypto Status to "no") does not mean that the issue is solved, right? |
|
|
I was somewhat lured into thinking that the plugin didn't want to clear the key but it seems there are multiple backups in the debug log so the real info somehow got lost. Its seems this is what happens: fs0-sd: scsicrypto-sd.c:401-0 scsicrypto-sd: Loading new crypto key <key> ... fs0-sd: block.c:591-0 ===== write retry=1 status=-1 errno=16: ERR=Device or resource busy fs0-sd: block.c:591-0 ===== write retry=2 status=-1 errno=16: ERR=Device or resource busy fs0-sd: block.c:591-0 ===== write retry=3 status=-1 errno=16: ERR=Device or resource busy ... fs0-sd: scsicrypto-sd.c:457-0 scsicrypto-sd: Clearing crypto key So the plugin is setting the key and clearing it but after setting it the drive seems to go into some weird state where it doesn't want things to be written to it. Normally that should not happen even when you changed keys over time it will just mean you cannot read older data on the tape because of changing the key. But in this case it puts the drive into some weird mode. Maybe it makes more sense to test the encryption of the drive outside of the storage daemon using bscrypto (which essentially uses the same low level routines) on a test tape. e.g. disable the bareos-sd and insert a test tape into the drive. # bscrypto -s - /dev/sg? Type Some key^D # bscrypto -e /dev/sg? # tar cvf /dev/nst0 /etc # bscrypto -c /dev/sg? First sets a key, second gets the drive encryption status. Then we write some data using tar last clears the key again. That should at least show how the drive reacts to the crypto operations. |
|
|
I tried the thing you suggested: # bscrypto -s - /dev/sg5 Type Some key^D # bscrypto -e /dev/sg5 Drive encryption status: Encryption Mode: Encrypt Decryption Mode: Mixed Raw Decryption Mode Disabled (RDMD): Enabled Volume Contains Encrypted Logical Blocks (VCELB): Enabled Logical Block encryption parameters: Application Managed Key Associated Data (KAD) Descriptor: Normal key # tar cvf /dev/nst0 /etc # bscrypto -c /dev/sg5 After this reading from the tape was not possible -> OK because key was missing. Then I setup the key again and tried to reread the tape. Worked -> OK I cleared the key and ejected the tape and mounted it again. After setting up the key again I was able to read the backup from tape. -> OK So in general it seems to work. Could it be that the positioning operation on the tape could produce the error? Background: we want to make three backups on the same tape. I got a similar error in the case I forgot to rewind the tape to the beginning when I tried to do a restore/test with tar. |
|
|
|
|
|
I just uploaded a new log file of the storage daemon. From the logs it looks for me that bareos is not able to append to the tape if there is already a backup on it. As you can see in the logs it mounts the tape with the encryption key. Then it tries to write the "session_label". I do not understand the logs; it mentions that the writing of the label seems to be ok ("Leave" log line) but then the following block writing operation fails: This operation fails according to the logs with the lines "retry=[1..3] status=-1 errno=16: ERR=Device or resource busy". After some tries it give up on this tape and want a new one (volume). I'm not sure about this block write error if it belongs to the labeling or already data will be written. Is there a way to confirm this issue somehow? Could we add some debug code to get more debug information? |
|
|
The new log shows exactly the same as before. After setting the crypto key the drive gets inaccessible for any further write. Some things that are strange in your sd config: - Two storage entries accessing the same physical drive (May work but not really within the design envelope) Maybe change the media type to LTO (instead of LTO4/LTO5) - Always open to no on tape No idea why you set this but for tapes that is most of the time not the best option. e.g. is keeps also rewinding tapes this way to re-read the tape label. B.T.W. the tape label gets reread but it doesn't get updated after its written once to the tape. We also made the actual tape label unencrypted as then we can determine a encrypted tape vs a empty tape without a label. Have you tried using the default blocksize on the tape instead of this big blocksize ? Any messages in the unix messages given that the device gives an EBUSY I wonder if the ST driver logs something. |
|
|
* Two storage entries: As I understood this is to specify which media the drive supports. Unfortunatly a LTO6 drive does not automatically support all older LTO tapes; as an example LTO4 is not supported on such drives. I'm not aware that different media types could be specifed in "Media Type" so this was for me the "cleanest" solution. To be sure that this config section does not interferer with "our" problem I changed the device to /dev/nst1 * Always Open: We set this option to NO because we anyway eject the tape after a backup operation to bring it to a different location (keyword BCM). So when we set this option to YES we just have to unmount it after a backup job by hand - anyway as soon as the tape gets ejected it was rewound before. So in my opinion I do not see any benefit of setting this value to YES. Yes, we tried also with the default block size with the same error. Unfortunatly the ST driver does not log anything - is there a special option to force the ST driver to log something? |
|
|
You could try if setting always open to yes and release (e.g. not unmount) the device after the backup to see if that changes anything. Given the exact same setting using bscrypto work it seems the drive is somewhat sick as it locks itself with device busy after loading a key. Never seen that on any piece of other hardware so I'm out of ideas. Given the ST driver also doesn't log anything only option left if trying with always open = yes. If that also doesn't bring anything then I guess we are out of options for a non subscription bug report. |
|
|
As it turned out the problem was the setting Maximum Blocksize = 1048576; We removed it so it reverted to default and as it looks now it works like expected (as mentioned before). In my opinion the documentation should be adjusted because we followed the section to optimize the tape speed which suggests to adjust this parameter according to the defined procedure in the manual. Another side note: You must be careful with mounting of the tape; in our experience the encryption was only active if we mounted the tape manual before a scheduled task wrote data on it. From my point of view this bug can be closed. |
|
|
Given that each vendor implements its hardware different it can happen that it doesn't support hardware encryption with bigger blocksizes that was also one of the reasons that I advised earlier to test with the default blocksize. As to mounting the tape every trace I saw until now showed things worked fine as to loading the key and the right key was cleared or loaded when needed. The key gets loaded when the tape label is verified by an event that is then triggered and that is true for any invocation of a backup e.g. mounted volume or not. I will close the ticket and ask the original author of the tape tuning document to add a section on the EBUSY error the Linux ST driver seems to issue when it has buffering problems with bigger blocksizes. |
- Anonymous
