View Issue Details

IDProjectCategoryView StatusLast Update
0001341bareos-core[All Projects] Generalpublic2021-04-26 15:14
ReporterdigidaxAssigned To 
PriorityimmediateSeveritymajorReproducibilityalways
Status newResolutionopen 
Platformx86_64OSCentOSOS Version8.4
Product Version20.0.1 
Fixed in Version 
Summary0001341: LTO-7 tape lost capacity during recycling
DescriptionMade some backup / restore testst with some LTO-7 Ultrium Tape in a Quantum LTO-7 HH drive
Want to recycle the tapes for use it now in productive environment
After 1,6 TB, beros say: End of Media (6 TB are expected)
recycled manually the tape again and start backup:
Now, after 1,1 TB bareos say: End of Media
=> with each reycling, the capacity decreases
Steps To Reproducein beconsole, deleted the volumes which I want to cecycle
checked in GUI => the voulumes are not shown in the pool, no volumes in the pool

Mount the tape into drive an then:

[root@bareos ~]# mt -f /dev/st0 rewind
[root@bareos ~]# mt -f /dev/st0 weof
[root@bareos ~]# mt -f /dev/st0 rewind
[root@bareos ~]# mt -f /dev/st0 status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x5c (no translation).
Soft error count since last status=0
General status bits on (41010000):
 BOT ONLINE IM_REP_EN

In bconsole, labeling and add to the pool:

Sending label command for Volume "Audio-001" Slot 0 ...
3000 OK label. VolBytes=64512 Volume="Audio-001" Device="LTO7-Tape-Drive" (/dev/nst0)
Catalog record for Volume "Audio-001", Slot 0 successfully created.
Requesting to mount LTO7-Tape-Drive ...
3001 Mounted Volume: Audio-001
3001 Device "LTO7-Tape-Drive" (/dev/nst0) is already mounted with Volume "Audio-001"

check the tape:
# bls -V Audio-001 /dev/nst0
bls: stored/butil.cc:291-0 Using device: "/dev/nst0" for reading.
22-Apr 08:53 bls JobId 0: Ready to read from volume "Audio-001" on device "LTO7-Tape-Drive" (/dev/nst0).
22-Apr 08:53 bls JobId 0: End of Volume at file 1 on device "LTO7-Tape-Drive" (/dev/nst0), Volume "Audio-001"
22-Apr 08:53 bls JobId 0: End of all volumes.
0 files and directories found.
22-Apr 08:53 bls JobId 0: Releasing device "LTO7-Tape-Drive" (/dev/nst0).

# btape /dev/nst0
*cap
Configured device capabilities:
EOF BSR BSF FSR FSF FASTFSF !BSFATEOM EOM REM !RACCESS AUTOMOUNT !LABEL !ANONVOLS !ALWAYSOPEN MTIOCGET
Device status:
OPENED TAPE !LABEL !MALLOC APPEND !READ !EOT !WEOT !EOF !NEXTVOL !SHORT
Device parameters:
Device name: /dev/nst0
File=0 block=0
Min block=0 Max block=524288
Status:
 Bareos status: file=0 block=0
 Device status: BOT ONLINE IM_REP_EN file=0 block=0
Device status: TAPE BOT ONLINE IMMREPORT. ERR=

*eod
btape: stored/btape.cc:620-0 Moved to end of medium.

*cap
Configured device capabilities:
EOF BSR BSF FSR FSF FASTFSF !BSFATEOM EOM REM !RACCESS AUTOMOUNT !LABEL !ANONVOLS !ALWAYSOPEN MTIOCGET
Device status:
OPENED TAPE !LABEL !MALLOC APPEND !READ !EOT !WEOT EOF !NEXTVOL !SHORT
Device parameters:
Device name: /dev/nst0
File=1 block=0
Min block=0 Max block=524288
Status:
 EOF Bareos status: file=1 block=0
 Device status: EOD ONLINE IM_REP_EN file=1 block=-1
Device status: TAPE EOF EOD ONLINE IMMREPORT. ERR=

*readlabel
btape: stored/btape.cc:528-0 Volume label read correctly.

Volume Label:
Id : Bareos 2.0 immortal
VerNo : 20
VolName : Audio-001
PrevVolName :
VolFile : 0
LabelType : PRE_LABEL
LabelSize : 183
PoolName : AudioBackup
MediaType : LTO-7
PoolType : Backup
HostName : bareos.123.com
Date label written: 22-Apr-2021 08:50

Starting a 629 GB backup job:

22-Apr 14:05 bareos-dir JobId 246: Start Backup JobId 246, Job=BackupAudio.2021-04-22_14.05.44_15
22-Apr 14:05 bareos-dir JobId 246: Connected Storage daemon at 192.168.130.211:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
22-Apr 14:05 bareos-dir JobId 246: Using Device "LTO7-Tape-Drive" to write.
22-Apr 14:05 bareos-dir JobId 246: Probing client protocol... (result will be saved until config reload)
22-Apr 14:05 bareos-dir JobId 246: Connected Client: bareos-fd at localhost:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
22-Apr 14:05 bareos-dir JobId 246: Handshake: Immediate TLS 22-Apr 14:05 bareos-dir JobId 246: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
22-Apr 14:05 localhost-fd JobId 246: Connected Storage daemon at 192.168.130.211:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
22-Apr 14:05 localhost-fd JobId 246: Extended attribute support is enabled
22-Apr 14:05 localhost-fd JobId 246: ACL support is enabled
22-Apr 14:05 bareos-sd JobId 246: Wrote label to prelabeled Volume "Audio-001" on device "LTO7-Tape-Drive" (/dev/nst0)
22-Apr 14:05 bareos-sd JobId 246: Spooling data ...
22-Apr 15:36 bareos-sd JobId 246: Committing spooled data to Volume "Audio-001". Despooling 629,102,376,675 bytes ...
22-Apr 17:01 bareos-sd JobId 246: Error: stored/block.cc:803 Write error at 29:11710 on device "LTO7-Tape-Drive" (/dev/nst0). ERR=Input/output error.
22-Apr 17:01 bareos-sd JobId 246: Error: stored/block.cc:822 Write error on fd=6 at file:blk 29:11710 on device "LTO7-Tape-Drive" (/dev/nst0). ERR=Input/output error.
22-Apr 17:01 bareos-sd JobId 246: Error: Error writing final EOF to tape. This Volume may not be readable.
backends/generic_tape_device.cc:403 ioctl MTWEOF error on "LTO7-Tape-Drive" (/dev/nst0). ERR=Input/output error.
22-Apr 17:01 bareos-sd JobId 246: End of medium on Volume "Audio-001" Bytes=586,124,688,384 Blocks=1,117,944 at 22-Apr-2021 17:01.
22-Apr 17:01 bareos-sd JobId 246: Job BackupAudio.2021-04-22_14.05.44_15 is waiting. Cannot find any appendable volumes.
TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-19.2: impact
bareos-19.2: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action

Activities

digidax

digidax

2021-04-24 09:06

reporter   ~0004117

2nd try:
in bconsole, deleted the volume "Audio-001"
Result: successful

then erasing the tape:

[root@bareos ~]# mt -f /dev/st0 rewind
[root@bareos ~]# mt -f /dev/st0 status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x5c (no translation).
Soft error count since last status=0
General status bits on (41010000):
 BOT ONLINE IM_REP_EN

[root@bareos ~]# mt -f /dev/st0 erase

After around 21 hours the erasing procedure was finished. Then:

[root@bareos ~]# mt -f /dev/st0 rewind
[root@bareos ~]# mt -f /dev/st0 status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x5c (no translation).
Soft error count since last status=0
General status bits on (41010000):
 BOT ONLINE IM_REP_EN

in bconsole label the tape and add it to the pool.
Then start backup:

22-Apr 14:05 bareos-dir JobId 246: Start Backup JobId 246, Job=BackupAudio.2021-04-22_14.05.44_15
22-Apr 14:05 bareos-dir JobId 246: Connected Storage daemon at 192.168.130.211:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
22-Apr 14:05 bareos-dir JobId 246: Using Device "LTO7-Tape-Drive" to write.
22-Apr 14:05 bareos-dir JobId 246: Probing client protocol... (result will be saved until config reload)
22-Apr 14:05 bareos-dir JobId 246: Connected Client: bareos-fd at localhost:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
22-Apr 14:05 bareos-dir JobId 246: Handshake: Immediate TLS 22-Apr 14:05 bareos-dir JobId 246: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
22-Apr 14:05 localhost-fd JobId 246: Connected Storage daemon at 192.168.130.211:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
22-Apr 14:05 localhost-fd JobId 246: Extended attribute support is enabled
22-Apr 14:05 localhost-fd JobId 246: ACL support is enabled
22-Apr 14:05 bareos-sd JobId 246: Wrote label to prelabeled Volume "Audio-001" on device "LTO7-Tape-Drive" (/dev/nst0)
22-Apr 14:05 bareos-sd JobId 246: Spooling data ...
22-Apr 15:36 bareos-sd JobId 246: Committing spooled data to Volume "Audio-001". Despooling 629,102,376,675 bytes ...
22-Apr 17:01 bareos-sd JobId 246: Error: stored/block.cc:803 Write error at 29:11710 on device "LTO7-Tape-Drive" (/dev/nst0). ERR=Input/output error.
22-Apr 17:01 bareos-sd JobId 246: Error: stored/block.cc:822 Write error on fd=6 at file:blk 29:11710 on device "LTO7-Tape-Drive" (/dev/nst0). ERR=Input/output error.
22-Apr 17:01 bareos-sd JobId 246: Error: Error writing final EOF to tape. This Volume may not be readable.
backends/generic_tape_device.cc:403 ioctl MTWEOF error on "LTO7-Tape-Drive" (/dev/nst0). ERR=Input/output error.
22-Apr 17:01 bareos-sd JobId 246: End of medium on Volume "Audio-001" Bytes=586,124,688,384 Blocks=1,117,944 at 22-Apr-2021 17:01.
22-Apr 17:01 bareos-sd JobId 246: Job BackupAudio.2021-04-22_14.05.44_15 is waiting. Cannot find any appendable volumes.
Please use the "label" command to create a new Volume for:

Pool: AudioBackup
+---------+------------+-----------+---------+-------------------+----------+--------------+---------+------+-----------+-----------+---------------------+---------+
| mediaid | volumename | volstatus | enabled | volbytes | volfiles | volretention | recycle | slot | inchanger | mediatype | lastwritten | storage |
+---------+------------+-----------+---------+-------------------+----------+--------------+---------+------+-----------+-----------+---------------------+---------+
| 44 | Audio-001 | Full | 1 | 483,611,704,320 | 24 | 946,080,000 | 0 | 0 | 0 | LTO-7 | 2021-04-23 21:26:37 | Tape |
+---------+------------+-----------+---------+-------------------+----------+--------------+---------+------+-----------+-----------+---------------------+---------+
 
Bareos set a LTO-7 tape (6 TB capacity) full with 630 GB.
digidax

digidax

2021-04-24 09:33

reporter   ~0004118

thats the pool definition:

Pool {
  Name = AudioBackup
  Pool Type = Backup
  Recycle = no
  AutoPrune = no
  Volume Retention = 30 years
  Maximum Volumes = 100
  Maximum Volume Jobs = 0
}
digidax

digidax

2021-04-26 08:13

reporter   ~0004120

st 9:0:0:0: [st0] Add. Sense: Warning - specified temperature exceeded

gaves me the hint: After checking the server and watching the fans with a endoscopic camera and closed cover inside the rack:

After starting the test, the speed of the fans goes up, but with increasing of the temperature, they lowered the speed instead of spooling up more.
Checking the the controller shows some damaged capacitors. After replacing the caps and simulating hot air with a hairdryer, the speed of the fans is now linear to the increasing temperature. The btape "fill" test is now running well and stable with a thruput of around 280 MB/s. After finishing the test, I will do a backup.
digidax

digidax

2021-04-26 15:14

reporter   ~0004122

testting now with backup:
Despooling elapsed time = 00:39:31, Transfer rate = 265.3 M Bytes/second

backup successful.

feature request: monitoring tape drive health like temperature ...

Issue History

Date Modified Username Field Change
2021-04-23 07:14 digidax New Issue
2021-04-24 09:06 digidax Note Added: 0004117
2021-04-24 09:33 digidax Note Added: 0004118
2021-04-26 08:13 digidax Note Added: 0004120
2021-04-26 15:14 digidax Note Added: 0004122