View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001271bareos-corestorage daemonpublic2020-09-08 23:162021-09-28 15:27
Reporterkabassanov Assigned Tobruno-at-bareos  
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionunable to reproduce 
PlatformLinuxOSDebianOS Version10
Product Version19.2.8 
Summary0001271: Btape speed tests with Bareos block structure have half of the Bacula throughput
DescriptionHello,

Some months ago I migrated a backup server from Bacula to Bareos.
I've just remarked that with btape tests (and real backups) write throughtput is decreased by about 50% when bareos block structure is used.
So I wonder if there is something wrong there... Like a bareos block structure that overlaps and needs 2 blocks while sending data to the drive?

Bacula:
----------

btape: btape.c:1062-0 Test with zero data, should give the maximum throughput.
btape: btape.c:911-0 Begin writing 3 files of 3.221 GB with raw blocks of 1048576 bytes.
btape: btape.c:385-0 Total Volume bytes=9.663 GB. Total Write rate = 508.6 MB/s

+++

btape: btape.c:1074-0 Test with random data, should give the minimum throughput.
btape: btape.c:911-0 Begin writing 3 files of 3.221 GB with raw blocks of 1048576 bytes.
btape: btape.c:385-0 Total Volume bytes=9.663 GB. Total Write rate = 214.7 MB/s

+++

btape: btape.c:1088-0 Test with zero data and bacula block structure.
btape: btape.c:966-0 Begin writing 3 files of 3.221 GB with blocks of 1048576 bytes.
btape: btape.c:385-0 Total Volume bytes=9.663 GB. Total Write rate = 536.8 MB/s

btape: btape.c:1100-0 Test with random data, should give the minimum throughput.
btape: btape.c:966-0 Begin writing 3 files of 3.221 GB with blocks of 1048576 bytes.
btape: btape.c:385-0 Total Volume bytes=9.663 GB. Total Write rate = 214.7 MB/s

Bareos:

btape: stored/btape.cc:1026-0 Test with zero data, should give the maximum throughput.
btape: stored/btape.cc:887-0 Begin writing 3 files of 3.221 GB with raw blocks of 1048576 bytes.
btape: stored/btape.cc:396-0 Total Volume bytes=9.663 GB. Total Write rate = 536.8 MB/s

+++

btape: stored/btape.cc:1038-0 Test with random data, should give the minimum throughput.
btape: stored/btape.cc:887-0 Begin writing 3 files of 3.221 GB with raw blocks of 1048576 bytes.
btape: stored/btape.cc:396-0 Total Volume bytes=9.663 GB. Total Write rate = 214.7 MB/s

+++

btape: stored/btape.cc:1052-0 Test with zero data and bareos block structure.
btape: stored/btape.cc:942-0 Begin writing 3 files of 3.221 GB with blocks of 1048576 bytes.
btape: stored/btape.cc:396-0 Total Volume bytes=9.663 GB. Total Write rate = 292.8 MB/s

+++

btape: stored/btape.cc:1064-0 Test with random data, should give the minimum throughput.
btape: stored/btape.cc:942-0 Begin writing 3 files of 3.221 GB with blocks of 1048576 bytes.
btape: stored/btape.cc:396-0 Total Volume bytes=9.663 GB. Total Write rate = 108.5 MB/s
TagsNo tags attached.

Activities

bruno-at-bareos

bruno-at-bareos

2021-09-07 09:27

manager   ~0004251

Hello, thanks for the report, would you mind to give a bit more information ?

bacula version used and edition
Manufacturer name of tape
output of sudo lshw -json
output of uname -a

Regards
kabassanov

kabassanov

2021-09-11 17:14

reporter   ~0004262

Hi,

Bacula 9.4.2 (04Feb19)
Tapes: Fujifilm LTO Ultrium 7 - 6 TB /15 TB
Attached LTO related part of lshw. If more needed, please contact me in private.
Linux "hostname" 5.10.0-1-amd64 0000001 SMP Debian 5.10.4-1 (2020-12-31) x86_64 GNU/Linux (probably kernel has been updated since bacula btape tests in August 2019...)

Regards
lshw-lto-related.txt (5,167 bytes)    
        {
          "id" : "pci:3",
          "class" : "bridge",
          "claimed" : true,
          "handle" : "PCIBUS:0000:83",
          "description" : "PCI bridge",
          "product" : "Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2",
          "vendor" : "Intel Corporation",
          "physid" : "2",
          "businfo" : "pci@0000:80:02.0",
          "version" : "01",
          "width" : 32,
          "clock" : 33000000,
          "configuration" : {
            "driver" : "pcieport"
          },
          "capabilities" : {
            "pci" : true,
            "msi" : "Message Signalled Interrupts",
            "pciexpress" : "PCI Express",
            "pm" : "Power Management",
            "normal_decode" : true,
            "bus_master" : "bus mastering",
            "cap_list" : "PCI capabilities listing"
          }            {
              "id" : "fiber:0",
              "class" : "bus",
              "claimed" : true,
              "handle" : "PCI:0000:83:00.0",
              "description" : "Fibre Channel",
              "product" : "ISP2532-based 8Gb Fibre Channel to PCI Express HBA",
              "vendor" : "QLogic Corp.",
              "physid" : "0",
              "businfo" : "pci@0000:83:00.0",
              "logicalname" : "scsi1",
              "version" : "02",
              "width" : 64,
              "clock" : 33000000,
              "configuration" : {
                "driver" : "qla2xxx",
                "latency" : "0"
              },
              "capabilities" : {
                "pm" : "Power Management",
                "pciexpress" : "PCI Express",
                "msi" : "Message Signalled Interrupts",
                "vpd" : "Vital Product Data",
                "msix" : "MSI-X",
                "bus_master" : "bus mastering",
                "cap_list" : "PCI capabilities listing",
                "rom" : "extension ROM"
              }                {
                  "id" : "tape",
                  "class" : "tape",
                  "handle" : "SCSI:01:00:00:00",
                  "description" : "SCSI Tape",
                  "product" : "ULTRIUM-HH7",
                  "vendor" : "IBM",
                  "physid" : "0.0.0",
                  "businfo" : "scsi@1:0.0.0",
                  "version" : "M571",
                  "serial" : "XXX",
                  "configuration" : {
                    "ansiversion" : "6"
                  },
                  "capabilities" : {
                    "removable" : "support is removable"
                  }
                },
                {
                  "id" : "changer",
                  "class" : "generic",
                  "handle" : "SCSI:01:00:00:01",
                  "description" : "SCSI Medium Changer",
                  "product" : "FlexStor II",
                  "vendor" : "BDT",
                  "physid" : "0.0.1",
                  "businfo" : "scsi@1:0.0.1",
                  "version" : "5.60",
                  "serial" : "XXX",
                  "configuration" : {
                    "ansiversion" : "5"
                  },
                  "capabilities" : {
                    "removable" : "support is removable"
                  }
                },

            },
            {
              "id" : "fiber:1",
              "class" : "bus",
              "claimed" : true,
              "handle" : "PCI:0000:83:00.1",
              "description" : "Fibre Channel",
              "product" : "ISP2532-based 8Gb Fibre Channel to PCI Express HBA",
              "vendor" : "QLogic Corp.",
              "physid" : "0.1",
              "businfo" : "pci@0000:83:00.1",
              "logicalname" : "scsi13",
              "version" : "02",
              "width" : 64,
              "clock" : 33000000,
              "configuration" : {
                "driver" : "qla2xxx",
                "latency" : "0"
              },
              "capabilities" : {
                "pm" : "Power Management",
                "pciexpress" : "PCI Express",
                "msi" : "Message Signalled Interrupts",
                "vpd" : "Vital Product Data",
                "msix" : "MSI-X",
                "bus_master" : "bus mastering",
                "cap_list" : "PCI capabilities listing",
                "rom" : "extension ROM"
              }                {
                  "id" : "tape",
                  "class" : "tape",
                  "handle" : "SCSI:13:00:00:00",
                  "description" : "SCSI Tape",
                  "product" : "ULTRIUM-HH7",
                  "vendor" : "IBM",
                  "physid" : "0.0.0",
                  "businfo" : "scsi@13:0.0.0",
                  "version" : "M571",
                  "serial" : "XXX",
                  "configuration" : {
                    "ansiversion" : "6"
                  },
                  "capabilities" : {
                    "removable" : "support is removable"
                  }
                },

            },

        }
lshw-lto-related.txt (5,167 bytes)   
bruno-at-bareos

bruno-at-bareos

2021-09-21 13:30

manager   ~0004269

Hello we wonder if you can share your sd configuration (/etc/bareos/bareos-dir.d/storage/* + /etc/bareos/bareos-sd.d/*)
Don't forget to remove the Password line.
Thanks.
kabassanov

kabassanov

2021-09-21 14:14

reporter   ~0004270

Here it is...

cat /etc/bareos/bareos-sd.d/device/Autochanger.conf
#
# An autochanger device with two drives
#
Autochanger {
  Name = LTO-7-Autochanger
  Device = Drive-1, Drive-2
  Changer Device = /dev/changer
  Changer Command = "/etc/bareos/custom-configs/scripts/mtx-changer %c %o %S %a %d"
  Description = "Tape device library. A connecting Director must have the same Name and MediaType."

}

cat /etc/bareos/bareos-sd.d/device/Drive-1.conf
# A Linux or Solaris LTO-7 tape drive
#
Device {
  Name = Drive-1
  Drive Index = 0
  Media Type = LTO-7
  Archive Device = /dev/nst/tape0
  AutoChanger = yes
  LabelMedia = no;
  AutomaticMount = yes; # when device opened, read it
  AlwaysOpen = yes;
  RemovableMedia = yes;
  RandomAccess = no;
  Minimum Block Size = 1048576
  Maximum Block Size = 1048576
  Label Block Size = 1048576
  Maximum Concurrent Jobs = 1
  Spool Directory = /bareos-spool
  Maximum Spool Size = 4398046511104
}

cat /etc/bareos/bareos-sd.d/device/Drive-2.conf
Device {
  Name = Drive-2
  Drive Index = 1
  Media Type = LTO-7
  Archive Device = /dev/nst/tape1
  AutoChanger = yes
  LabelMedia = no;
  AutomaticMount = yes; # when device opened, read it
  AlwaysOpen = yes;
  RemovableMedia = yes;
  RandomAccess = no;
  Minimum Block Size = 1048576
  Maximum Block Size = 1048576
  Label Block Size = 1048576
  Maximum Concurrent Jobs = 1
  Spool Directory = /bareos-spool
  Maximum Spool Size = 4398046511104
}

Storage {
  Name = XXXXX
  Maximum Concurrent Jobs = 100
  SDAddress = XXXXX
  SDPort = 9103 # Director's port
  WorkingDirectory = "/var/lib/bareos"
  Pid Directory = "/run/bareos"
  TLS Enable = No
}


cat /etc/bareos/bareos-dir.d/storage/Changer.conf
# -----------------------------------------------
# Definition of LTO-7 tape changer in Autochanger device
Storage {
  Name = Changer
# Do not use "localhost" here
  Address = XXXXX # N.B. Use a fully qualified name here
  SDPort = 9103
  Password = "xxxxxx"
  Device = LTO-7-Autochanger
  Auto Changer = yes
  Media Type = LTO-7 # must be same as MediaType in Storage daemon
  Maximum Concurrent Jobs = 10
  TLS Enable = No
}

Regards.
bruno-at-bareos

bruno-at-bareos

2021-09-28 15:26

manager   ~0004281

Hello, we tried to reproduce the reported behavior but didn't success.

It was not possible to reproduce the mis-behaviour reported by you, we tried on VM with using mhvtl with actual code.
We have previously some reports of tests with LTO4 drives (code was at 12.4 and 13.2) with no differences too.

I was able to reproduce one time by using device with compression off for bareos, and compression on for bacula.
Then bareos is 100% slower than bacula, but that's trying to compare apples and bananas.
Would it be possible that compression was disabled when you tried Bareos ?

Methodology : btape is writing 3 times the file of 3.221 GB and return an average.
The test is run 3 times, and then the average of the 3 resulting averages is summed in the table below.

```
 |-------------|-----------|-------------|-------------------------|
 |Run | zero data | random data | zero+block | rand+block |
 |-------------|-----------|-------------|-------------------------|
 |bacula 64 | 327.6 | 233.6 | 124.0 | 226.7 |
 |bacula 256 | 328.1 | 297.4 | 193.5 | 214.4 |
 |bacula 512 | 371.6 | 297.4 | 264.8 | 233.7 |
 |_Average_ bac| 342.4 | 276.1 | 194.1 | 224.9 |
 |-------------|-----------|-------------|-------------------------|
 |bareos 64 | 307.7 | 268.6 | 128.4 | 178.6 |
 |bareos 256 | 268.5 | 292.8 | 195.2 | 167.0 |
 |bareos 512 | 331.5 | 272.7 | 261.1 | 210.0 |
 |_Average_ bar| 302.6 | 278.0 | 192.9 | 222.8 |
 |-------------|-----------|-------------|-------------------------|
 |_FactorXslow_| 1.13 | 0.99 | 1.00 | 1.00 |
 |-------------|-----------|-------------|-------------------------|
```
 **Note** mhvtl doesn't work with block of 1048576.

I will close this report as non-fix (we don't know what to fix, as we can't reproduce it actually). But please feel free to reopen it if you have any other clue or evidence to add.
Best regards.
bruno-at-bareos

bruno-at-bareos

2021-09-28 15:27

manager   ~0004282

see previous note

Issue History

Date Modified Username Field Change
2020-09-08 23:16 kabassanov New Issue
2021-09-07 09:27 bruno-at-bareos Note Added: 0004251
2021-09-11 17:14 kabassanov File Added: lshw-lto-related.txt
2021-09-11 17:14 kabassanov Note Added: 0004262
2021-09-21 13:30 bruno-at-bareos Note Added: 0004269
2021-09-21 14:14 kabassanov Note Added: 0004270
2021-09-28 15:26 bruno-at-bareos Note Added: 0004281
2021-09-28 15:27 bruno-at-bareos Assigned To => bruno-at-bareos
2021-09-28 15:27 bruno-at-bareos Status new => assigned
2021-09-28 15:27 bruno-at-bareos Status assigned => closed
2021-09-28 15:27 bruno-at-bareos Resolution open => unable to reproduce
2021-09-28 15:27 bruno-at-bareos Note Added: 0004282