View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001325bareos-corestorage daemonpublic2021-03-03 07:442024-03-27 15:14
Reportersergii.chebotkin Assigned Tobruno-at-bareos  
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionwon't fix 
PlatformLinuxOSCentOSOS Version7
Product Version20.0.0 
Summary0001325: Volume management issue
DescriptionWe have faced with the constant issues with volumes management during AWS backups with BareOS v20 . We did not see this issue with BareOS v18.
The issue is creation more then one (sometime up to 15) volume with random size for 1 backup. As big backup job (2Gb and more) as many volumes will be created.
Some of the volumes marked as "Full", some "Error". Following examples for a Full backup, but Incremental having the same issue.

Full-0001 S3_Object S3_Object1 11 day(s) ago Error 1 year(s) 1.07 TB 68.07 GB
Full-0029 S3_Object S3_Object1 11 day(s) ago Full expires in 354 days 1.07 TB 1.36 GB
Full-0030 S3_Object S3_Object1 11 day(s) ago Full expires in 354 days 1.07 TB 3.36 GB
Full-0031 S3_Object S3_Object1 11 day(s) ago Full expires in 354 days 1.07 TB 5.66 GB

We are using following Full backup configuration :

DIR POOL:
Maximum Volume Jobs - is not set.
Maximum Volumes = 1500
Maximum Volume Bytes = '1000G'
Volume recycle - true
Volume retention - '360 days'

SD DEVICE:
Name = "AWS_S3_1-00"
Always Open = false
Archive Device = "AWS S3 Storage"
Automatic Mount = true
Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/aws.profile,bucket=bareos-backups,chunksize=100M,retries=0,iothreads=0,ioslots=0"
Device Type = droplet
Label Media = true
Media Type = "S3_Object1"
Random Access = true
Removable Media = false

Logs example:

Job name : Full-0029
AWS - 12 files - 100.0 MB

Full-0029 S3_Object S3_Object1 2 day(s) ago Full expires in 363 days 1.07 TB 1.36 GB
Volume jobs 1
Volume name(s) in use: Full-0029|Full-0030|Full-0031|Full-0032

47 2021-02-20 18:04:26 bareos-dir JobId 92: Insert of attributes batch table done
46 2021-02-20 18:04:17 bareos-dir JobId 92: Insert of attributes batch table with 737036 entries start
45 2021-02-20 18:04:17 bareos-sd JobId 92: Elapsed time=01:13:31, Transfer rate=2.371 M Bytes/second
44 2021-02-20 18:04:15 bareos-sd JobId 92: Releasing device "AWS_S3_1-00" (AWS S3 Storage).
43 2021-02-20 18:03:47 bareos-sd JobId 92: New volume "Full-0032" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 20-Feb-2021 18:03.
42 2021-02-20 18:03:47 bareos-sd JobId 92: Wrote label to prelabeled Volume "Full-0032" on device "AWS_S3_1-00" (AWS S3 Storage)
41 2021-02-20 18:03:47 bareos-sd JobId 92: Labeled new Volume "Full-0032" on device "AWS_S3_1-00" (AWS S3 Storage).
40 2021-02-20 18:03:46 bareos-dir JobId 92: Created new Volume "Full-0032" in catalog.
39 2021-02-20 18:03:46 bareos-sd JobId 92: End of medium on Volume "Full-0031" Bytes=5,662,281,530 Blocks=87,771 at 20-Feb-2021 18:03.
38 2021-02-20 18:03:46 bareos-sd JobId 92: Error: stored/block.cc:822 Write error on fd=0 at file:blk 1:1367314233 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
37 2021-02-20 18:03:46 bareos-sd JobId 92: Error: stored/block.cc:803 Write error at 1:1367314233 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
36 2021-02-20 17:40:54 bareos-sd JobId 92: New volume "Full-0031" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 20-Feb-2021 17:40.
35 2021-02-20 17:40:54 bareos-sd JobId 92: Wrote label to prelabeled Volume "Full-0031" on device "AWS_S3_1-00" (AWS S3 Storage)
34 2021-02-20 17:40:54 bareos-sd JobId 92: Labeled new Volume "Full-0031" on device "AWS_S3_1-00" (AWS S3 Storage).
33 2021-02-20 17:40:53 bareos-dir JobId 92: Created new Volume "Full-0031" in catalog.
32 2021-02-20 17:40:53 bareos-sd JobId 92: End of medium on Volume "Full-0030" Bytes=3,355,396,766 Blocks=52,012 at 20-Feb-2021 17:40.
31 2021-02-20 17:40:53 bareos-sd JobId 92: Error: stored/block.cc:822 Write error on fd=0 at file:blk 0:3355396765 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
30 2021-02-20 17:40:53 bareos-sd JobId 92: Error: stored/block.cc:803 Write error at 0:3355396765 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
29 2021-02-20 17:07:07 some.host.com JobId 92: Disallowed filesystem. Will not descend from / into /var/lib/nfs/rpc_pipefs
28 2021-02-20 17:06:44 some.host.com JobId 92: Disallowed filesystem. Will not descend from / into /run
27 2021-02-20 17:06:44 some.host.com JobId 92: Disallowed filesystem. Will not descend from / into /sys
26 2021-02-20 17:06:44 some.host.com JobId 92: Disallowed filesystem. Will not descend from / into /dev
25 2021-02-20 17:02:29 bareos-sd JobId 92: New volume "Full-0030" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 20-Feb-2021 17:02.
24 2021-02-20 17:02:29 bareos-sd JobId 92: Wrote label to prelabeled Volume "Full-0030" on device "AWS_S3_1-00" (AWS S3 Storage)
23 2021-02-20 17:02:29 bareos-sd JobId 92: Labeled new Volume "Full-0030" on device "AWS_S3_1-00" (AWS S3 Storage).
22 2021-02-20 17:02:24 bareos-dir JobId 92: Created new Volume "Full-0030" in catalog.
21 2021-02-20 17:02:24 bareos-sd JobId 92: End of medium on Volume "Full-0029" Bytes=1,363,138,070 Blocks=21,130 at 20-Feb-2021 17:02.
20 2021-02-20 17:02:24 bareos-sd JobId 92: Error: stored/block.cc:822 Write error on fd=0 at file:blk 0:1363138069 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
19 2021-02-20 17:02:24 bareos-sd JobId 92: Error: stored/block.cc:803 Write error at 0:1363138069 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
18 2021-02-20 16:50:39 bareos-sd JobId 92: Wrote label to prelabeled Volume "Full-0029" on device "AWS_S3_1-00" (AWS S3 Storage)
17 2021-02-20 16:50:39 bareos-sd JobId 92: Labeled new Volume "Full-0029" on device "AWS_S3_1-00" (AWS S3 Storage).
16 2021-02-20 16:50:38 bareos-dir JobId 92: Created new Volume "Full-0029" in catalog.
15 2021-02-20 16:50:38 bareos-sd JobId 92: Marking Volume "Full-0001" in Error in Catalog.
14 2021-02-20 16:50:38 bareos-sd JobId 92: Error: Bareos cannot write on disk Volume "Full-0001" because: The sizes do not match! Volume=13107200000 Catalog=68068053996
13 2021-02-20 16:50:06 bareos-sd JobId 92: Volume "Full-0001" previously written, moving to end of data.
Steps To ReproduceJust start scheduled backup.
Additional InformationWe can not see this issue in version 18. Switched on version 20 cause of AWS issue.
Tagsdroplet, s3;droplet;aws;storage, storage, volume

Activities

sergii.chebotkin

sergii.chebotkin

2021-03-03 10:38

reporter   ~0004096

I have changed configuration like:
DIR POOL:
Maximum Volume Jobs = 0
Maximum Volumes = 0
Maximum Volume Bytes = '0G'
Volume recycle - true
Volume retention - '360 days

But anyway got one job with 115Gb backup which created 3 volumes :

Incremental-0956 S3_Object S3_Object1 1 day(s) ago Full expires in 6 days 0.00 B 14.68 GB
Incremental-0957 S3_Object S3_Object1 1 day(s) ago Full expires in 6 days 0.00 B 14.68 GB
Incremental-0958 S3_Object S3_Object1 today Used expires in 7 days 0.00 B 86.02 GB

37 2021-03-03 01:13:20 bareos-dir JobId 26619: Insert of attributes batch table done
36 2021-03-03 01:13:20 bareos-dir JobId 26619: Insert of attributes batch table with 4619 entries start
35 2021-03-03 01:13:20 bareos-sd JobId 26619: Elapsed time=02:22:35, Transfer rate=13.47 M Bytes/second
34 2021-03-03 01:13:13 bareos-sd JobId 26619: Releasing device "AWS_S3_1-00" (AWS S3 Storage).
33 2021-03-02 23:27:35 bareos-sd JobId 26619: New volume "Incremental-0958" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 02-Mar-2021 23:27.
32 2021-03-02 23:27:35 bareos-dir JobId 26619: Max Volume jobs=1 exceeded. Marking Volume "Incremental-0958" as Used.
31 2021-03-02 23:27:35 bareos-sd JobId 26619: Wrote label to prelabeled Volume "Incremental-0958" on device "AWS_S3_1-00" (AWS S3 Storage)
30 2021-03-02 23:27:35 bareos-sd JobId 26619: Labeled new Volume "Incremental-0958" on device "AWS_S3_1-00" (AWS S3 Storage).
29 2021-03-02 23:27:34 bareos-dir JobId 26619: Created new Volume "Incremental-0958" in catalog.
28 2021-03-02 23:27:33 bareos-sd JobId 26619: End of medium on Volume "Incremental-0957" Bytes=14,680,028,204 Blocks=227,555 at 02-Mar-2021 23:27.
27 2021-03-02 23:27:33 bareos-sd JobId 26619: Error: stored/block.cc:822 Write error on fd=0 at file:blk 3:1795126315 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
26 2021-03-02 23:27:33 bareos-sd JobId 26619: Error: stored/block.cc:803 Write error at 3:1795126315 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
25 2021-03-02 23:08:58 bareos-sd JobId 26619: New volume "Incremental-0957" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 02-Mar-2021 23:08.
24 2021-03-02 23:08:58 bareos-dir JobId 26619: Max Volume jobs=1 exceeded. Marking Volume "Incremental-0957" as Used.
23 2021-03-02 23:08:58 bareos-sd JobId 26619: Wrote label to prelabeled Volume "Incremental-0957" on device "AWS_S3_1-00" (AWS S3 Storage)
22 2021-03-02 23:08:58 bareos-sd JobId 26619: Labeled new Volume "Incremental-0957" on device "AWS_S3_1-00" (AWS S3 Storage).
21 2021-03-02 23:08:58 bareos-dir JobId 26619: Created new Volume "Incremental-0957" in catalog.
20 2021-03-02 23:08:57 bareos-sd JobId 26619: End of medium on Volume "Incremental-0956" Bytes=14,680,028,152 Blocks=227,555 at 02-Mar-2021 23:08.
19 2021-03-02 23:08:57 bareos-sd JobId 26619: Error: stored/block.cc:822 Write error on fd=0 at file:blk 3:1795126263 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
18 2021-03-02 23:08:57 bareos-sd JobId 26619: Error: stored/block.cc:803 Write error at 3:1795126263 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error.
17 2021-03-02 22:51:05 some.host.com JobId 26619: Disallowed filesystem. Will not descend from / into /var/lib/nfs/rpc_pipefs
16 2021-03-02 22:50:53 some.host.com JobId 26619: Disallowed filesystem. Will not descend from / into /run
15 2021-03-02 22:50:53 some.host.com JobId 26619: Disallowed filesystem. Will not descend from / into /sys
14 2021-03-02 22:50:53 some.host.com JobId 26619: Disallowed filesystem. Will not descend from / into /dev
13 2021-03-02 22:50:42 bareos-dir JobId 26619: Max Volume jobs=1 exceeded. Marking Volume "Incremental-0956" as Used.
12 2021-03-02 22:50:42 bareos-sd JobId 26619: Wrote label to prelabeled Volume "Incremental-0956" on device "AWS_S3_1-00" (AWS S3 Storage)
bruno-at-bareos

bruno-at-bareos

2024-03-27 15:14

manager   ~0005877

droplet will be completely replaced by a new component in 23.1
closing here, before migrating bug tracker to github issues

Issue History

Date Modified Username Field Change
2021-03-03 07:44 sergii.chebotkin New Issue
2021-03-03 07:44 sergii.chebotkin Tag Attached: aws
2021-03-03 07:44 sergii.chebotkin Tag Attached: director
2021-03-03 07:44 sergii.chebotkin Tag Attached: storage
2021-03-03 07:44 sergii.chebotkin Tag Attached: volume
2021-03-03 10:26 sergii.chebotkin Tag Detached: aws
2021-03-03 10:26 sergii.chebotkin Tag Attached: s3;droplet;aws;storage
2021-03-03 10:38 sergii.chebotkin Note Added: 0004096
2023-12-12 15:23 bruno-at-bareos Description Updated
2023-12-12 15:23 bruno-at-bareos Tag Attached: droplet
2023-12-13 13:23 bruno-at-bareos Tag Detached: director
2024-03-27 15:14 bruno-at-bareos Assigned To => bruno-at-bareos
2024-03-27 15:14 bruno-at-bareos Status new => closed
2024-03-27 15:14 bruno-at-bareos Resolution open => won't fix
2024-03-27 15:14 bruno-at-bareos Note Added: 0005877