View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0001325 | bareos-core | storage daemon | public | 2021-03-03 07:44 | 2024-03-27 15:14 |
Reporter | sergii.chebotkin | Assigned To | bruno-at-bareos | ||
Priority | high | Severity | major | Reproducibility | always |
Status | closed | Resolution | won't fix | ||
Platform | Linux | OS | CentOS | OS Version | 7 |
Product Version | 20.0.0 | ||||
Summary | 0001325: Volume management issue | ||||
Description | We have faced with the constant issues with volumes management during AWS backups with BareOS v20 . We did not see this issue with BareOS v18. The issue is creation more then one (sometime up to 15) volume with random size for 1 backup. As big backup job (2Gb and more) as many volumes will be created. Some of the volumes marked as "Full", some "Error". Following examples for a Full backup, but Incremental having the same issue. Full-0001 S3_Object S3_Object1 11 day(s) ago Error 1 year(s) 1.07 TB 68.07 GB Full-0029 S3_Object S3_Object1 11 day(s) ago Full expires in 354 days 1.07 TB 1.36 GB Full-0030 S3_Object S3_Object1 11 day(s) ago Full expires in 354 days 1.07 TB 3.36 GB Full-0031 S3_Object S3_Object1 11 day(s) ago Full expires in 354 days 1.07 TB 5.66 GB We are using following Full backup configuration : DIR POOL: Maximum Volume Jobs - is not set. Maximum Volumes = 1500 Maximum Volume Bytes = '1000G' Volume recycle - true Volume retention - '360 days' SD DEVICE: Name = "AWS_S3_1-00" Always Open = false Archive Device = "AWS S3 Storage" Automatic Mount = true Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/aws.profile,bucket=bareos-backups,chunksize=100M,retries=0,iothreads=0,ioslots=0" Device Type = droplet Label Media = true Media Type = "S3_Object1" Random Access = true Removable Media = false Logs example: Job name : Full-0029 AWS - 12 files - 100.0 MB Full-0029 S3_Object S3_Object1 2 day(s) ago Full expires in 363 days 1.07 TB 1.36 GB Volume jobs 1 Volume name(s) in use: Full-0029|Full-0030|Full-0031|Full-0032 47 2021-02-20 18:04:26 bareos-dir JobId 92: Insert of attributes batch table done 46 2021-02-20 18:04:17 bareos-dir JobId 92: Insert of attributes batch table with 737036 entries start 45 2021-02-20 18:04:17 bareos-sd JobId 92: Elapsed time=01:13:31, Transfer rate=2.371 M Bytes/second 44 2021-02-20 18:04:15 bareos-sd JobId 92: Releasing device "AWS_S3_1-00" (AWS S3 Storage). 43 2021-02-20 18:03:47 bareos-sd JobId 92: New volume "Full-0032" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 20-Feb-2021 18:03. 42 2021-02-20 18:03:47 bareos-sd JobId 92: Wrote label to prelabeled Volume "Full-0032" on device "AWS_S3_1-00" (AWS S3 Storage) 41 2021-02-20 18:03:47 bareos-sd JobId 92: Labeled new Volume "Full-0032" on device "AWS_S3_1-00" (AWS S3 Storage). 40 2021-02-20 18:03:46 bareos-dir JobId 92: Created new Volume "Full-0032" in catalog. 39 2021-02-20 18:03:46 bareos-sd JobId 92: End of medium on Volume "Full-0031" Bytes=5,662,281,530 Blocks=87,771 at 20-Feb-2021 18:03. 38 2021-02-20 18:03:46 bareos-sd JobId 92: Error: stored/block.cc:822 Write error on fd=0 at file:blk 1:1367314233 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 37 2021-02-20 18:03:46 bareos-sd JobId 92: Error: stored/block.cc:803 Write error at 1:1367314233 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 36 2021-02-20 17:40:54 bareos-sd JobId 92: New volume "Full-0031" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 20-Feb-2021 17:40. 35 2021-02-20 17:40:54 bareos-sd JobId 92: Wrote label to prelabeled Volume "Full-0031" on device "AWS_S3_1-00" (AWS S3 Storage) 34 2021-02-20 17:40:54 bareos-sd JobId 92: Labeled new Volume "Full-0031" on device "AWS_S3_1-00" (AWS S3 Storage). 33 2021-02-20 17:40:53 bareos-dir JobId 92: Created new Volume "Full-0031" in catalog. 32 2021-02-20 17:40:53 bareos-sd JobId 92: End of medium on Volume "Full-0030" Bytes=3,355,396,766 Blocks=52,012 at 20-Feb-2021 17:40. 31 2021-02-20 17:40:53 bareos-sd JobId 92: Error: stored/block.cc:822 Write error on fd=0 at file:blk 0:3355396765 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 30 2021-02-20 17:40:53 bareos-sd JobId 92: Error: stored/block.cc:803 Write error at 0:3355396765 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 29 2021-02-20 17:07:07 some.host.com JobId 92: Disallowed filesystem. Will not descend from / into /var/lib/nfs/rpc_pipefs 28 2021-02-20 17:06:44 some.host.com JobId 92: Disallowed filesystem. Will not descend from / into /run 27 2021-02-20 17:06:44 some.host.com JobId 92: Disallowed filesystem. Will not descend from / into /sys 26 2021-02-20 17:06:44 some.host.com JobId 92: Disallowed filesystem. Will not descend from / into /dev 25 2021-02-20 17:02:29 bareos-sd JobId 92: New volume "Full-0030" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 20-Feb-2021 17:02. 24 2021-02-20 17:02:29 bareos-sd JobId 92: Wrote label to prelabeled Volume "Full-0030" on device "AWS_S3_1-00" (AWS S3 Storage) 23 2021-02-20 17:02:29 bareos-sd JobId 92: Labeled new Volume "Full-0030" on device "AWS_S3_1-00" (AWS S3 Storage). 22 2021-02-20 17:02:24 bareos-dir JobId 92: Created new Volume "Full-0030" in catalog. 21 2021-02-20 17:02:24 bareos-sd JobId 92: End of medium on Volume "Full-0029" Bytes=1,363,138,070 Blocks=21,130 at 20-Feb-2021 17:02. 20 2021-02-20 17:02:24 bareos-sd JobId 92: Error: stored/block.cc:822 Write error on fd=0 at file:blk 0:1363138069 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 19 2021-02-20 17:02:24 bareos-sd JobId 92: Error: stored/block.cc:803 Write error at 0:1363138069 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 18 2021-02-20 16:50:39 bareos-sd JobId 92: Wrote label to prelabeled Volume "Full-0029" on device "AWS_S3_1-00" (AWS S3 Storage) 17 2021-02-20 16:50:39 bareos-sd JobId 92: Labeled new Volume "Full-0029" on device "AWS_S3_1-00" (AWS S3 Storage). 16 2021-02-20 16:50:38 bareos-dir JobId 92: Created new Volume "Full-0029" in catalog. 15 2021-02-20 16:50:38 bareos-sd JobId 92: Marking Volume "Full-0001" in Error in Catalog. 14 2021-02-20 16:50:38 bareos-sd JobId 92: Error: Bareos cannot write on disk Volume "Full-0001" because: The sizes do not match! Volume=13107200000 Catalog=68068053996 13 2021-02-20 16:50:06 bareos-sd JobId 92: Volume "Full-0001" previously written, moving to end of data. | ||||
Steps To Reproduce | Just start scheduled backup. | ||||
Additional Information | We can not see this issue in version 18. Switched on version 20 cause of AWS issue. | ||||
Tags | droplet, s3;droplet;aws;storage, storage, volume | ||||
I have changed configuration like: DIR POOL: Maximum Volume Jobs = 0 Maximum Volumes = 0 Maximum Volume Bytes = '0G' Volume recycle - true Volume retention - '360 days But anyway got one job with 115Gb backup which created 3 volumes : Incremental-0956 S3_Object S3_Object1 1 day(s) ago Full expires in 6 days 0.00 B 14.68 GB Incremental-0957 S3_Object S3_Object1 1 day(s) ago Full expires in 6 days 0.00 B 14.68 GB Incremental-0958 S3_Object S3_Object1 today Used expires in 7 days 0.00 B 86.02 GB 37 2021-03-03 01:13:20 bareos-dir JobId 26619: Insert of attributes batch table done 36 2021-03-03 01:13:20 bareos-dir JobId 26619: Insert of attributes batch table with 4619 entries start 35 2021-03-03 01:13:20 bareos-sd JobId 26619: Elapsed time=02:22:35, Transfer rate=13.47 M Bytes/second 34 2021-03-03 01:13:13 bareos-sd JobId 26619: Releasing device "AWS_S3_1-00" (AWS S3 Storage). 33 2021-03-02 23:27:35 bareos-sd JobId 26619: New volume "Incremental-0958" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 02-Mar-2021 23:27. 32 2021-03-02 23:27:35 bareos-dir JobId 26619: Max Volume jobs=1 exceeded. Marking Volume "Incremental-0958" as Used. 31 2021-03-02 23:27:35 bareos-sd JobId 26619: Wrote label to prelabeled Volume "Incremental-0958" on device "AWS_S3_1-00" (AWS S3 Storage) 30 2021-03-02 23:27:35 bareos-sd JobId 26619: Labeled new Volume "Incremental-0958" on device "AWS_S3_1-00" (AWS S3 Storage). 29 2021-03-02 23:27:34 bareos-dir JobId 26619: Created new Volume "Incremental-0958" in catalog. 28 2021-03-02 23:27:33 bareos-sd JobId 26619: End of medium on Volume "Incremental-0957" Bytes=14,680,028,204 Blocks=227,555 at 02-Mar-2021 23:27. 27 2021-03-02 23:27:33 bareos-sd JobId 26619: Error: stored/block.cc:822 Write error on fd=0 at file:blk 3:1795126315 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 26 2021-03-02 23:27:33 bareos-sd JobId 26619: Error: stored/block.cc:803 Write error at 3:1795126315 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 25 2021-03-02 23:08:58 bareos-sd JobId 26619: New volume "Incremental-0957" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 02-Mar-2021 23:08. 24 2021-03-02 23:08:58 bareos-dir JobId 26619: Max Volume jobs=1 exceeded. Marking Volume "Incremental-0957" as Used. 23 2021-03-02 23:08:58 bareos-sd JobId 26619: Wrote label to prelabeled Volume "Incremental-0957" on device "AWS_S3_1-00" (AWS S3 Storage) 22 2021-03-02 23:08:58 bareos-sd JobId 26619: Labeled new Volume "Incremental-0957" on device "AWS_S3_1-00" (AWS S3 Storage). 21 2021-03-02 23:08:58 bareos-dir JobId 26619: Created new Volume "Incremental-0957" in catalog. 20 2021-03-02 23:08:57 bareos-sd JobId 26619: End of medium on Volume "Incremental-0956" Bytes=14,680,028,152 Blocks=227,555 at 02-Mar-2021 23:08. 19 2021-03-02 23:08:57 bareos-sd JobId 26619: Error: stored/block.cc:822 Write error on fd=0 at file:blk 3:1795126263 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 18 2021-03-02 23:08:57 bareos-sd JobId 26619: Error: stored/block.cc:803 Write error at 3:1795126263 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=Input/output error. 17 2021-03-02 22:51:05 some.host.com JobId 26619: Disallowed filesystem. Will not descend from / into /var/lib/nfs/rpc_pipefs 16 2021-03-02 22:50:53 some.host.com JobId 26619: Disallowed filesystem. Will not descend from / into /run 15 2021-03-02 22:50:53 some.host.com JobId 26619: Disallowed filesystem. Will not descend from / into /sys 14 2021-03-02 22:50:53 some.host.com JobId 26619: Disallowed filesystem. Will not descend from / into /dev 13 2021-03-02 22:50:42 bareos-dir JobId 26619: Max Volume jobs=1 exceeded. Marking Volume "Incremental-0956" as Used. 12 2021-03-02 22:50:42 bareos-sd JobId 26619: Wrote label to prelabeled Volume "Incremental-0956" on device "AWS_S3_1-00" (AWS S3 Storage) |
|
droplet will be completely replaced by a new component in 23.1 closing here, before migrating bug tracker to github issues |
|
Date Modified | Username | Field | Change |
---|---|---|---|
2021-03-03 07:44 | sergii.chebotkin | New Issue | |
2021-03-03 07:44 | sergii.chebotkin | Tag Attached: aws | |
2021-03-03 07:44 | sergii.chebotkin | Tag Attached: director | |
2021-03-03 07:44 | sergii.chebotkin | Tag Attached: storage | |
2021-03-03 07:44 | sergii.chebotkin | Tag Attached: volume | |
2021-03-03 10:26 | sergii.chebotkin | Tag Detached: aws | |
2021-03-03 10:26 | sergii.chebotkin | Tag Attached: s3;droplet;aws;storage | |
2021-03-03 10:38 | sergii.chebotkin | Note Added: 0004096 | |
2023-12-12 15:23 | bruno-at-bareos | Description Updated | |
2023-12-12 15:23 | bruno-at-bareos | Tag Attached: droplet | |
2023-12-13 13:23 | bruno-at-bareos | Tag Detached: director | |
2024-03-27 15:14 | bruno-at-bareos | Assigned To | => bruno-at-bareos |
2024-03-27 15:14 | bruno-at-bareos | Status | new => closed |
2024-03-27 15:14 | bruno-at-bareos | Resolution | open => won't fix |
2024-03-27 15:14 | bruno-at-bareos | Note Added: 0005877 |