View Issue Details

IDProjectCategoryView StatusLast Update
0001534bareos-corestorage daemonpublic2023-11-02 10:54
Reporterpavelr Assigned Tobruno-at-bareos  
PriorityimmediateSeverityblockReproducibilityalways
Status closedResolutionfixed 
PlatformLinuxOSRHEL (and clones)OS Version9
Product Version22.0.3 
Summary0001534: s3 backend doesn't work
Descriptionintegration with aws S3 does not work. used https://docs.bareos.org/TasksAndConcepts/StorageBackends.html and no matter what i configure i get the following error
2023-04-30 17:31:32 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/profile.c:251: conf_cb_func: invalid value 'use_https'
i do have correct permissions and tokens (confirmed with aws cli and was able to put a file in the bucket.
Steps To Reproduceusing this configuration:
host = s3.amazonaws.com # This parameter is only used as baseurl and will be prepended with bucket and location set in device resource to form correct url
use_https = true
access_key = key
secret_key = secret
pricing_dir = "" # If not empty, an droplet.csv file will be created which will record all S3 operations.
backend = s3
aws_auth_sign_version = 4 # Currently, AWS S3 uses version 4. The Ceph S3 gateway uses version 2.
aws_region = us-west-2

throws:
2023-04-30 17:31:32 bareos-sd JobId 512: Warning: stored/label.cc:372 Open device "AWS_S3_1-00" (AWS S3 Storage) Volume "s3-longterm-0403" failed: ERR=stored/dev.cc:602 Could not open: AWS S3 Storage/s3-longterm-0403

2023-04-30 17:31:32 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/profile.c:251: conf_cb_func: invalid value 'use_https'
2023-04-30 17:31:32 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/profile.c:251: conf_cb_func: invalid value 'use_https'
2023-04-30 17:31:32 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/profile.c:251: conf_cb_func: invalid value 'use_https'
Additional Informationrunning on centos 9.
version:

bareos-webui-22.0.3~pre37.3c0624880-27.el9.x86_64
bareos-common-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-filedaemon-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-database-common-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-database-postgresql-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-database-tools-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-director-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-filedaemon-python-plugins-common-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-filedaemon-python3-plugin-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-bconsole-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-client-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-tools-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-storage-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-storage-droplet-22.0.4~pre78.1134d9896-47.el9.x86_64
bareos-filedaemon-percona-xtrabackup-python-plugin-22.0.4~pre78.1134d9896-47.el9.x86_64
TagsNo tags attached.

Activities

pavelr

pavelr

2023-04-30 21:05

reporter   ~0004994

sorry. mixed up the errors:
#option 1:
host = s3.amazonaws.com:443
#result
2023-04-30 17:31:32 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/profile.c:251: conf_cb_func: invalid value 'use_https'

#option 2:
host = s3.amazonaws.com
#error:
2023-04-30 19:01:16 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/addrlist.c:600: dpl_addrlist_add: cannot lookup host s3.amazonaws.com
bruno-at-bareos

bruno-at-bareos

2023-05-03 15:44

manager   ~0004998

Did you tried to remove the comment ?

host = s3.amazonaws.com # This parameter is only used as baseurl and will be prepended with bucket and location set in device resource to form correct url
use_https = true
access_key = key
secret_key = secret
pricing_dir = "" # If not empty, an droplet.csv file will be created which will record all S3 operations.
backend = s3
aws_auth_sign_version = 4 # Currently, AWS S3 uses version 4. The Ceph S3 gateway uses version 2.
aws_region = us-west-2

to

host = s3.amazonaws.com
use_https = true
access_key = key
secret_key = secret
pricing_dir = ""
backend = s3
aws_auth_sign_version = 4
aws_region = us-west-2

look like the parser was confused by previous line ?
pavelr

pavelr

2023-05-04 09:48

reporter   ~0005004

i tried every possible option during the 2 days i've spent on it (including what you've suggested) - it just fails with errors above.
none related question to the issues - i did manage to do a work around using another option - but not sure it's correct.
what i did is mount an backblaze b2 bucket locally using s3fs - and i just configure the bareos storage to work with local folder - it does work - however i did get a strange behavior while i briefly tested it - i did a backup which saved 4 files, but when i'm trying to restore it throws warning that it expected 4 files but restored only 3 - is it a known issue? does this considered a proper way of working with buckets? this issue happened with a specific backup where part of the backup were on actual local folder (full) and the other part (incremental) was on locally mounted b2 bucket. when i did a full backup and restore to and from the bucket it worked without issues. again i didn't tested it toughly so wanted to know if it's ok or not.
arogge

arogge

2023-05-10 14:52

manager   ~0005041

It's a long shot, but are you using a file with unix line endings?
Because if you had DOS/Windows line endings, then the value is "true<CR>", which is neither "true" nor "false" and thus an invalid value.
Maybe running "dos2unix" on your profile configuration file fixes the problem.
pavelr

pavelr

2023-05-16 08:45

reporter   ~0005061

well looks like it indeed was the issue :) however now i get a different error:
2023-05-16 06:43:07 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:376: init_ssl_conn: SSL certificate verification status: 0: ok
2023-05-16 06:43:07 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:373: init_ssl_conn: SSL connect error: -1 (error:00000001:lib(0)::reason(1))
2023-05-16 06:43:07 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:372: init_ssl_conn: SSL_connect failed with 40F6FF4F617F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:320:
glani

glani

2023-05-26 19:06

reporter   ~0005062

I can confirm the following:
Backend AWS does not work on 22.0.3 centos 9 either.
Docu is misleading and slightly incorrect. The region is not taken into consideration.
I have tried many ways: only one I got worked health check for storage: <bucket-name>.s3.amazonaws.com:443

*status
Status available for:
1: Director
2: Storage
3: Client
4: Scheduler
5: All
Select daemon type for status (1-5): 2
The defined Storage resources are:
1: File
2: S3_ObjectStorage_General_smblob_10.0.0.100
3: S3_ObjectStorage_smblob_10.0.0.3
4: S3_ObjectStorage_smblob_10.0.0.4
Select Storage resource (1-4): 2
Connecting to Storage daemon S3_ObjectStorage_General_smblob_10.0.0.100 at 10.0.0.100:9103
 Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3

bareos-sd Version: 22.0.3 (24 March 2023) CentOS Stream release 9
Daemon started 26-May-23 16:21. Jobs: run=1, running=0, self-compiled binary
 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 bwlimit=0kB/s

There are WARNINGS for this storagedaemon's configuration!
See output of 'bareos-sd -t' for details.

Running Jobs:
No Jobs running.
====

Jobs waiting to reserve a drive:
====

Device status:

Device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1) is not open.
Backend connection is working.
No pending IO flush requests.
==

Device "S3_ObjectDevice_General_smblob_10.0.0.100_2" (S3_ObjectDevice_General_smblob_2) is not open.
Backend connection is working.
No pending IO flush requests.
==
====

Used Volume status:
====

====


but when I do backup

I got :
26-May 16:36 bareos-dir JobId 22: No prior Full backup Job record found.
26-May 16:36 bareos-dir JobId 22: No prior or suitable Full backup found in catalog. Doing FULL backup.
26-May 16:36 bareos-dir JobId 22: Start Backup JobId 22, Job=bareos-dir-conf.2023-05-26_16.36.22_07
26-May 16:36 bareos-dir JobId 22: Connected Storage daemon at 10.0.0.100:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
26-May 16:36 bareos-dir JobId 22: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
26-May 16:36 bareos-dir JobId 22: Connected Client: client-fd-smblob_10-0-0-100 at 10.0.0.100:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
26-May 16:36 bareos-dir JobId 22: Handshake: Immediate TLS
26-May 16:36 bareos-dir JobId 22: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
26-May 16:36 bareos-dir JobId 22: Using Device "S3_ObjectDevice_General_smblob_10.0.0.100_1" to write.
26-May 16:36 client-fd-smblob_10-0-0-100 JobId 22: Connected Storage daemon at 10.0.0.100:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
26-May 16:36 client-fd-smblob_10-0-0-100 JobId 22: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
26-May 16:36 client-fd-smblob_10-0-0-100 JobId 22: Extended attribute support is enabled
26-May 16:36 client-fd-smblob_10-0-0-100 JobId 22: ACL support is enabled
26-May 16:36 bareos-sd JobId 22: Warning: Volume "Full-0010" not on device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1).
26-May 16:36 bareos-sd JobId 22: Marking Volume "Full-0010" in Error in Catalog.
26-May 16:36 bareos-sd JobId 22: Warning: Volume "Full-0010" not on device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1).
26-May 16:36 bareos-sd JobId 22: Marking Volume "Full-0010" in Error in Catalog.
26-May 16:36 bareos-dir JobId 22: Created new Volume "Full-0011" in catalog.
26-May 16:36 bareos-sd JobId 22: Labeled new Volume "Full-0011" on device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1).
26-May 16:36 bareos-sd JobId 22: Wrote label to prelabeled Volume "Full-0011" on device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1)
26-May 16:36 bareos-sd JobId 22: Releasing device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1).
26-May 16:36 bareos-sd JobId 22: Fatal error: Failed to flush device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1).
26-May 16:36 bareos-sd JobId 22: Elapsed time=00:00:30, Transfer rate=1.433 K Bytes/second
26-May 16:36 bareos-dir JobId 22: Insert of attributes batch table with 102 entries start
26-May 16:36 bareos-dir JobId 22: Insert of attributes batch table done
26-May 16:37 bareos-dir JobId 22: Error: Bareos bareos-dir 22.0.3 (24Mar23):

I googled a little bit looks like this issue has existed since the fork from bacula.

I have spent two days as well. I cannot mount s3fs because plans are backup postgres cluster with 3 nodes and only filedaemon is sitting on replica
glani

glani

2023-05-26 19:46

reporter   ~0005063

Looks like I found the issue.

The working config for the device profile is:

host = s3.amazonaws.com:443
use_https = true
backend = s3
aws_region = {{ aws_region }}
aws_auth_sign_version = 4
access_key = "{{ aws_access_id }}"
secret_key = "{{ aws_secret_key }}"
pricing_dir = ""

AWS bucket should have ACL to handle requests from API. The default creation of the bucket denies it (bareos hides this info because of no implementation here https://github.com/scality/Droplet). Better to test bucket and AWS creds on something independent. I tested via amazon ansible module. The same bucket and creds

The bucket is specified in the device itself. NO COMMENTS! After removing all comments from the profile it started working.
bruno-at-bareos

bruno-at-bareos

2023-08-22 20:42

manager   ~0005338

#glani, I'm wondering if you want to make the solution as a remark into documentation by doing a PR ?
bruno-at-bareos

bruno-at-bareos

2023-11-02 10:54

manager   ~0005492

Documentation will be fixed to remove comments in droplet profiles examples
https://github.com/bareos/bareos/pull/1581

Issue History

Date Modified Username Field Change
2023-04-30 20:58 pavelr New Issue
2023-04-30 21:05 pavelr Note Added: 0004994
2023-05-03 15:44 bruno-at-bareos Note Added: 0004998
2023-05-04 09:48 pavelr Note Added: 0005004
2023-05-10 14:52 arogge Status new => feedback
2023-05-10 14:52 arogge Note Added: 0005041
2023-05-16 08:45 pavelr Note Added: 0005061
2023-05-16 08:45 pavelr Status feedback => new
2023-05-26 19:06 glani Note Added: 0005062
2023-05-26 19:46 glani Note Added: 0005063
2023-08-22 20:42 bruno-at-bareos Assigned To => bruno-at-bareos
2023-08-22 20:42 bruno-at-bareos Status new => feedback
2023-08-22 20:42 bruno-at-bareos Note Added: 0005338
2023-11-02 10:54 bruno-at-bareos Status feedback => closed
2023-11-02 10:54 bruno-at-bareos Resolution open => fixed
2023-11-02 10:54 bruno-at-bareos Note Added: 0005492