 |
sorry. mixed up the errors: #option 1: host = s3.amazonaws.com:443 #result 2023-04-30 17:31:32 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/profile.c:251: conf_cb_func: invalid value 'use_https' #option 2: host = s3.amazonaws.com #error: 2023-04-30 19:01:16 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/addrlist.c:600: dpl_addrlist_add: cannot lookup host s3.amazonaws.com |
|
|
Did you tried to remove the comment ? host = s3.amazonaws.com # This parameter is only used as baseurl and will be prepended with bucket and location set in device resource to form correct url use_https = true access_key = key secret_key = secret pricing_dir = "" # If not empty, an droplet.csv file will be created which will record all S3 operations. backend = s3 aws_auth_sign_version = 4 # Currently, AWS S3 uses version 4. The Ceph S3 gateway uses version 2. aws_region = us-west-2 to host = s3.amazonaws.com use_https = true access_key = key secret_key = secret pricing_dir = "" backend = s3 aws_auth_sign_version = 4 aws_region = us-west-2 look like the parser was confused by previous line ? |
|
|
i tried every possible option during the 2 days i've spent on it (including what you've suggested) - it just fails with errors above. none related question to the issues - i did manage to do a work around using another option - but not sure it's correct. what i did is mount an backblaze b2 bucket locally using s3fs - and i just configure the bareos storage to work with local folder - it does work - however i did get a strange behavior while i briefly tested it - i did a backup which saved 4 files, but when i'm trying to restore it throws warning that it expected 4 files but restored only 3 - is it a known issue? does this considered a proper way of working with buckets? this issue happened with a specific backup where part of the backup were on actual local folder (full) and the other part (incremental) was on locally mounted b2 bucket. when i did a full backup and restore to and from the bucket it worked without issues. again i didn't tested it toughly so wanted to know if it's ok or not. |
|
|
It's a long shot, but are you using a file with unix line endings? Because if you had DOS/Windows line endings, then the value is "true<CR>", which is neither "true" nor "false" and thus an invalid value. Maybe running "dos2unix" on your profile configuration file fixes the problem. |
|
|
well looks like it indeed was the issue :) however now i get a different error: 2023-05-16 06:43:07 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:376: init_ssl_conn: SSL certificate verification status: 0: ok 2023-05-16 06:43:07 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:373: init_ssl_conn: SSL connect error: -1 (error:00000001:lib(0)::reason(1)) 2023-05-16 06:43:07 bareos-sd: ERROR in backends/droplet_device.cc:109 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:372: init_ssl_conn: SSL_connect failed with 40F6FF4F617F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:320: |
|
|
I can confirm the following: Backend AWS does not work on 22.0.3 centos 9 either. Docu is misleading and slightly incorrect. The region is not taken into consideration. I have tried many ways: only one I got worked health check for storage: <bucket-name>.s3.amazonaws.com:443 *status Status available for: 1: Director 2: Storage 3: Client 4: Scheduler 5: All Select daemon type for status (1-5): 2 The defined Storage resources are: 1: File 2: S3_ObjectStorage_General_smblob_10.0.0.100 3: S3_ObjectStorage_smblob_10.0.0.3 4: S3_ObjectStorage_smblob_10.0.0.4 Select Storage resource (1-4): 2 Connecting to Storage daemon S3_ObjectStorage_General_smblob_10.0.0.100 at 10.0.0.100:9103 Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 bareos-sd Version: 22.0.3 (24 March 2023) CentOS Stream release 9 Daemon started 26-May-23 16:21. Jobs: run=1, running=0, self-compiled binary Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 bwlimit=0kB/s There are WARNINGS for this storagedaemon's configuration! See output of 'bareos-sd -t' for details. Running Jobs: No Jobs running. ==== Jobs waiting to reserve a drive: ==== Device status: Device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1) is not open. Backend connection is working. No pending IO flush requests. == Device "S3_ObjectDevice_General_smblob_10.0.0.100_2" (S3_ObjectDevice_General_smblob_2) is not open. Backend connection is working. No pending IO flush requests. == ==== Used Volume status: ==== ==== but when I do backup I got : 26-May 16:36 bareos-dir JobId 22: No prior Full backup Job record found. 26-May 16:36 bareos-dir JobId 22: No prior or suitable Full backup found in catalog. Doing FULL backup. 26-May 16:36 bareos-dir JobId 22: Start Backup JobId 22, Job=bareos-dir-conf.2023-05-26_16.36.22_07 26-May 16:36 bareos-dir JobId 22: Connected Storage daemon at 10.0.0.100:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 26-May 16:36 bareos-dir JobId 22: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 26-May 16:36 bareos-dir JobId 22: Connected Client: client-fd-smblob_10-0-0-100 at 10.0.0.100:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 26-May 16:36 bareos-dir JobId 22: Handshake: Immediate TLS 26-May 16:36 bareos-dir JobId 22: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 26-May 16:36 bareos-dir JobId 22: Using Device "S3_ObjectDevice_General_smblob_10.0.0.100_1" to write. 26-May 16:36 client-fd-smblob_10-0-0-100 JobId 22: Connected Storage daemon at 10.0.0.100:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 26-May 16:36 client-fd-smblob_10-0-0-100 JobId 22: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 26-May 16:36 client-fd-smblob_10-0-0-100 JobId 22: Extended attribute support is enabled 26-May 16:36 client-fd-smblob_10-0-0-100 JobId 22: ACL support is enabled 26-May 16:36 bareos-sd JobId 22: Warning: Volume "Full-0010" not on device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1). 26-May 16:36 bareos-sd JobId 22: Marking Volume "Full-0010" in Error in Catalog. 26-May 16:36 bareos-sd JobId 22: Warning: Volume "Full-0010" not on device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1). 26-May 16:36 bareos-sd JobId 22: Marking Volume "Full-0010" in Error in Catalog. 26-May 16:36 bareos-dir JobId 22: Created new Volume "Full-0011" in catalog. 26-May 16:36 bareos-sd JobId 22: Labeled new Volume "Full-0011" on device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1). 26-May 16:36 bareos-sd JobId 22: Wrote label to prelabeled Volume "Full-0011" on device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1) 26-May 16:36 bareos-sd JobId 22: Releasing device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1). 26-May 16:36 bareos-sd JobId 22: Fatal error: Failed to flush device "S3_ObjectDevice_General_smblob_10.0.0.100_1" (S3_ObjectDevice_General_smblob_1). 26-May 16:36 bareos-sd JobId 22: Elapsed time=00:00:30, Transfer rate=1.433 K Bytes/second 26-May 16:36 bareos-dir JobId 22: Insert of attributes batch table with 102 entries start 26-May 16:36 bareos-dir JobId 22: Insert of attributes batch table done 26-May 16:37 bareos-dir JobId 22: Error: Bareos bareos-dir 22.0.3 (24Mar23): I googled a little bit looks like this issue has existed since the fork from bacula. I have spent two days as well. I cannot mount s3fs because plans are backup postgres cluster with 3 nodes and only filedaemon is sitting on replica |
|
|
Looks like I found the issue. The working config for the device profile is: host = s3.amazonaws.com:443 use_https = true backend = s3 aws_region = {{ aws_region }} aws_auth_sign_version = 4 access_key = "{{ aws_access_id }}" secret_key = "{{ aws_secret_key }}" pricing_dir = "" AWS bucket should have ACL to handle requests from API. The default creation of the bucket denies it (bareos hides this info because of no implementation here https://github.com/scality/Droplet). Better to test bucket and AWS creds on something independent. I tested via amazon ansible module. The same bucket and creds The bucket is specified in the device itself. NO COMMENTS! After removing all comments from the profile it started working. |
|
|
#glani, I'm wondering if you want to make the solution as a remark into documentation by doing a PR ? |
|
|
Documentation will be fixed to remove comments in droplet profiles examples https://github.com/bareos/bareos/pull/1581 |
- Anonymous
