|Anonymous | Login | Signup for a new account||2018-08-16 02:35 CEST|
|My View | View Issues | Change Log | Roadmap | | Repositories | Search | My Account|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0000949||bareos-core||[All Projects] storage daemon||public||2018-05-17 16:57||2018-07-16 19:07|
|Target Version||Fixed in Version||18.3.1|
|Summary||0000949: S3 Droplet backend documentation missing|
|Description||Please get the S3 Droplet documented, as currently it is barely documented and setting it up is a real struggle, everything is unknown and I personally find it impossible to set S3 Droplet to work with AWS S3 Bucket.|
|Steps To Reproduce||1. Try to find documentation|
2. Find no documentation
|Additional Information||the only "documentation wanna-be" is the comment in example droplet config, which is insufficient to call it documentation and it is not really helpful.|
people on IRC channel are neither able to help.
|Tags||aws, droplet, s3|
|Attached Files||bareos-sd.trace [^] (22,555 bytes) 2018-06-08 17:29|
On the Bug Reporting Page, it says
Before reporting a bug, please read the Bug Reporting Howto:
Here especially interesting:
"Also, please keep your language polite. You want somebody to help you with your problem and solve it, that is seldomly achieved by being impolite."
Take a look here. For a beta feature this contains more than enough documentation.
I have already seen this and it is not sufficient.
I didn't succeed setting connection to AWS S3 properly and seeing posts on bareos-users mailing group, I'm not the only one.
Nothing really is explained there.
According to what is stated in this draft, I have set up everything properly, so following that I am either wrong or there is a bug and droplet doesn't work.
But having it so barely documented I cannot say which one is right.
edited on: 2018-05-26 17:24
Please post your configuration and the output of storage daemon running with debug level set to 250 when running a job. Maybe I can help. I will also write some kind of readme for the bareos/S3 setup down.
use_https = False
host = my-bucket.s3.eu-west-2.amazonaws.com
access_key = "myaccesskey"
secret_key = "mysecretkey"
pricing_dir = ""
backend = s3
aws_auth_sign_version = 4
Name = S3Droplet-lon-001
Media Type = s3bucket
Archive Device = S3 Object Storage
# Config options:
# profile= - Droplet profile to use either absolute PATH or logical name (e.g. ~/.droplet/<profile>.profile
# location= - AWS location (e.g. us-east etc.)
# acl= - Canned ACL
# storageclass= - Storage Class to use.
# bucket= - Bucket to store objects in.
# chunksize= - Size of Volume Chunks (default = 10 Mb)
# iothreads= - Number of IO-threads to use for uploads (use blocking uploads if not set.)
# ioslots= - Number of IO-slots per IO-thread (default 10)
# mmap - Use mmap to allocate Chunk memory instead of malloc().
Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/droplet.profile,bucket=my-bucket,iothreads=1,ioslots=1,location=eu-west-2"
Device Type = droplet
LabelMedia = yes # lets Bareos label unlabeled media
Random Access = yes
AutomaticMount = yes # when device opened, read it
RemovableMedia = no
AlwaysOpen = no
Description = "S3 Object device. A connecting Director must have the same Name and MediaType."
Maximum File Size = 200M # 500 MB (Allows for seeking to small portions of the Volume)
Maximum Concurrent Jobs = 1
#Maximum Spool Size = 15000M
I've tried manipulating iothreads, ioslots, bucket name, with/without location and Maximum Spool Size, different Maximum File Size and for profile I've tried different formats of URL, use_https = True and nothing has worked so far.
Will upload debug level 250 log file soon, can also provide more verbose log if needed.
I've also posted some debug in https://groups.google.com/forum/#!topic/bareos-users/8-qOrSvcI2E [^] when I was trying one of the varying configs.
Change your host in your droplet.profile to "s3.amazonaws.com", as the backend will insert the bucket and location parameters from your Device Options there.
But as I assume that you tested this before. What you can try is to test your droplet profile with dplsh, it is a shell tool that tests your profile and direct-connects to your cloud storage.
No, I haven't tried to set it up like you say before, as in bareos-users group which I linked previously, you can see that thraizz has posted very similar configuration (using full URL bucket) and claimed that this setup works for him.
Also nothing from the short manual regarding droplet suggested that I could try this option, the manual even suggests the opposite.
As for the droplet-sh you mentioned - I don't really see how could this help me verifying the droplet.
I have successfully connected to that bucket with S3FS and with `aws s3 ls`, so the access to the bucket is fine.
If the droplet-sh would let me test my profile file, I don't see it at all as you say that bucket name and region is taken by backend from Device Options, so the droplet-sh would not be able to get these data if the 'host' line wouldn't contain them.
I have just tested following options:
host = s3.eu-west-2.amazonaws.com
host = s3.amazonaws.com
and it doesn't work, my bucket didn't register any traffic, so bareos didn't even try to contact it.
edited on: 2018-06-14 15:24
The "host=" argument in your droplet profile is prepended by "bucket=" and "location=" arguments in your device ressource. If the host is set as s3.eu-west-2-amazonaws.com bareos-sd will try to reach the bucket "my-bucket.s3-eu-west-2.my-bucket.s3.eu-west-2-amazonaws.com". thraizz's setup cant work.
I cannot see any droplet errors in your trace file, none the less I have created my own AWS S3 bucket and it seems as the resolving is still buggy - our internal version can reach and resolve the host right, while the same configuration fails with public version and the error: "Failed to lookup hostname ".s3.amazonaws.com": Unknown host".
Be sure to keep your version updated as there is already a working solution, it's just not out yet.
I am also writing a plugin documentation like we have it for the vmware plugin so that the configuration will be easier.
Sorry for the inconvenience and thanks for the feedback.
ok, cool, that clarifies the setup, I will amend mine and it would be great if you updated this ticket when you roll out the version that fixes the issue so I can test it and come back with feedback on newer version.
still needs documentation though, as you see, the part you've written regarding the bucket URL construction is not described anywhere apart from this ticket and it is not really clear.
|Fix committed to bareos master branch with changesetid 8966.|
bareos: master 96a8725b
Timestamp: 2018-07-09 18:03:09
|doc: Create own chapter for Storage Backends
Rework chapter about Droplet Storage Backend.
Fixes 0000949: S3 Droplet backend documentation missing
|mod - docs/manuals/en/main/bareos-manual-main-reference.tex|
|mod - docs/manuals/en/main/bareos-sd-resource-device-definitions.tex|
|mod - docs/manuals/en/main/bareos.sty|
|rm - docs/manuals/en/main/plugins-droplet-plugin.tex|
|mod - docs/manuals/en/main/plugins.tex|
|add - docs/manuals/en/main/storage-backend-droplet.tex|
|add - docs/manuals/en/main/storage-backends.tex|
|2018-05-17 16:57||tymik||New Issue|
|2018-05-18 17:11||stephand||Note Added: 0003006|
|2018-05-21 21:04||aron_s||Note Added: 0003008|
|2018-05-22 10:39||tymik||Note Added: 0003010|
|2018-05-23 19:33||aron_s||Note Added: 0003016|
|2018-05-26 17:24||aron_s||Note Edited: 0003016||View Revisions|
|2018-06-08 17:27||tymik||Note Added: 0003036|
|2018-06-08 17:28||tymik||Tag Attached: s3;droplet;aws|
|2018-06-08 17:28||tymik||Tag Detached: s3;droplet;aws|
|2018-06-08 17:28||tymik||Tag Attached: aw|
|2018-06-08 17:28||tymik||Tag Attached: droplet|
|2018-06-08 17:28||tymik||Tag Attached: s3|
|2018-06-08 17:29||tymik||Tag Detached: aw|
|2018-06-08 17:29||tymik||Tag Attached: aws|
|2018-06-08 17:29||tymik||File Added: bareos-sd.trace|
|2018-06-13 13:29||aron_s||Note Added: 0003038|
|2018-06-13 16:31||tymik||Note Added: 0003039|
|2018-06-14 15:21||aron_s||Note Added: 0003044|
|2018-06-14 15:23||aron_s||Note Edited: 0003044||View Revisions|
|2018-06-14 15:24||aron_s||Note Edited: 0003044||View Revisions|
|2018-07-06 11:12||tymik||Note Added: 0003064|
|2018-07-16 19:05||joergs||Changeset attached||=> bareos master 96a8725b|
|2018-07-16 19:05||joergs||Note Added: 0003072|
|2018-07-16 19:05||joergs||Status||new => resolved|
|2018-07-16 19:05||joergs||Resolution||open => fixed|
|2018-07-16 19:06||joergs||Relationship added||child of 0000966|
|2018-07-16 19:07||joergs||bareos-master: impact||=> yes|
|2018-07-16 19:07||joergs||bareos-master: action||=> fixed|
|2018-07-16 19:07||joergs||bareos-17.2: impact||=> no|
|2018-07-16 19:07||joergs||bareos-16.2: impact||=> no|
|2018-07-16 19:07||joergs||bareos-15.2: impact||=> no|
|2018-07-16 19:07||joergs||bareos-14.2: impact||=> no|
|2018-07-16 19:07||joergs||bareos-13.2: impact||=> no|
|2018-07-16 19:07||joergs||bareos-12.4: impact||=> no|
|2018-07-16 19:07||joergs||Status||resolved => closed|
|2018-07-16 19:07||joergs||Assigned To||=> joergs|
|2018-07-16 19:07||joergs||Fixed in Version||=> 18.3.1|
|2018-07-16 19:07||joergs||Assigned To||joergs =>|
|Copyright © 2000 - 2018 MantisBT Team|