View Issue Details

IDProjectCategoryView StatusLast Update
0000949bareos-core[All Projects] storage daemonpublic2019-01-31 10:23
ReportertymikAssigned Toarogge_adm 
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Platformx86_64OSCentOSOS Version7
Product Version17.2.5 
Target VersionFixed in Version18.3.1 
Summary0000949: S3 Droplet backend documentation missing
DescriptionPlease get the S3 Droplet documented, as currently it is barely documented and setting it up is a real struggle, everything is unknown and I personally find it impossible to set S3 Droplet to work with AWS S3 Bucket.
Steps To Reproduce1. Try to find documentation
2. Find no documentation
<sarcasm>
3. ???
4. profit
</sarcasm>
Additional Informationthe only "documentation wanna-be" is the comment in example droplet config, which is insufficient to call it documentation and it is not really helpful.
people on IRC channel are neither able to help.
Tagsaws, droplet, s3
bareos-master: impactyes
bareos-master: actionfixed
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impactno
bareos-17.2: action
bareos-16.2: impactno
bareos-16.2: action
bareos-15.2: impactno
bareos-15.2: action
bareos-14.2: impactno
bareos-14.2: action
bareos-13.2: impactno
bareos-13.2: action
bareos-12.4: impactno
bareos-12.4: action

Relationships

child of 0000966 closedjoergs Release bareos-17.2.7 
child of 0001040 resolvedpstorz Release bareos-18.2.5 

Activities

stephand

stephand

2018-05-18 17:11

developer   ~0003006

On the Bug Reporting Page, it says
Before reporting a bug, please read the Bug Reporting Howto:
https://www.bareos.org/en/HOWTO/articles/how-to-create-a-bugreport.html

Here especially interesting:
"Also, please keep your language polite. You want somebody to help you with your problem and solve it, that is seldomly achieved by being impolite."
aron_s

aron_s

2018-05-21 21:04

developer   ~0003008

Take a look here. For a beta feature this contains more than enough documentation.
https://fossies.org/linux/bareos-Release/README.droplet
tymik

tymik

2018-05-22 10:39

reporter   ~0003010

I have already seen this and it is not sufficient.
I didn't succeed setting connection to AWS S3 properly and seeing posts on bareos-users mailing group, I'm not the only one.
Nothing really is explained there.
According to what is stated in this draft, I have set up everything properly, so following that I am either wrong or there is a bug and droplet doesn't work.
But having it so barely documented I cannot say which one is right.
aron_s

aron_s

2018-05-23 19:33

developer   ~0003016

Last edited: 2018-05-26 17:24

View 2 revisions

Please post your configuration and the output of storage daemon running with debug level set to 250 when running a job. Maybe I can help. I will also write some kind of readme for the bareos/S3 setup down.

tymik

tymik

2018-06-08 17:27

reporter   ~0003036

/etc/bareos/bareos-sd.d/device/droplet/droplet.profile:
use_https = False
host = my-bucket.s3.eu-west-2.amazonaws.com
access_key = "myaccesskey"
secret_key = "mysecretkey"
pricing_dir = ""
backend = s3
aws_auth_sign_version = 4

/etc/bareos/bareos-sd.d/device/S3-droplet-lon-001.conf:
Device {
  Name = S3Droplet-lon-001
  Media Type = s3bucket
  Archive Device = S3 Object Storage
  #
  # Config options:
  # profile= - Droplet profile to use either absolute PATH or logical name (e.g. ~/.droplet/<profile>.profile
  # location= - AWS location (e.g. us-east etc.)
  # acl= - Canned ACL
  # storageclass= - Storage Class to use.
  # bucket= - Bucket to store objects in.
  # chunksize= - Size of Volume Chunks (default = 10 Mb)
  # iothreads= - Number of IO-threads to use for uploads (use blocking uploads if not set.)
  # ioslots= - Number of IO-slots per IO-thread (default 10)
  # mmap - Use mmap to allocate Chunk memory instead of malloc().
  #
  Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/droplet.profile,bucket=my-bucket,iothreads=1,ioslots=1,location=eu-west-2"
  Device Type = droplet
  LabelMedia = yes # lets Bareos label unlabeled media
  Random Access = yes
  AutomaticMount = yes # when device opened, read it
  RemovableMedia = no
  AlwaysOpen = no
  Description = "S3 Object device. A connecting Director must have the same Name and MediaType."
  Maximum File Size = 200M # 500 MB (Allows for seeking to small portions of the Volume)
  Maximum Concurrent Jobs = 1
  #Maximum Spool Size = 15000M
}

I've tried manipulating iothreads, ioslots, bucket name, with/without location and Maximum Spool Size, different Maximum File Size and for profile I've tried different formats of URL, use_https = True and nothing has worked so far.

Will upload debug level 250 log file soon, can also provide more verbose log if needed.

I've also posted some debug in https://groups.google.com/forum/#!topic/bareos-users/8-qOrSvcI2E when I was trying one of the varying configs.
tymik

tymik

2018-06-08 17:29

reporter  

bareos-sd.trace (22,555 bytes)
aron_s

aron_s

2018-06-13 13:29

developer   ~0003038

Change your host in your droplet.profile to "s3.amazonaws.com", as the backend will insert the bucket and location parameters from your Device Options there.

But as I assume that you tested this before. What you can try is to test your droplet profile with dplsh, it is a shell tool that tests your profile and direct-connects to your cloud storage.
https://github.com/scality/Droplet-sh
tymik

tymik

2018-06-13 16:31

reporter   ~0003039

No, I haven't tried to set it up like you say before, as in bareos-users group which I linked previously, you can see that thraizz has posted very similar configuration (using full URL bucket) and claimed that this setup works for him.
Also nothing from the short manual regarding droplet suggested that I could try this option, the manual even suggests the opposite.

As for the droplet-sh you mentioned - I don't really see how could this help me verifying the droplet.
I have successfully connected to that bucket with S3FS and with `aws s3 ls`, so the access to the bucket is fine.
If the droplet-sh would let me test my profile file, I don't see it at all as you say that bucket name and region is taken by backend from Device Options, so the droplet-sh would not be able to get these data if the 'host' line wouldn't contain them.

I have just tested following options:
 host = s3.eu-west-2.amazonaws.com
 host = s3.amazonaws.com
and it doesn't work, my bucket didn't register any traffic, so bareos didn't even try to contact it.
aron_s

aron_s

2018-06-14 15:21

developer   ~0003044

Last edited: 2018-06-14 15:24

View 3 revisions

The "host=" argument in your droplet profile is prepended by "bucket=" and "location=" arguments in your device ressource. If the host is set as s3.eu-west-2-amazonaws.com bareos-sd will try to reach the bucket "my-bucket.s3-eu-west-2.my-bucket.s3.eu-west-2-amazonaws.com". thraizz's setup cant work.

I cannot see any droplet errors in your trace file, none the less I have created my own AWS S3 bucket and it seems as the resolving is still buggy - our internal version can reach and resolve the host right, while the same configuration fails with public version and the error: "Failed to lookup hostname ".s3.amazonaws.com": Unknown host".
Be sure to keep your version updated as there is already a working solution, it's just not out yet.

I am also writing a plugin documentation like we have it for the vmware plugin so that the configuration will be easier.

Sorry for the inconvenience and thanks for the feedback.

tymik

tymik

2018-07-06 11:12

reporter   ~0003064

ok, cool, that clarifies the setup, I will amend mine and it would be great if you updated this ticket when you roll out the version that fixes the issue so I can test it and come back with feedback on newer version.

still needs documentation though, as you see, the part you've written regarding the bucket URL construction is not described anywhere apart from this ticket and it is not really clear.
joergs

joergs

2018-07-16 19:05

administrator   ~0003072

Fix committed to bareos master branch with changesetid 8966.
franku

franku

2018-09-06 19:08

developer   ~0003104

Fix committed to bareos dev branch with changesetid 9356.

Related Changesets

bareos: master 96a8725b

2018-07-09 18:03:09

joergs

Ported: N/A

Details Diff
doc: Create own chapter for Storage Backends

Rework chapter about Droplet Storage Backend.

Fixes 0000949: S3 Droplet backend documentation missing
Affected Issues
0000949
mod - docs/manuals/en/main/bareos-manual-main-reference.tex Diff File
mod - docs/manuals/en/main/bareos-sd-resource-device-definitions.tex Diff File
mod - docs/manuals/en/main/bareos.sty Diff File
rm - docs/manuals/en/main/plugins-droplet-plugin.tex Diff File
mod - docs/manuals/en/main/plugins.tex Diff File
add - docs/manuals/en/main/storage-backend-droplet.tex Diff File
add - docs/manuals/en/main/storage-backends.tex Diff File

bareos: dev 3f34dc93

2018-07-09 18:03:09

joergs


Committer: franku

Ported: N/A

Details Diff
doc: Create own chapter for Storage Backends

Rework chapter about Droplet Storage Backend.

Fixes 0000949: S3 Droplet backend documentation missing
Affected Issues
0000949
mod - docs/manuals/en/main/bareos-manual-main-reference.tex Diff File
mod - docs/manuals/en/main/bareos-sd-resource-device-definitions.tex Diff File
mod - docs/manuals/en/main/bareos.sty Diff File
rm - docs/manuals/en/main/plugins-droplet-plugin.tex Diff File
mod - docs/manuals/en/main/plugins.tex Diff File
add - docs/manuals/en/main/storage-backend-droplet.tex Diff File
add - docs/manuals/en/main/storage-backends.tex Diff File

Issue History

Date Modified Username Field Change
2018-05-17 16:57 tymik New Issue
2018-05-18 17:11 stephand Note Added: 0003006
2018-05-21 21:04 aron_s Note Added: 0003008
2018-05-22 10:39 tymik Note Added: 0003010
2018-05-23 19:33 aron_s Note Added: 0003016
2018-05-26 17:24 aron_s Note Edited: 0003016 View Revisions
2018-06-08 17:27 tymik Note Added: 0003036
2018-06-08 17:28 tymik Tag Attached: s3;droplet;aws
2018-06-08 17:28 tymik Tag Detached: s3;droplet;aws
2018-06-08 17:28 tymik Tag Attached: aw
2018-06-08 17:28 tymik Tag Attached: droplet
2018-06-08 17:28 tymik Tag Attached: s3
2018-06-08 17:29 tymik Tag Detached: aw
2018-06-08 17:29 tymik Tag Attached: aws
2018-06-08 17:29 tymik File Added: bareos-sd.trace
2018-06-13 13:29 aron_s Note Added: 0003038
2018-06-13 16:31 tymik Note Added: 0003039
2018-06-14 15:21 aron_s Note Added: 0003044
2018-06-14 15:23 aron_s Note Edited: 0003044 View Revisions
2018-06-14 15:24 aron_s Note Edited: 0003044 View Revisions
2018-07-06 11:12 tymik Note Added: 0003064
2018-07-16 19:05 joergs Changeset attached => bareos master 96a8725b
2018-07-16 19:05 joergs Note Added: 0003072
2018-07-16 19:05 joergs Status new => resolved
2018-07-16 19:05 joergs Resolution open => fixed
2018-07-16 19:06 joergs Relationship added child of 0000966
2018-07-16 19:07 joergs bareos-master: impact => yes
2018-07-16 19:07 joergs bareos-master: action => fixed
2018-07-16 19:07 joergs bareos-17.2: impact => no
2018-07-16 19:07 joergs bareos-16.2: impact => no
2018-07-16 19:07 joergs bareos-15.2: impact => no
2018-07-16 19:07 joergs bareos-14.2: impact => no
2018-07-16 19:07 joergs bareos-13.2: impact => no
2018-07-16 19:07 joergs bareos-12.4: impact => no
2018-07-16 19:07 joergs Status resolved => closed
2018-07-16 19:07 joergs Assigned To => joergs
2018-07-16 19:07 joergs Fixed in Version => 18.3.1
2018-07-16 19:07 joergs Assigned To joergs =>
2018-09-06 19:08 franku Changeset attached => bareos dev 3f34dc93
2018-09-06 19:08 franku Note Added: 0003104
2018-09-06 19:08 franku Status closed => resolved
2019-01-31 10:11 arogge_adm Relationship added child of 0001040
2019-01-31 10:23 arogge_adm Assigned To => arogge_adm
2019-01-31 10:23 arogge_adm Status resolved => closed