Bareos Bug Tracker - bareos-core
View Issue Details
0000949bareos-core[All Projects] storage daemonpublic2018-05-17 16:572018-09-06 19:08
tymik 
 
normalmajoralways
resolvedfixed 
x86_64CentOS7
17.2.5 
18.3.1 
yes
fixed
no
no
no
no
no
no
0000949: S3 Droplet backend documentation missing
Please get the S3 Droplet documented, as currently it is barely documented and setting it up is a real struggle, everything is unknown and I personally find it impossible to set S3 Droplet to work with AWS S3 Bucket.
1. Try to find documentation
2. Find no documentation
<sarcasm>
3. ???
4. profit
</sarcasm>
the only "documentation wanna-be" is the comment in example droplet config, which is insufficient to call it documentation and it is not really helpful.
people on IRC channel are neither able to help.
aws, droplet, s3
child of 0000966closed joergs Release bareos-17.2.7 
? bareos-sd.trace (22,555) 2018-06-08 17:29
http://bugs.bareos.org/file_download.php?file_id=297&type=bug
Issue History
2018-05-17 16:57tymikNew Issue
2018-05-18 17:11stephandNote Added: 0003006
2018-05-21 21:04aron_sNote Added: 0003008
2018-05-22 10:39tymikNote Added: 0003010
2018-05-23 19:33aron_sNote Added: 0003016
2018-05-26 17:24aron_sNote Edited: 0003016bug_revision_view_page.php?bugnote_id=3016#r326
2018-06-08 17:27tymikNote Added: 0003036
2018-06-08 17:28tymikTag Attached: s3;droplet;aws
2018-06-08 17:28tymikTag Detached: s3;droplet;aws
2018-06-08 17:28tymikTag Attached: aw
2018-06-08 17:28tymikTag Attached: droplet
2018-06-08 17:28tymikTag Attached: s3
2018-06-08 17:29tymikTag Detached: aw
2018-06-08 17:29tymikTag Attached: aws
2018-06-08 17:29tymikFile Added: bareos-sd.trace
2018-06-13 13:29aron_sNote Added: 0003038
2018-06-13 16:31tymikNote Added: 0003039
2018-06-14 15:21aron_sNote Added: 0003044
2018-06-14 15:23aron_sNote Edited: 0003044bug_revision_view_page.php?bugnote_id=3044#r334
2018-06-14 15:24aron_sNote Edited: 0003044bug_revision_view_page.php?bugnote_id=3044#r335
2018-07-06 11:12tymikNote Added: 0003064
2018-07-16 19:05joergsChangeset attached => bareos master 96a8725b
2018-07-16 19:05joergsNote Added: 0003072
2018-07-16 19:05joergsStatusnew => resolved
2018-07-16 19:05joergsResolutionopen => fixed
2018-07-16 19:06joergsRelationship addedchild of 0000966
2018-07-16 19:07joergsbareos-master: impact => yes
2018-07-16 19:07joergsbareos-master: action => fixed
2018-07-16 19:07joergsbareos-17.2: impact => no
2018-07-16 19:07joergsbareos-16.2: impact => no
2018-07-16 19:07joergsbareos-15.2: impact => no
2018-07-16 19:07joergsbareos-14.2: impact => no
2018-07-16 19:07joergsbareos-13.2: impact => no
2018-07-16 19:07joergsbareos-12.4: impact => no
2018-07-16 19:07joergsStatusresolved => closed
2018-07-16 19:07joergsAssigned To => joergs
2018-07-16 19:07joergsFixed in Version => 18.3.1
2018-07-16 19:07joergsAssigned Tojoergs =>
2018-09-06 19:08frankuChangeset attached => bareos dev 3f34dc93
2018-09-06 19:08frankuNote Added: 0003104
2018-09-06 19:08frankuStatusclosed => resolved

Notes
(0003006)
stephand   
2018-05-18 17:11   
On the Bug Reporting Page, it says
Before reporting a bug, please read the Bug Reporting Howto:
https://www.bareos.org/en/HOWTO/articles/how-to-create-a-bugreport.html [^]

Here especially interesting:
"Also, please keep your language polite. You want somebody to help you with your problem and solve it, that is seldomly achieved by being impolite."
(0003008)
aron_s   
2018-05-21 21:04   
Take a look here. For a beta feature this contains more than enough documentation.
https://fossies.org/linux/bareos-Release/README.droplet [^]
(0003010)
tymik   
2018-05-22 10:39   
I have already seen this and it is not sufficient.
I didn't succeed setting connection to AWS S3 properly and seeing posts on bareos-users mailing group, I'm not the only one.
Nothing really is explained there.
According to what is stated in this draft, I have set up everything properly, so following that I am either wrong or there is a bug and droplet doesn't work.
But having it so barely documented I cannot say which one is right.
(0003016)
aron_s   
2018-05-23 19:33   
(edited on: 2018-05-26 17:24)
Please post your configuration and the output of storage daemon running with debug level set to 250 when running a job. Maybe I can help. I will also write some kind of readme for the bareos/S3 setup down.

(0003036)
tymik   
2018-06-08 17:27   
/etc/bareos/bareos-sd.d/device/droplet/droplet.profile:
use_https = False
host = my-bucket.s3.eu-west-2.amazonaws.com
access_key = "myaccesskey"
secret_key = "mysecretkey"
pricing_dir = ""
backend = s3
aws_auth_sign_version = 4

/etc/bareos/bareos-sd.d/device/S3-droplet-lon-001.conf:
Device {
  Name = S3Droplet-lon-001
  Media Type = s3bucket
  Archive Device = S3 Object Storage
  #
  # Config options:
  # profile= - Droplet profile to use either absolute PATH or logical name (e.g. ~/.droplet/<profile>.profile
  # location= - AWS location (e.g. us-east etc.)
  # acl= - Canned ACL
  # storageclass= - Storage Class to use.
  # bucket= - Bucket to store objects in.
  # chunksize= - Size of Volume Chunks (default = 10 Mb)
  # iothreads= - Number of IO-threads to use for uploads (use blocking uploads if not set.)
  # ioslots= - Number of IO-slots per IO-thread (default 10)
  # mmap - Use mmap to allocate Chunk memory instead of malloc().
  #
  Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/droplet.profile,bucket=my-bucket,iothreads=1,ioslots=1,location=eu-west-2"
  Device Type = droplet
  LabelMedia = yes # lets Bareos label unlabeled media
  Random Access = yes
  AutomaticMount = yes # when device opened, read it
  RemovableMedia = no
  AlwaysOpen = no
  Description = "S3 Object device. A connecting Director must have the same Name and MediaType."
  Maximum File Size = 200M # 500 MB (Allows for seeking to small portions of the Volume)
  Maximum Concurrent Jobs = 1
  #Maximum Spool Size = 15000M
}

I've tried manipulating iothreads, ioslots, bucket name, with/without location and Maximum Spool Size, different Maximum File Size and for profile I've tried different formats of URL, use_https = True and nothing has worked so far.

Will upload debug level 250 log file soon, can also provide more verbose log if needed.

I've also posted some debug in https://groups.google.com/forum/#!topic/bareos-users/8-qOrSvcI2E [^] when I was trying one of the varying configs.
(0003038)
aron_s   
2018-06-13 13:29   
Change your host in your droplet.profile to "s3.amazonaws.com", as the backend will insert the bucket and location parameters from your Device Options there.

But as I assume that you tested this before. What you can try is to test your droplet profile with dplsh, it is a shell tool that tests your profile and direct-connects to your cloud storage.
https://github.com/scality/Droplet-sh [^]
(0003039)
tymik   
2018-06-13 16:31   
No, I haven't tried to set it up like you say before, as in bareos-users group which I linked previously, you can see that thraizz has posted very similar configuration (using full URL bucket) and claimed that this setup works for him.
Also nothing from the short manual regarding droplet suggested that I could try this option, the manual even suggests the opposite.

As for the droplet-sh you mentioned - I don't really see how could this help me verifying the droplet.
I have successfully connected to that bucket with S3FS and with `aws s3 ls`, so the access to the bucket is fine.
If the droplet-sh would let me test my profile file, I don't see it at all as you say that bucket name and region is taken by backend from Device Options, so the droplet-sh would not be able to get these data if the 'host' line wouldn't contain them.

I have just tested following options:
 host = s3.eu-west-2.amazonaws.com
 host = s3.amazonaws.com
and it doesn't work, my bucket didn't register any traffic, so bareos didn't even try to contact it.
(0003044)
aron_s   
2018-06-14 15:21   
(edited on: 2018-06-14 15:24)
The "host=" argument in your droplet profile is prepended by "bucket=" and "location=" arguments in your device ressource. If the host is set as s3.eu-west-2-amazonaws.com bareos-sd will try to reach the bucket "my-bucket.s3-eu-west-2.my-bucket.s3.eu-west-2-amazonaws.com". thraizz's setup cant work.

I cannot see any droplet errors in your trace file, none the less I have created my own AWS S3 bucket and it seems as the resolving is still buggy - our internal version can reach and resolve the host right, while the same configuration fails with public version and the error: "Failed to lookup hostname ".s3.amazonaws.com": Unknown host".
Be sure to keep your version updated as there is already a working solution, it's just not out yet.

I am also writing a plugin documentation like we have it for the vmware plugin so that the configuration will be easier.

Sorry for the inconvenience and thanks for the feedback.

(0003064)
tymik   
2018-07-06 11:12   
ok, cool, that clarifies the setup, I will amend mine and it would be great if you updated this ticket when you roll out the version that fixes the issue so I can test it and come back with feedback on newer version.

still needs documentation though, as you see, the part you've written regarding the bucket URL construction is not described anywhere apart from this ticket and it is not really clear.
(0003072)
joergs   
2018-07-16 19:05   
Fix committed to bareos master branch with changesetid 8966.
(0003104)
franku   
2018-09-06 19:08   
Fix committed to bareos dev branch with changesetid 9356.