View Issue Details

IDProjectCategoryView StatusLast Update
0000277bareos-corestorage daemonpublic2014-05-16 17:50
Reporterjurgeng Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
Status closedResolutionsuspended 
PlatformLinuxOSanyOS Version3
Summary0000277: Backup to object store
DescriptionMore and more providers offer cloud storage in the form of an object store. These object stores are cheap and reliable storage and ideal for storing backups. Storage is offered via openstack swift or s3 emulation.

Currently, bareos cannot natively use object stores. Only via workarounds such as cloudfuse, that mounts an object store as a file systen, but this has its drawbacks and performance limitations.
Steps To ReproduceBareos is able to direct write backups to an object store and read from the object store when restoring.
TagsNo tags attached.

Activities

mvwieringen

mvwieringen

2014-02-11 16:43

developer   ~0000811

We get this feedback more and more and it has our attention. Also during
FOSDEM 2014 people asked a lot about "cloud" storage. We are currently doing
some first investigations. First things to tackle is finding a good library
that abstracts the inner working of things like S3 and swift etc.

We did our first steps this week writing a plugin for gluster which also
bypasses the FUSE layer. Nice things about that gluster API is that its
mostly a POSIX api with open/close/read/write which can be used without to
many problems after some little refactoring of the already existing device
abstraction in the SD.

Problem I see with most object stores is the need to do some local caching
because you need to upload blobs of data (and also download) which is currently
not such a great match with Bareos unless you threat things a tape (e.g. no
random access) and use spooling so we spool bigger chunks of data when
writing to the "volume"

Given you already did some investigation I would be interested in what you
tried and what your experiences are. We are looking in the direction of
libdroplet as that is a least a native C library unlike most solutions
which are Python/PHP/Name your other High level language which isn't such
a great match and programming on top of libcurl like what Amanda did with
their s3 device abstraction is not really something I would like to do.
So if there are good libraries put them on the table and we can see what
we can do there. Its on the radar only it depends a bit how well we can
abstract things if we will see anything usable any time soon.
jurgeng

jurgeng

2014-02-11 22:08

reporter   ~0000813

Here's a setup that I tested with:
- Mount the objectstore with cloudfuse. This way, it becomes a 'normal' filesystem.
I have had some issues with rights, I only got cloudfuse to access the objectstore for root. Running the bareos SD as root for a workaround.
- Let bareos SD write in this mounted filesystem
- Keep the volume size small. In my test I set the Volume size to be 1 GB. In the cloudfuse documentation, I read that eg appending to a file means that cloudfuse has to read the whole file in memory and write all back afterwards. You don't want to do this with very large files.

I tested with performing a backup. This seems to run fine. The log:

albatros-dir Start Backup JobId 1098, Job=regulier_ekster.2014-01-19_12.07.52_19
 Using Device "cloud01" to write.
cloud01 Volume "file-0065" previously written, moving to end of data.
 Ready to append to end of Volume "file-0065" size=16971104
cloud01 End of medium on Volume "file-0065" Bytes=1,073,677,562 Blocks=16,644 at 19-jan-2014 12:13.
 User defined maximum volume capacity 1,073,741,824 exceeded on device "cloud01" (/mnt/objectstore/ekster).
cloud01 Wrote label to prelabeled Volume "file-0066" on device "cloud01" (/mnt/objectstore/ekster)
 New volume "file-0066" mounted on device "cloud01" (/mnt/objectstore/ekster) at 19-jan-2014 12:14.
 Labeled new Volume "file-0066" on device "cloud01" (/mnt/objectstore/ekster).
albatros-dir Recycled volume "file-0066"
 All records pruned from Volume "file-0066"; marking it "Purged"
 There are no more Jobs associated with Volume "file-0066". Marking it purged.
cloud01 User defined maximum volume capacity 1,073,741,824 exceeded on device "cloud01" (/mnt/objectstore/ekster).
 End of medium on Volume "file-0066" Bytes=1,073,737,884 Blocks=16,644 at 19-jan-2014 12:18.
albatros-dir There are no more Jobs associated with Volume "file-0067". Marking it purged.
 Recycled volume "file-0067"
 All records pruned from Volume "file-0067"; marking it "Purged"
cloud01 Labeled new Volume "file-0067" on device "cloud01" (/mnt/objectstore/ekster).
cloud01 New volume "file-0067" mounted on device "cloud01" (/mnt/objectstore/ekster) at 19-jan-2014 12:18.
 Wrote label to prelabeled Volume "file-0067" on device "cloud01" (/mnt/objectstore/ekster)
cloud01 Elapsed time=00:14:04, Transfer rate=3.422 M Bytes/second
albatros-dir Bareos albatros-dir 12.4.4 (12Jun13):
  Build OS: x86_64-unknown-linux-gnu redhat Red Hat Enterprise Linux Server release 6.0 (Santiago)
  JobId: 1098
  Job: regulier_ekster.2014-01-19_12.07.52_19
  Backup Level: Full
  Client: "ekster" 12.4.4 (12Jun13) x86_64-unknown-linux-gnu,redhat,CentOS release 6.2 (Final)
  FileSet: "Full Set" 2014-01-19 11:58:35
  Pool: "File" (From command line)
  Catalog: "MyCatalog" (From Pool resource)
  Storage: "cloud01" (From command line)
  Scheduled time: 19-Jan-2014 12:07:36
  Start time: 19-Jan-2014 12:07:54
  End time: 19-Jan-2014 12:23:48
  Elapsed time: 15 mins 54 secs
  Priority: 10
  FD Files Written: 91,219
  SD Files Written: 91,219
  FD Bytes Written: 2,874,271,736 (2.874 GB)
  SD Bytes Written: 2,888,176,427 (2.888 GB)
  Rate: 3012.9 KB/s
  Software Compression: 43.0 %
  VSS: no
  Encryption: no
  Accurate: yes
  Volume name(s): file-0065|file-0066|file-0067
  Volume Session Id: 1
  Volume Session Time: 1390129644
  Last Volume Bytes: 763,042,833 (763.0 MB)
  Non-fatal FD errors: 0
  SD Errors: 0
  FD termination status: OK
  SD termination status: OK
  Termination: Backup OK

I also tested with a test restore of /var from those volumes. This also seems to run fine. The logs:

albatros-dir Start Restore Job RestoreFiles.2014-02-11_22.02.23_56
albatros-dir Using Device "cloud01" to read.
cloud01 Ready to read from volume "file-0066" on device "cloud01" (/mnt/objectstore/ekster).
cloud01 Forward spacing Volume "file-0066" to file:block 0:203.
cloud01 End of Volume at file 0 on device "cloud01" (/mnt/objectstore/ekster), Volume "file-0066"
cloud01 Ready to read from volume "file-0067" on device "cloud01" (/mnt/objectstore/ekster).
 Forward spacing Volume "file-0067" to file:block 0:203.
albatros-dir Bareos albatros-dir 12.4.4 (12Jun13):
  Build OS: x86_64-unknown-linux-gnu redhat Red Hat Enterprise Linux Server release 6.0 (Santiago)
  JobId: 1224
  Job: RestoreFiles.2014-02-11_22.02.23_56
  Restore Client: ekster
  Start time: 11-Feb-2014 22:02:25
  End time: 11-Feb-2014 22:04:45
  Files Expected: 34,635
  Files Restored: 34,635
  Bytes Restored: 2,029,278,159
  Rate: 14494.8 KB/s
  FD Errors: 0
  FD termination status: OK
  SD termination status: OK
  Termination: Restore OK
cloud01 End of Volume at file 0 on device "cloud01" (/mnt/objectstore/ekster), Volume "file-0067"
 End of all volumes.
albatros-dir Max configured use duration=604,800 sec. exceeded. Marking Volume "file-0067" as Used.

If you want me to do additional testing on this setup, please let me know.
mvwieringen

mvwieringen

2014-02-11 22:20

developer   ~0000814

I would think we better have a discussion about this on the devel
mailinglist so others can join in.

See http://www.bareos.org/en/howto-contribute.html

I had a brief look at the code for cloudfuse but it looks "interesting"
with stdio fopen etc. I got the feeling that libdroplet looks more promising
but who knows some people come up with better alternatives a search of
github and sourceforge doesn't show much for old school programming languages.
(Most is for Python and although we support plugins in Python that doesn't work
 for storage plugins and its also not the way I want to take it now so I would
 opt for a native C or C++ library.)

Issue History

Date Modified Username Field Change
2014-02-11 15:51 jurgeng New Issue
2014-02-11 16:43 mvwieringen Note Added: 0000811
2014-02-11 16:43 mvwieringen Assigned To => mvwieringen
2014-02-11 16:43 mvwieringen Status new => feedback
2014-02-11 22:08 jurgeng Note Added: 0000813
2014-02-11 22:08 jurgeng Status feedback => assigned
2014-02-11 22:20 mvwieringen Note Added: 0000814
2014-02-11 22:20 mvwieringen Status assigned => feedback
2014-02-12 21:41 tigerfoot Assigned To mvwieringen => tigerfoot
2014-02-12 21:41 tigerfoot Status feedback => assigned
2014-02-12 21:42 tigerfoot Assigned To tigerfoot => mvwieringen
2014-05-16 17:50 mvwieringen Assigned To mvwieringen =>
2014-05-16 17:50 mvwieringen Status assigned => closed
2014-05-16 17:50 mvwieringen Resolution open => suspended