Changesets: bareos

Search Browse Back to Index

master d31cd38e

2017-08-14 12:35

pstorz

Ported: N/A

Details Diff		 NDMP: add support for 2-way NDMP with DAR and DDAR

The current implementation of NDMP in Bareos has some limitations
because of the way it was implemented.

- The backup is always sent over the network from the Data Agent to the
Tape Agent (3-way NDMP)
- DAR(Direct Access Recovery) and DDAR (Directory DAR) are not supported,
single file restores are VERY slow as the whole backup stream needs to
be scanned during restore.

This commit adds support for 2-way NDMP backups inside of the Director.
During 2-way NDMP Backups, the Director is the only bareos daemon involved,
the Storage Daemon does not do anything in this setup.

To be able to do 2-way NDMP backups and to support DAR and DDAR, the
director
- controls the NDMP changer via NDMP Protocol.
- does the bookkeeping for the NDMP media.
- stores the full NDMP file information being sent during backups (FhInfo and FhNode)
in the catalog database.
- sends back the full NDMP file information during restore to enable DAR
and DDAR.

The advantage of the 2-way NDMP backups are:
- Much higher backup speed as the data goes directly from the disks to
the tape without going over the network.
- Much higher restore speed for single files/directories with DAR and
DDAR, as only the required blocks are read during restore instead of
restoring the whole backup stream and only restoring the required
data.

The following NDMP Protocol keywords are now available:

* NDMP : which is an alias for NDMP_BAREOS for backward compatibility
* NDMP_BAREOS: which is NDMP backups with paired storage and NDMP Tape
Agent inside of the Bareos Storage Daemon
* NDMP_NATIVE: this is doing NDMP backups with only the DMA inside of
the Bareos Director without any connection to the SD

These keywords reflect the two ways of NDMP that are now supported in
Bareos, which is either the special way of NDMP backups implemented in
Bareos before, or the "NATIVE" way to do NDMP backups.

To be able to store the FileHandle information for DAR and DDAR, the
File table how has two additional columns for FhInfo and FhNode.
The JobMedia table how has the additional column JobBytes which is used
to store NDMP media Information.

This column is planned to hold the JobBytes that are stored in the
corresponding JobMedia record.

The ndmp_dma_backup and ndmp_dma_restore files were split up for the two
different ways of NDMP backups that we have now.
mod - src/cats/cats.h Diff File
mod - src/cats/ddl/creates/mysql.sql Diff File
mod - src/cats/ddl/creates/postgresql.sql Diff File
mod - src/cats/ddl/creates/sqlite3.sql Diff File
mod - src/cats/ddl/updates/mysql.2004_2170.sql Diff File
add - src/cats/ddl/updates/mysql.2170_2171.sql Diff File
add - src/cats/ddl/updates/postgresql.2170_2171.sql Diff File
add - src/cats/ddl/updates/sqlite3.2170_2171.sql Diff File
mod - src/cats/dml/0034_select_recent_version_with_basejob Diff File
mod - src/cats/dml/0034_select_recent_version_with_basejob.postgresql Diff File
mod - src/cats/dml/0035_select_recent_version_with_basejob_and_delta Diff File
mod - src/cats/dml/0035_select_recent_version_with_basejob_and_delta.postgresql Diff File
mod - src/cats/dml/0036_select_recent_version Diff File
mod - src/cats/dml/0036_select_recent_version.postgresql Diff File
mod - src/cats/dml/0039_create_temp_new_basefile Diff File
mod - src/cats/dml/0039_create_temp_new_basefile.ingres Diff File
mod - src/cats/ingres.c Diff File
mod - src/cats/mysql.c Diff File
mod - src/cats/mysql_queries.inc Diff File
mod - src/cats/postgresql.c Diff File
mod - src/cats/postgresql_queries.inc Diff File
mod - src/cats/protos.h Diff File
mod - src/cats/sql_create.c Diff File
mod - src/cats/sql_get.c Diff File
mod - src/cats/sql_pooling.c Diff File
mod - src/cats/sqlite.c Diff File
mod - src/cats/sqlite_queries.inc Diff File
mod - src/dird/Makefile.in Diff File
mod - src/dird/backup.c Diff File
mod - src/dird/catreq.c Diff File
mod - src/dird/dird_conf.c Diff File
mod - src/dird/job.c Diff File
add - src/dird/ndmp_dma_backup_NDMP_NATIVE.c Diff File
add - src/dird/ndmp_dma_backup_common.c Diff File
mod - src/dird/ndmp_dma_generic.c Diff File
mod - src/dird/ndmp_dma_priv.h Diff File
add - src/dird/ndmp_dma_restore_NDMP_NATIVE.c Diff File
add - src/dird/ndmp_dma_restore_common.c Diff File
mod - src/dird/ndmp_dma_storage.c Diff File
mod - src/dird/ndmp_fhdb_common.c Diff File
mod - src/dird/ndmp_fhdb_helpers.c Diff File
mod - src/dird/ndmp_fhdb_lmdb.c Diff File
add - src/dird/ndmp_ndmmedia_db_helpers.c Diff File
mod - src/dird/protos.h Diff File
mod - src/dird/restore.c Diff File
mod - src/dird/storage.c Diff File
mod - src/dird/ua_restore.c Diff File
mod - src/dird/ua_tree.c Diff File
mod - src/include/jcr.h Diff File
mod - src/lib/tree.h Diff File
mod - src/ndmp/Makefile.in Diff File
mod - src/ndmp/ndma_comm_job.c Diff File
mod - src/ndmp/ndma_comm_session.c Diff File
mod - src/ndmp/ndma_control.c Diff File
mod - src/ndmp/ndma_cops_backreco.c Diff File
mod - src/ndmp/ndma_cops_robot.c Diff File
mod - src/ndmp/ndma_ctrl_media.c Diff File
mod - src/ndmp/ndma_ctrl_robot.c Diff File
mod - src/ndmp/ndma_data_fh.c Diff File
mod - src/ndmp/ndma_listmgmt.c Diff File
mod - src/ndmp/ndma_tape.c Diff File
mod - src/ndmp/ndmagents.h Diff File
mod - src/ndmp/ndmjob_args.c Diff File
mod - src/ndmp/ndmjob_main.c Diff File
mod - src/ndmp/ndmjob_simulator.c Diff File
mod - src/ndmp/ndml_chan.c Diff File
mod - src/ndmp/ndml_conn.c Diff File
mod - src/ndmp/ndmos.h Diff File
mod - src/ndmp/ndmp4_pp.c Diff File
mod - src/stored/bscan.c Diff File
mod - src/stored/bsr.c Diff File
mod - src/stored/dir_cmd.c Diff File
mod - src/stored/job.c Diff File
mod - src/win32/dird/Makefile Diff File

bareos-17.2 29733cc2

2017-08-08 14:53

mvwieringen


Committer: pstorz

Ported: N/A

Details Diff		 build: Add support for object storage backend
mod - platforms/packaging/bareos.spec Diff File
mod - src/stored/backends/Makefile.in Diff File

bareos-16.2-droplet f6a23af2

2017-08-08 14:53

mvwieringen

Ported: N/A

Details Diff		 build: Add support for object storage backend
mod - platforms/packaging/bareos.spec Diff File
mod - src/stored/backends/Makefile.in Diff File

bareos-17.2 4992a4d3

2017-08-08 13:32

mvwieringen


Committer: pstorz

Ported: N/A

Details Diff		 stored: Add example configs for objectstorage device.
add - src/stored/backends/object_store_device.d/bareos-dir.d/storage/Object.conf.example Diff File
add - src/stored/backends/object_store_device.d/bareos-sd.d/device/ObjectStorage.conf.example Diff File

bareos-16.2-droplet 26b22dcd

2017-08-08 13:32

mvwieringen

Ported: N/A

Details Diff		 stored: Add example configs for objectstorage device.
add - src/stored/backends/object_store_device.d/bareos-dir.d/storage/Object.conf.example Diff File
add - src/stored/backends/object_store_device.d/bareos-sd.d/device/ObjectStorage.conf.example Diff File

bareos-16.2 674a5296

2017-08-02 23:06

medicmomcilo


Committer: joergs

Ported: N/A

Details Diff		 Fix crash when debugging output is enabled

Fixes 0000826: Enabling debugging output crashes Director		 Affected Issues
0000826
mod - AUTHORS Diff File
mod - src/dird/job.c Diff File

bareos-17.2 35b2fc08

2017-08-01 19:38

Aron Schüler


Committer: joergs

Ported: N/A

Details Diff		 Fixes Mac Os X build
mod - platforms/osx/Makefile.in Diff File
mod - platforms/osx/README Diff File
mod - platforms/osx/files/org.bareos.bareos-fd.plist.in Diff File
mod - platforms/osx/resources/ReadMe.html.in Diff File
mod - platforms/osx/resources/postflight.in Diff File
mod - scripts/bareos-config-lib.sh.in Diff File

bareos-17.2 82576dd3

2017-08-01 15:01

joergs

Ported: N/A

Details Diff		 configure
mod - configure Diff File

bareos-17.2 e1c434b2

2017-08-01 14:59

joergs

Ported: N/A

Details Diff		 add bareos-regress-config package

This package contains required file for running Bareos regression tests
for the bareos-regress package.
mod - GNUmakefile.in Diff File
mod - Makefile.in Diff File
mod - autoconf/configure.in Diff File
add - debian/bareos-regress-config.install.in Diff File
add - debian/bareos-regress-config.preinst.in Diff File
mod - debian/control Diff File
mod - debian/control.in Diff File
mod - platforms/packaging/bareos.spec Diff File
mod - scripts/Makefile.in Diff File
mod - scripts/bareos-ctl-dir.in Diff File
mod - scripts/bareos-ctl-fd.in Diff File
mod - scripts/bareos-ctl-funcs Diff File
mod - scripts/bareos-ctl-sd.in Diff File
add - scripts/bareos-regress.conf.in Diff File
mod - scripts/bareos.in Diff File
mod - src/stored/Makefile.in Diff File
mod - src/tests/Makefile.in Diff File
mod - src/tests/cats_test.c Diff File
mod - src/tests/ing_test.c Diff File
mod - src/tools/Makefile.in Diff File

master 621c2f84

2017-07-27 15:00

joergs

Ported: N/A

Details Diff		 bls: show SHA signatures
mod - src/stored/record.c Diff File

bareos-16.2 38d0aec7

2017-07-26 23:57

joergs

Ported: N/A

Details Diff		 bugfix: prevents file corruptions by SHA1

Fixes a bug when using SHA1 file signatures in Bareos version compiled
without openssl.

Fixes 0000838: File corruption with SHA1 signature		 Affected Issues
0000838
mod - src/lib/sha1.c Diff File

master 452a8a4e

2017-07-21 19:15

stephand

Ported: N/A

Details Diff		 Fix .bvfs_lsdirs: Don't show empty dirs from accurate jobs

The SQL Query used by .bvfs_lsdirs did not consider the FileIndex
column, which is 0 for deleted directories in accurate backup jobs,
so that empty directories were returned by .bvfs_lsdirs.

This commit also contains an optimized index for the above SQL query,
without such an index .bvfs_lsdirs could be very slow on large
databases.

Also all variable length columns of the File table have been moved
to the end, this may allow the database to optimize the physical
layout.
mod - src/cats/bdb_query_enum_class.h Diff File
mod - src/cats/bdb_query_names.inc Diff File
mod - src/cats/bvfs.c Diff File
mod - src/cats/ddl/creates/mysql.sql Diff File
mod - src/cats/ddl/creates/postgresql.sql Diff File
mod - src/cats/ddl/updates/mysql.2004_2170.sql Diff File
mod - src/cats/ddl/updates/postgresql.2004_2170.sql Diff File
mod - src/cats/mysql_queries.inc Diff File
mod - src/cats/postgresql_queries.inc Diff File
mod - src/cats/sqlite_queries.inc Diff File

master 96266afb

2017-07-07 13:06

joergs

Ported: N/A

Details Diff		 Merge remote-tracking branch 'origin/bareos-16.2'
mod - .travis.yml Diff File
mod - configure Diff File
mod - debian/control Diff File
mod - debian/control.in Diff File
add - platforms/packaging/bareos-Univention_4.2.dsc Diff File
mod - platforms/packaging/bareos.dsc Diff File
mod - platforms/packaging/bareos.spec Diff File
mod - platforms/systemd/bareos-sd.service.in Diff File
mod - scripts/git-info.sh Diff File
mod - src/cats/sql_find.c Diff File
mod - src/dird/dird_conf.c Diff File
mod - src/dird/sd_cmds.c Diff File
mod - src/dird/ua_label.c Diff File
mod - src/dird/ua_purge.c Diff File
mod - src/dird/ua_status.c Diff File
mod - src/include/version.h Diff File
mod - src/qt-tray-monitor/bareos-tray-monitor.xpm Diff File
mod - src/stored/spool.c Diff File

master 49b543f7

2017-07-06 19:35

joergs

Ported: N/A

Details Diff		 show console command: fix help string
mod - src/dird/ua_cmds.c Diff File

master 841ca8f7

2017-07-06 13:44

Torsten Ueberschar


Committer: pstorz

Ported: N/A

Details Diff		 Add support for OpenSSL 1.1

OpenSSL 1.1 has quite some changes in the API, and this patch
enables to use the new API and still supports the old API as well.
mod - src/lib/crypto_openssl.c Diff File
mod - src/lib/tls_openssl.c Diff File

master 70587830

2017-07-06 00:12

joergs

Ported: N/A

Details Diff		 purge console command: add confirmation
mod - src/dird/ua_input.c Diff File
mod - src/dird/ua_purge.c Diff File

bareos-17.2 dfc67f17

2017-07-02 01:22

mvwieringen


Committer: pstorz

Ported: N/A

Details Diff		 lib: Add edit_pthread() function to print pthread_t ids.

There is no real clean way to print the pthread_t type returned by
various pthreads functions. Until now things were printed as pointer
addresses. The edit_pthread() method is analog to other edit functions
and tries to print things in an uniform way and should be used to print
the content of the pthread_t opaque structure. The benefit of this is
that the representation is in one central place and we can implement
specific representations for different platforms.

As part of the implementation of this new function the edit.c source
file was also refactored and reindented to be better readable. Further
more now all modifiers are stored as static const char pointers which
used to be done for some functions but also not in others.
mod - src/lib/btimers.c Diff File
mod - src/lib/edit.c Diff File
mod - src/lib/jcr.c Diff File
mod - src/lib/lockmgr.c Diff File
mod - src/lib/protos.h Diff File
mod - src/lib/res.c Diff File
mod - src/stored/lock.c Diff File

bareos-17.2 ba2cafbe

2017-07-02 01:22

mvwieringen


Committer: pstorz

Ported: N/A

Details Diff		 stored: Use SEEK_CUR not SEEK_END to check positioning.

When we want to determine if the position is set right in
DCR::is_eod_valid() we need to perform a lseek(fd, 0, SEEK_CUR)
and not a lseek(fd, 0, SEEK_END) as that is already done in
DEVICE::eod(). We only want to determine if the lseek was done
right in DCR::is_eod_valid() and by doing a SEEK_END we do the
same seek twice which leads to the same result but is not correct.
mod - src/stored/mount.c Diff File

bareos-17.2 80dc3bee

2017-07-02 01:22

mvwieringen


Committer: pstorz

Ported: N/A

Details Diff		 stored: Add device specific status trigger.

This adds code that allows you to trigger a device to return specific
device status information analog to the way which was already available
to plugins using the bsdEventDriveStatus and bsdEventVolumeStatus
events.

In essence this is equivalent to the bsdEventDriveStatus which allows a
plugin (for example the scsicrypto-sd plugin to return the crypto status
of an LTO4+ drive.) but then for any device without the need to have a
plugin loaded for the specific device.
mod - src/stored/dev.h Diff File
mod - src/stored/sd_plugins.h Diff File
mod - src/stored/status.c Diff File

bareos-17.2 f7d97b88

2017-07-02 01:22

mvwieringen


Committer: pstorz

Ported: N/A

Details Diff		 stored: Implement object device chunked volumes.

As some of the REST protocols only allow you to write blobs in
total we now offer a chunked object store device type in which
the volumes are chunked into pieces and these pieces are flushed
to the backing store either on close or when we reach the maximum
size of the chunk and need a new one. We also implemented the
reading of such chunked devices.

This runs on top of Scality's libdroplet with some local fixes and
changes.
mod - autoconf/configure.in Diff File
mod - src/lib/Makefile.in Diff File
mod - src/lib/cbuf.c Diff File
mod - src/lib/cbuf.h Diff File
add - src/lib/ordered_cbuf.c Diff File
add - src/lib/ordered_cbuf.h Diff File
mod - src/stored/Makefile.in Diff File
mod - src/stored/backends/Makefile.in Diff File
add - src/stored/backends/chunked_device.c Diff File
add - src/stored/backends/chunked_device.h Diff File
mod - src/stored/backends/object_store_device.c Diff File
mod - src/stored/backends/object_store_device.h Diff File
mod - src/stored/backends/unix_tape_device.c Diff File
mod - src/stored/dev.c Diff File
mod - src/win32/stored/backends/win32_tape_device.c Diff File

bareos-17.2 0ab6d804

2017-07-02 01:22

mvwieringen


Committer: pstorz

Ported: N/A

Details Diff		 build: Rebuild configure and config.h.in
mod - autoconf/config.h.in Diff File
mod - configure Diff File

bareos-16.2 2ef36e5f

2017-07-02 01:22

mvwieringen

Ported: N/A

Details Diff		 stored: Use SEEK_CUR not SEEK_END to check positioning.

When we want to determine if the position is set right in
DCR::is_eod_valid() we need to perform a lseek(fd, 0, SEEK_CUR)
and not a lseek(fd, 0, SEEK_END) as that is already done in
DEVICE::eod(). We only want to determine if the lseek was done
right in DCR::is_eod_valid() and by doing a SEEK_END we do the
same seek twice which leads to the same result but is not correct.
mod - src/stored/mount.c Diff File

bareos-16.2-droplet 9c13a428

2017-07-02 01:22

mvwieringen

Ported: N/A

Details Diff		 lib: Add edit_pthread() function to print pthread_t ids.

There is no real clean way to print the pthread_t type returned by
various pthreads functions. Until now things were printed as pointer
addresses. The edit_pthread() method is analog to other edit functions
and tries to print things in an uniform way and should be used to print
the content of the pthread_t opaque structure. The benefit of this is
that the representation is in one central place and we can implement
specific representations for different platforms.

As part of the implementation of this new function the edit.c source
file was also refactored and reindented to be better readable. Further
more now all modifiers are stored as static const char pointers which
used to be done for some functions but also not in others.
mod - src/lib/btimers.c Diff File
mod - src/lib/edit.c Diff File
mod - src/lib/jcr.c Diff File
mod - src/lib/lockmgr.c Diff File
mod - src/lib/protos.h Diff File
mod - src/lib/res.c Diff File
mod - src/stored/lock.c Diff File

bareos-16.2-droplet 44bddc87

2017-07-02 01:22

mvwieringen

Ported: N/A

Details Diff		 stored: Add device specific status trigger.

This adds code that allows you to trigger a device to return specific
device status information analog to the way which was already available
to plugins using the bsdEventDriveStatus and bsdEventVolumeStatus
events.

In essence this is equivalent to the bsdEventDriveStatus which allows a
plugin (for example the scsicrypto-sd plugin to return the crypto status
of an LTO4+ drive.) but then for any device without the need to have a
plugin loaded for the specific device.
mod - src/stored/dev.h Diff File
mod - src/stored/sd_plugins.h Diff File
mod - src/stored/status.c Diff File

bareos-16.2-droplet 45651202

2017-07-02 01:22

mvwieringen

Ported: N/A

Details Diff		 stored: Implement object device chunked volumes.

As some of the REST protocols only allow you to write blobs in
total we now offer a chunked object store device type in which
the volumes are chunked into pieces and these pieces are flushed
to the backing store either on close or when we reach the maximum
size of the chunk and need a new one. We also implemented the
reading of such chunked devices.

This runs on top of Scality's libdroplet with some local fixes and
changes.
mod - autoconf/configure.in Diff File
mod - src/lib/Makefile.in Diff File
mod - src/lib/cbuf.c Diff File
mod - src/lib/cbuf.h Diff File
add - src/lib/ordered_cbuf.c Diff File
add - src/lib/ordered_cbuf.h Diff File
mod - src/stored/Makefile.in Diff File
mod - src/stored/backends/Makefile.in Diff File
add - src/stored/backends/chunked_device.c Diff File
add - src/stored/backends/chunked_device.h Diff File
mod - src/stored/backends/object_store_device.c Diff File
mod - src/stored/backends/object_store_device.h Diff File
mod - src/stored/backends/unix_tape_device.c Diff File
mod - src/stored/dev.c Diff File
mod - src/win32/stored/backends/win32_tape_device.c Diff File
 First  Prev  1 2 3 ... 60 ... 120 ... 180 ... 240 ... 300 ... 360 ... 420 ... 451 452 453 454 455 456 457 ... 480 ... 540 ... 600 ... 635 636 637  Next  Last