bareos: master d31cd38e

Diff Back to Repository
Author Committer Branch Timestamp Parent
pstorz pstorz master 2017-08-14 12:35 master 7d517efc Pending
Changeset NDMP: add support for 2-way NDMP with DAR and DDAR

The current implementation of NDMP in Bareos has some limitations
because of the way it was implemented.

- The backup is always sent over the network from the Data Agent to the
  Tape Agent (3-way NDMP)
- DAR(Direct Access Recovery) and DDAR (Directory DAR) are not supported,
  single file restores are VERY slow as the whole backup stream needs to
  be scanned during restore.

This commit adds support for 2-way NDMP backups inside of the Director.
During 2-way NDMP Backups, the Director is the only bareos daemon involved,
the Storage Daemon does not do anything in this setup.

To be able to do 2-way NDMP backups and to support DAR and DDAR, the
director
- controls the NDMP changer via NDMP Protocol.
- does the bookkeeping for the NDMP media.
- stores the full NDMP file information being sent during backups (FhInfo and FhNode)
  in the catalog database.
- sends back the full NDMP file information during restore to enable DAR
  and DDAR.

The advantage of the 2-way NDMP backups are:
- Much higher backup speed as the data goes directly from the disks to
  the tape without going over the network.
- Much higher restore speed for single files/directories with DAR and
  DDAR, as only the required blocks are read during restore instead of
  restoring the whole backup stream and only restoring the required
  data.

The following NDMP Protocol keywords are now available:

 * NDMP : which is an alias for NDMP_BAREOS for backward compatibility
 * NDMP_BAREOS: which is NDMP backups with paired storage and NDMP Tape
                 Agent inside of the Bareos Storage Daemon
 * NDMP_NATIVE: this is doing NDMP backups with only the DMA inside of
                the Bareos Director without any connection to the SD

These keywords reflect the two ways of NDMP that are now supported in
Bareos, which is either the special way of NDMP backups implemented in
Bareos before, or the "NATIVE" way to do NDMP backups.

To be able to store the FileHandle information for DAR and DDAR, the
File table how has two additional columns for FhInfo and FhNode.
The JobMedia table how has the additional column JobBytes which is used
to store NDMP media Information.

This column is planned to hold the JobBytes that are stored in the
corresponding JobMedia record.

The ndmp_dma_backup and ndmp_dma_restore files were split up for the two
different ways of NDMP backups that we have now.
mod - src/cats/cats.h Diff File
mod - src/cats/ddl/creates/mysql.sql Diff File
mod - src/cats/ddl/creates/postgresql.sql Diff File
mod - src/cats/ddl/creates/sqlite3.sql Diff File
mod - src/cats/ddl/updates/mysql.2004_2170.sql Diff File
add - src/cats/ddl/updates/mysql.2170_2171.sql Diff File
add - src/cats/ddl/updates/postgresql.2170_2171.sql Diff File
add - src/cats/ddl/updates/sqlite3.2170_2171.sql Diff File
mod - src/cats/dml/0034_select_recent_version_with_basejob Diff File
mod - src/cats/dml/0034_select_recent_version_with_basejob.postgresql Diff File
mod - src/cats/dml/0035_select_recent_version_with_basejob_and_delta Diff File
mod - src/cats/dml/0035_select_recent_version_with_basejob_and_delta.postgresql Diff File
mod - src/cats/dml/0036_select_recent_version Diff File
mod - src/cats/dml/0036_select_recent_version.postgresql Diff File
mod - src/cats/dml/0039_create_temp_new_basefile Diff File
mod - src/cats/dml/0039_create_temp_new_basefile.ingres Diff File
mod - src/cats/ingres.c Diff File
mod - src/cats/mysql.c Diff File
mod - src/cats/mysql_queries.inc Diff File
mod - src/cats/postgresql.c Diff File
mod - src/cats/postgresql_queries.inc Diff File
mod - src/cats/protos.h Diff File
mod - src/cats/sql_create.c Diff File
mod - src/cats/sql_get.c Diff File
mod - src/cats/sql_pooling.c Diff File
mod - src/cats/sqlite.c Diff File
mod - src/cats/sqlite_queries.inc Diff File
mod - src/dird/Makefile.in Diff File
mod - src/dird/backup.c Diff File
mod - src/dird/catreq.c Diff File
mod - src/dird/dird_conf.c Diff File
mod - src/dird/job.c Diff File
add - src/dird/ndmp_dma_backup_NDMP_NATIVE.c Diff File
add - src/dird/ndmp_dma_backup_common.c Diff File
mod - src/dird/ndmp_dma_generic.c Diff File
mod - src/dird/ndmp_dma_priv.h Diff File
add - src/dird/ndmp_dma_restore_NDMP_NATIVE.c Diff File
add - src/dird/ndmp_dma_restore_common.c Diff File
mod - src/dird/ndmp_dma_storage.c Diff File
mod - src/dird/ndmp_fhdb_common.c Diff File
mod - src/dird/ndmp_fhdb_helpers.c Diff File
mod - src/dird/ndmp_fhdb_lmdb.c Diff File
add - src/dird/ndmp_ndmmedia_db_helpers.c Diff File
mod - src/dird/protos.h Diff File
mod - src/dird/restore.c Diff File
mod - src/dird/storage.c Diff File
mod - src/dird/ua_restore.c Diff File
mod - src/dird/ua_tree.c Diff File
mod - src/include/jcr.h Diff File
mod - src/lib/tree.h Diff File
mod - src/ndmp/Makefile.in Diff File
mod - src/ndmp/ndma_comm_job.c Diff File
mod - src/ndmp/ndma_comm_session.c Diff File
mod - src/ndmp/ndma_control.c Diff File
mod - src/ndmp/ndma_cops_backreco.c Diff File
mod - src/ndmp/ndma_cops_robot.c Diff File
mod - src/ndmp/ndma_ctrl_media.c Diff File
mod - src/ndmp/ndma_ctrl_robot.c Diff File
mod - src/ndmp/ndma_data_fh.c Diff File
mod - src/ndmp/ndma_listmgmt.c Diff File
mod - src/ndmp/ndma_tape.c Diff File
mod - src/ndmp/ndmagents.h Diff File
mod - src/ndmp/ndmjob_args.c Diff File
mod - src/ndmp/ndmjob_main.c Diff File
mod - src/ndmp/ndmjob_simulator.c Diff File
mod - src/ndmp/ndml_chan.c Diff File
mod - src/ndmp/ndml_conn.c Diff File
mod - src/ndmp/ndmos.h Diff File
mod - src/ndmp/ndmp4_pp.c Diff File
mod - src/stored/bscan.c Diff File
mod - src/stored/bsr.c Diff File
mod - src/stored/dir_cmd.c Diff File
mod - src/stored/job.c Diff File
mod - src/win32/dird/Makefile Diff File