bareos: master d4431a51

Diff Back to Repository
Author Committer Branch Timestamp Parent
mvwieringen adm mvwieringen adm master 2013-05-04 20:37 master 9096704e Pending
Affected Issues  0000007: Job migration/copy between different SDs
Changeset Initial working version of SD-SD replication.

This code implements the replication of data between two Storage Daemons
used by the migrate and copy jobs. This is an extenstion of the existing
migration and copy job functionality. The old code already could select
two different storage resources on two different Storage Daemons but it
would then only connect to the write storage and ask there for the read
storage which is ofcourse not known there. This code checks to see if
the read and write storage is located on diffferent Storage Daemons by
comparing the address, SDport and password of the remote storage daemon.

If it detects there are two Storage Daemons involved it will use the
new SD-SD communication protocol to perform a replicate of the data from
the read storage to the write storage.

This patch implements the following:
- Refactored code for authenticating a daemon, this was refactored so
  that clients like the bconsole and qt-console from now on use the
  now working authenticate_with_director method of the bsock class so
  we don't duplicate code to much. Also the other authentication was
  refactored as part of this overhaul of the authentication system.
  There are now generic functions which allow to authenticate a certain
  daemon and a specific authentication is implemented using these
  functions. For the SD-SD authentication there is some extra code to be
  able to have one storage daemon start the challenge and the let the
  other response otherwise the authentication would never work as both
  challenge each other at the same time.
- Refactored a lot of code to properly use a boolean when things are
  a boolean and not polute the code with assigning true to an integer
  variable. That by incident false is 0 and true is 1 doesn't mean
  that you shouldn't just call a variable of the type bool and mix
  and match integer and boolean all over the code. A lot done more
  to come.
- Most protocol messages are now put into static char variables in the
  code instead of defining them directly in the code call. This was done
  in some parts of the code but not nearly enough this is a good step
  in the right direction.
- You can bandwith limit your replication sessions in multiple ways.
  - set a bandwidth limit in the migration/copy job
  - set a bandwitdh limit for the writing SD in the SD definition
    in the director config.
  - set a bandwitdh limit for the reading SD in the SD definition
    in the director config.
  - set a bandwidth limit on the reading SD in the storage daemon config
  - set a bandwidth limit on the writing SD in the storage daemon config
  - change the bandwith via the console setbandwidth cmd.
- The already allocated mig_jcr for a migration and copy Job is used
  to track the communication with the writing SD. The reading SD is
  controlled via the normal jcr structure. For this to work the mig_jcr
  is created without a so called Messages resource so when using the
  mig_jcr for controlling the writing SD we don't get those nasty mail
  and database logging as if the Job actually did run.
- A new SD-SD communication protocol is implemented and this can be
  extended in the future to perform other SD-SD protocol features. For
  now the generic listener of the SD will understand the following
  incoming connections:
    - FD - SD backup and restore sessions which start with the following
      sequence: Hello Start Job <jobname>
    - SD - SD replication of data which start with the following
      sequence: Hello Start Storage Job <jobname>
    - DIR - SD controlling sessions for controlling the SD from the
      director.
- The old one Storage Daemon migrate and copy is still available and
  when the director finds out both the read and write storage is one
  the same SD it will use this optimized and fastest code path where
  the SD reads the data and directly writes the data to other media
  available to the same SD. SD-SD network replication is only used when
  the two SDs are determined to be different this way we only use the
  network when really neeeded and the one SD replication using internal
  stuctures is way faster then any loopback network send protocol.
- Comments are fixed along the way.
- Parts of lib/message.c are reindented as it used the wrong indention
  levels.

Fixes 0000007: Job migration/copy between different SDs
mod - src/win32/qt-console/bat.pro Diff File
mod - src/win32/console/Makefile Diff File
mod - src/stored/wait.c Diff File
mod - src/stored/stored_conf.h Diff File
mod - src/stored/stored_conf.c Diff File
mod - src/stored/stored.c Diff File
mod - src/stored/status.c Diff File
add - src/stored/sd_cmds.c Diff File
mod - src/stored/reserve.c Diff File
mod - src/stored/record.h Diff File
mod - src/stored/read.c Diff File
mod - src/stored/protos.h Diff File
mod - src/stored/mac.c Diff File
mod - src/stored/job.c Diff File
mod - src/stored/fd_cmds.c Diff File
mod - src/stored/dircmd.c Diff File
mod - src/stored/btape.c Diff File
mod - src/stored/bscan.c Diff File
mod - src/stored/bls.c Diff File
mod - src/stored/bextract.c Diff File
mod - src/stored/bcopy.c Diff File
mod - src/stored/authenticate.c Diff File
mod - src/stored/askdir.c Diff File
mod - src/stored/append.c Diff File
mod - src/stored/acquire.c Diff File
mod - src/stored/Makefile.in Diff File
rm - src/qt-console/bcomm/dircomm_auth.cpp Diff File
mod - src/qt-console/bcomm/dircomm.h Diff File
mod - src/qt-console/bcomm/dircomm.cpp Diff File
mod - src/qt-console/bat.pro.in Diff File
mod - src/lib/tls.c Diff File
mod - src/lib/protos.h Diff File
mod - src/lib/message.h Diff File
mod - src/lib/message.c Diff File
mod - src/lib/cram-md5.c Diff File
mod - src/lib/bsock.h Diff File
mod - src/lib/bsock.c Diff File
mod - src/lib/base64.c Diff File
mod - src/include/jcr.h Diff File
mod - src/filed/verify_vol.c Diff File
mod - src/filed/verify.c Diff File
mod - src/filed/status.c Diff File
mod - src/filed/restore.c Diff File
mod - src/filed/protos.h Diff File
mod - src/filed/job.c Diff File
mod - src/filed/filed.c Diff File
mod - src/filed/backup.c Diff File
mod - src/filed/authenticate.c Diff File
mod - src/filed/accurate.c Diff File
mod - src/dird/verify.c Diff File
mod - src/dird/vbackup.c Diff File
mod - src/dird/ua_status.c Diff File
mod - src/dird/ua_cmds.c Diff File
mod - src/dird/sd_cmds.c Diff File
mod - src/dird/restore.c Diff File
mod - src/dird/protos.h Diff File
mod - src/dird/ndmp_dma.c Diff File
mod - src/dird/msgchan.c Diff File
mod - src/dird/mountreq.c Diff File
mod - src/dird/migrate.c Diff File
mod - src/dird/job.c Diff File
mod - src/dird/getmsg.c Diff File
mod - src/dird/fd_cmds.c Diff File
mod - src/dird/dird_conf.h Diff File
mod - src/dird/dird_conf.c Diff File
mod - src/dird/catreq.c Diff File
mod - src/dird/backup.c Diff File
mod - src/dird/authenticate.c Diff File
mod - src/console/console.c Diff File
rm - src/console/authenticate.c Diff File
mod - src/console/Makefile.in Diff File