Bareos Bug Tracker
Bareos Bug Tracker

View Issue Details Jump to Notes ] Related Changesets ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000007bareos-core[All Projects] storage daemonpublic2012-12-05 14:222015-03-25 19:19
Reportermvwieringen 
Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusclosedResolutionfixed 
PlatformOpenIndianaOSOpenIndianaOS Versionb151a
Product Version13.1.0 
Target Version13.1.0Fixed in Version13.1.0 
Summary0000007: Job migration/copy between different SDs
DescriptionFeature request 20 from Projects file.

Allow to specify in migration/copy job devices on Storage Daemon other then
the one used for backup jobs (possibly on different/distant host)

Sometimes we have more then one system which requires backup
implementation. Often, these systems are functionally unrelated and
placed in different locations. Having a big backup device (a tape
library) in each location is not cost-effective. It would be much
better to have one powerful enough tape library which could handle
backups from all systems, assuming relatively fast and reliable WAN
connections. In such architecture backups are done in service windows
on local bacula servers, then migrated to central storage off the peak
hours.

If migration to different SD is working, migration to the same SD, as
now, could be done the same way (i mean 'localhost') to unify the
whole process
Additional InformationMigration/Copy between the same SD should keep working like it is as any
network overhead is unneeded even when sending data over the loopback interface.

The current code reads and writes the data in one session with the SD by
providing it with the read and write storage. As is things already work
in the selection phase only the code will only connect to the writing SD
and that will give an error that it doesn't have the read storage.
TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action
Attached Files

- Relationships

-  Notes
(0000482)
pstorz (administrator)
2013-06-27 09:01

regression tests works, documentation is written and available in the trac wiki and on the website.
(0000623)
mvwieringen adm (administrator)
2013-08-13 03:12

Fix committed to bareos master branch with changesetid 841.
(0001429)
mvwieringen (developer)
2015-03-25 16:51

Fix committed to bareos2015 bareos-13.2 branch with changesetid 4397.
(0001584)
joergs (administrator)
2015-03-25 19:19

Due to the reimport of the Github repository to bugs.bareos.org, the status of some tickets have been changed. These tickets will be closed again.
Sorry for the noise.

- Related Changesets
bareos: master d4431a51
Timestamp: 2013-05-04 18:37:55
Author: mvwieringen adm
Ported: N/A
Details ] Diff ]
Initial working version of SD-SD replication.

This code implements the replication of data between two Storage Daemons
used by the migrate and copy jobs. This is an extenstion of the existing
migration and copy job functionality. The old code already could select
two different storage resources on two different Storage Daemons but it
would then only connect to the write storage and ask there for the read
storage which is ofcourse not known there. This code checks to see if
the read and write storage is located on diffferent Storage Daemons by
comparing the address, SDport and password of the remote storage daemon.

If it detects there are two Storage Daemons involved it will use the
new SD-SD communication protocol to perform a replicate of the data from
the read storage to the write storage.

This patch implements the following:
- Refactored code for authenticating a daemon, this was refactored so
that clients like the bconsole and qt-console from now on use the
now working authenticate_with_director method of the bsock class so
we don't duplicate code to much. Also the other authentication was
refactored as part of this overhaul of the authentication system.
There are now generic functions which allow to authenticate a certain
daemon and a specific authentication is implemented using these
functions. For the SD-SD authentication there is some extra code to be
able to have one storage daemon start the challenge and the let the
other response otherwise the authentication would never work as both
challenge each other at the same time.
- Refactored a lot of code to properly use a boolean when things are
a boolean and not polute the code with assigning true to an integer
variable. That by incident false is 0 and true is 1 doesn't mean
that you shouldn't just call a variable of the type bool and mix
and match integer and boolean all over the code. A lot done more
to come.
- Most protocol messages are now put into static char variables in the
code instead of defining them directly in the code call. This was done
in some parts of the code but not nearly enough this is a good step
in the right direction.
- You can bandwith limit your replication sessions in multiple ways.
- set a bandwidth limit in the migration/copy job
- set a bandwitdh limit for the writing SD in the SD definition
in the director config.
- set a bandwitdh limit for the reading SD in the SD definition
in the director config.
- set a bandwidth limit on the reading SD in the storage daemon config
- set a bandwidth limit on the writing SD in the storage daemon config
- change the bandwith via the console setbandwidth cmd.
- The already allocated mig_jcr for a migration and copy Job is used
to track the communication with the writing SD. The reading SD is
controlled via the normal jcr structure. For this to work the mig_jcr
is created without a so called Messages resource so when using the
mig_jcr for controlling the writing SD we don't get those nasty mail
and database logging as if the Job actually did run.
- A new SD-SD communication protocol is implemented and this can be
extended in the future to perform other SD-SD protocol features. For
now the generic listener of the SD will understand the following
incoming connections:
- FD - SD backup and restore sessions which start with the following
sequence: Hello Start Job <jobname>
- SD - SD replication of data which start with the following
sequence: Hello Start Storage Job <jobname>
- DIR - SD controlling sessions for controlling the SD from the
director.
- The old one Storage Daemon migrate and copy is still available and
when the director finds out both the read and write storage is one
the same SD it will use this optimized and fastest code path where
the SD reads the data and directly writes the data to other media
available to the same SD. SD-SD network replication is only used when
the two SDs are determined to be different this way we only use the
network when really neeeded and the one SD replication using internal
stuctures is way faster then any loopback network send protocol.
- Comments are fixed along the way.
- Parts of lib/message.c are reindented as it used the wrong indention
levels.

Fixes 0000007: Job migration/copy between different SDs
mod - src/win32/qt-console/bat.pro Diff ] File ]
mod - src/win32/console/Makefile Diff ] File ]
mod - src/stored/wait.c Diff ] File ]
mod - src/stored/stored_conf.h Diff ] File ]
mod - src/stored/stored_conf.c Diff ] File ]
mod - src/stored/stored.c Diff ] File ]
mod - src/stored/status.c Diff ] File ]
add - src/stored/sd_cmds.c Diff ] File ]
mod - src/stored/reserve.c Diff ] File ]
mod - src/stored/record.h Diff ] File ]
mod - src/stored/read.c Diff ] File ]
mod - src/stored/protos.h Diff ] File ]
mod - src/stored/mac.c Diff ] File ]
mod - src/stored/job.c Diff ] File ]
mod - src/stored/fd_cmds.c Diff ] File ]
mod - src/stored/dircmd.c Diff ] File ]
mod - src/stored/btape.c Diff ] File ]
mod - src/stored/bscan.c Diff ] File ]
mod - src/stored/bls.c Diff ] File ]
mod - src/stored/bextract.c Diff ] File ]
mod - src/stored/bcopy.c Diff ] File ]
mod - src/stored/authenticate.c Diff ] File ]
mod - src/stored/askdir.c Diff ] File ]
mod - src/stored/append.c Diff ] File ]
mod - src/stored/acquire.c Diff ] File ]
mod - src/stored/Makefile.in Diff ] File ]
rm - src/qt-console/bcomm/dircomm_auth.cpp Diff ] File ]
mod - src/qt-console/bcomm/dircomm.h Diff ] File ]
mod - src/qt-console/bcomm/dircomm.cpp Diff ] File ]
mod - src/qt-console/bat.pro.in Diff ] File ]
mod - src/lib/tls.c Diff ] File ]
mod - src/lib/protos.h Diff ] File ]
mod - src/lib/message.h Diff ] File ]
mod - src/lib/message.c Diff ] File ]
mod - src/lib/cram-md5.c Diff ] File ]
mod - src/lib/bsock.h Diff ] File ]
mod - src/lib/bsock.c Diff ] File ]
mod - src/lib/base64.c Diff ] File ]
mod - src/include/jcr.h Diff ] File ]
mod - src/filed/verify_vol.c Diff ] File ]
mod - src/filed/verify.c Diff ] File ]
mod - src/filed/status.c Diff ] File ]
mod - src/filed/restore.c Diff ] File ]
mod - src/filed/protos.h Diff ] File ]
mod - src/filed/job.c Diff ] File ]
mod - src/filed/filed.c Diff ] File ]
mod - src/filed/backup.c Diff ] File ]
mod - src/filed/authenticate.c Diff ] File ]
mod - src/filed/accurate.c Diff ] File ]
mod - src/dird/verify.c Diff ] File ]
mod - src/dird/vbackup.c Diff ] File ]
mod - src/dird/ua_status.c Diff ] File ]
mod - src/dird/ua_cmds.c Diff ] File ]
mod - src/dird/sd_cmds.c Diff ] File ]
mod - src/dird/restore.c Diff ] File ]
mod - src/dird/protos.h Diff ] File ]
mod - src/dird/ndmp_dma.c Diff ] File ]
mod - src/dird/msgchan.c Diff ] File ]
mod - src/dird/mountreq.c Diff ] File ]
mod - src/dird/migrate.c Diff ] File ]
mod - src/dird/job.c Diff ] File ]
mod - src/dird/getmsg.c Diff ] File ]
mod - src/dird/fd_cmds.c Diff ] File ]
mod - src/dird/dird_conf.h Diff ] File ]
mod - src/dird/dird_conf.c Diff ] File ]
mod - src/dird/catreq.c Diff ] File ]
mod - src/dird/backup.c Diff ] File ]
mod - src/dird/authenticate.c Diff ] File ]
mod - src/console/console.c Diff ] File ]
rm - src/console/authenticate.c Diff ] File ]
mod - src/console/Makefile.in Diff ] File ]
bareos2015: bareos-13.2 edaf72d8
Timestamp: 2013-05-04 20:37:55
Author: mvwieringen
Ported: N/A
Details ] Diff ]
Initial working version of SD-SD replication.

This code implements the replication of data between two Storage Daemons
used by the migrate and copy jobs. This is an extenstion of the existing
migration and copy job functionality. The old code already could select
two different storage resources on two different Storage Daemons but it
would then only connect to the write storage and ask there for the read
storage which is ofcourse not known there. This code checks to see if
the read and write storage is located on diffferent Storage Daemons by
comparing the address, SDport and password of the remote storage daemon.

If it detects there are two Storage Daemons involved it will use the
new SD-SD communication protocol to perform a replicate of the data from
the read storage to the write storage.

This patch implements the following:
- Refactored code for authenticating a daemon, this was refactored so
that clients like the bconsole and qt-console from now on use the
now working authenticate_with_director method of the bsock class so
we don't duplicate code to much. Also the other authentication was
refactored as part of this overhaul of the authentication system.
There are now generic functions which allow to authenticate a certain
daemon and a specific authentication is implemented using these
functions. For the SD-SD authentication there is some extra code to be
able to have one storage daemon start the challenge and the let the
other response otherwise the authentication would never work as both
challenge each other at the same time.
- Refactored a lot of code to properly use a boolean when things are
a boolean and not polute the code with assigning true to an integer
variable. That by incident false is 0 and true is 1 doesn't mean
that you shouldn't just call a variable of the type bool and mix
and match integer and boolean all over the code. A lot done more
to come.
- Most protocol messages are now put into static char variables in the
code instead of defining them directly in the code call. This was done
in some parts of the code but not nearly enough this is a good step
in the right direction.
- You can bandwith limit your replication sessions in multiple ways.
- set a bandwidth limit in the migration/copy job
- set a bandwitdh limit for the writing SD in the SD definition
in the director config.
- set a bandwitdh limit for the reading SD in the SD definition
in the director config.
- set a bandwidth limit on the reading SD in the storage daemon config
- set a bandwidth limit on the writing SD in the storage daemon config
- change the bandwith via the console setbandwidth cmd.
- The already allocated mig_jcr for a migration and copy Job is used
to track the communication with the writing SD. The reading SD is
controlled via the normal jcr structure. For this to work the mig_jcr
is created without a so called Messages resource so when using the
mig_jcr for controlling the writing SD we don't get those nasty mail
and database logging as if the Job actually did run.
- A new SD-SD communication protocol is implemented and this can be
extended in the future to perform other SD-SD protocol features. For
now the generic listener of the SD will understand the following
incoming connections:
- FD - SD backup and restore sessions which start with the following
sequence: Hello Start Job <jobname>
- SD - SD replication of data which start with the following
sequence: Hello Start Storage Job <jobname>
- DIR - SD controlling sessions for controlling the SD from the
director.
- The old one Storage Daemon migrate and copy is still available and
when the director finds out both the read and write storage is one
the same SD it will use this optimized and fastest code path where
the SD reads the data and directly writes the data to other media
available to the same SD. SD-SD network replication is only used when
the two SDs are determined to be different this way we only use the
network when really neeeded and the one SD replication using internal
stuctures is way faster then any loopback network send protocol.
- Comments are fixed along the way.
- Parts of lib/message.c are reindented as it used the wrong indention
levels.

Fixes 0000007: Job migration/copy between different SDs
mod - src/console/Makefile.in Diff ] File ]
rm - src/console/authenticate.c Diff ] File ]
mod - src/console/console.c Diff ] File ]
mod - src/dird/authenticate.c Diff ] File ]
mod - src/dird/backup.c Diff ] File ]
mod - src/dird/catreq.c Diff ] File ]
mod - src/dird/dird_conf.c Diff ] File ]
mod - src/dird/dird_conf.h Diff ] File ]
mod - src/dird/fd_cmds.c Diff ] File ]
mod - src/dird/getmsg.c Diff ] File ]
mod - src/dird/job.c Diff ] File ]
mod - src/dird/migrate.c Diff ] File ]
mod - src/dird/mountreq.c Diff ] File ]
mod - src/dird/msgchan.c Diff ] File ]
mod - src/dird/ndmp_dma.c Diff ] File ]
mod - src/dird/protos.h Diff ] File ]
mod - src/dird/restore.c Diff ] File ]
mod - src/dird/sd_cmds.c Diff ] File ]
mod - src/dird/ua_cmds.c Diff ] File ]
mod - src/dird/ua_status.c Diff ] File ]
mod - src/dird/vbackup.c Diff ] File ]
mod - src/dird/verify.c Diff ] File ]
mod - src/filed/accurate.c Diff ] File ]
mod - src/filed/authenticate.c Diff ] File ]
mod - src/filed/backup.c Diff ] File ]
mod - src/filed/filed.c Diff ] File ]
mod - src/filed/job.c Diff ] File ]
mod - src/filed/protos.h Diff ] File ]
mod - src/filed/restore.c Diff ] File ]
mod - src/filed/status.c Diff ] File ]
mod - src/filed/verify.c Diff ] File ]
mod - src/filed/verify_vol.c Diff ] File ]
mod - src/include/jcr.h Diff ] File ]
mod - src/lib/base64.c Diff ] File ]
mod - src/lib/bsock.c Diff ] File ]
mod - src/lib/bsock.h Diff ] File ]
mod - src/lib/cram-md5.c Diff ] File ]
mod - src/lib/message.c Diff ] File ]
mod - src/lib/message.h Diff ] File ]
mod - src/lib/protos.h Diff ] File ]
mod - src/lib/tls.c Diff ] File ]
mod - src/qt-console/bat.pro.in Diff ] File ]
mod - src/qt-console/bcomm/dircomm.cpp Diff ] File ]
mod - src/qt-console/bcomm/dircomm.h Diff ] File ]
rm - src/qt-console/bcomm/dircomm_auth.cpp Diff ] File ]
mod - src/stored/Makefile.in Diff ] File ]
mod - src/stored/acquire.c Diff ] File ]
mod - src/stored/append.c Diff ] File ]
mod - src/stored/askdir.c Diff ] File ]
mod - src/stored/authenticate.c Diff ] File ]
mod - src/stored/bcopy.c Diff ] File ]
mod - src/stored/bextract.c Diff ] File ]
mod - src/stored/bls.c Diff ] File ]
mod - src/stored/bscan.c Diff ] File ]
mod - src/stored/btape.c Diff ] File ]
mod - src/stored/dircmd.c Diff ] File ]
mod - src/stored/fd_cmds.c Diff ] File ]
mod - src/stored/job.c Diff ] File ]
mod - src/stored/mac.c Diff ] File ]
mod - src/stored/protos.h Diff ] File ]
mod - src/stored/read.c Diff ] File ]
mod - src/stored/record.h Diff ] File ]
mod - src/stored/reserve.c Diff ] File ]
add - src/stored/sd_cmds.c Diff ] File ]
mod - src/stored/status.c Diff ] File ]
mod - src/stored/stored.c Diff ] File ]
mod - src/stored/stored_conf.c Diff ] File ]
mod - src/stored/stored_conf.h Diff ] File ]
mod - src/stored/wait.c Diff ] File ]
mod - src/win32/console/Makefile Diff ] File ]
mod - src/win32/qt-console/bat.pro Diff ] File ]

- Issue History
Date Modified Username Field Change
2012-12-05 14:22 mvwieringen New Issue
2012-12-05 14:22 mvwieringen Status new => assigned
2012-12-05 14:22 mvwieringen Assigned To => mvwieringen
2012-12-10 13:58 maik Category General => storage daemon
2012-12-24 19:18 mvwieringen Product Version 12.4.0 => 13.1.0
2013-01-08 17:17 mvwieringen Time allocated Deleted 2013-01-06: 5,00 h. => deleted
2013-02-13 18:56 mvwieringen Changeset attached => bareos master 01e4409e
2013-02-13 18:56 mvwieringen Note Added: 0000098
2013-02-13 18:56 mvwieringen Status assigned => closed
2013-02-13 18:56 mvwieringen Resolution open => fixed
2013-02-13 19:05 mvwieringen Target Version => 13.1.0
2013-02-13 19:05 mvwieringen Fixed in Version 12.4.0 => 13.1.0
2013-02-13 19:05 mvwieringen Assigned To mvwieringen =>
2013-03-04 10:31 mvwieringen Product Version => 13.1.0
2013-03-04 10:31 mvwieringen Fixed in Version => 13.1.0
2013-03-04 10:31 mvwieringen Target Version => 13.1.0
2013-05-24 12:51 pstorz Assigned To => pstorz
2013-05-24 12:51 pstorz Status closed => resolved
2013-06-27 09:01 pstorz Note Added: 0000482
2013-07-04 13:02 pstorz Status resolved => closed
2013-08-13 03:12 mvwieringen adm Changeset attached => bareos master d4431a51
2013-08-13 03:12 mvwieringen adm Note Added: 0000623
2013-08-13 03:12 mvwieringen adm Assigned To pstorz => mvwieringen adm
2013-08-13 03:12 mvwieringen adm Status closed => resolved
2013-08-13 08:52 mvwieringen adm Note Deleted: 0000098
2013-08-13 08:53 mvwieringen adm Assigned To mvwieringen adm =>
2013-08-13 08:53 mvwieringen adm Status resolved => closed
2015-03-25 16:51 mvwieringen Changeset attached => bareos2015 bareos-13.2 edaf72d8
2015-03-25 16:51 mvwieringen Note Added: 0001429
2015-03-25 16:51 mvwieringen Status closed => resolved
2015-03-25 19:19 joergs Note Added: 0001584
2015-03-25 19:19 joergs Status resolved => closed


Copyright © 2000 - 2018 MantisBT Team
Powered by Mantis Bugtracker