0000007: Job migration/copy between different SDs

ID	Project	Category	View Status	Date Submitted	Last Update

0000007	bareos-core	storage daemon	public	2012-12-05 14:22	2015-03-25 19:19

Reporter	mvwieringen	Assigned To
Priority	normal	Severity	feature	Reproducibility	N/A
Status	closed	Resolution	fixed
Platform	OpenIndiana	OS	OpenIndiana	OS Version	b151a
Product Version	13.1.0
Fixed in Version	13.1.0

Summary	0000007: Job migration/copy between different SDs
Description	Feature request 20 from Projects file. Allow to specify in migration/copy job devices on Storage Daemon other then the one used for backup jobs (possibly on different/distant host) Sometimes we have more then one system which requires backup implementation. Often, these systems are functionally unrelated and placed in different locations. Having a big backup device (a tape library) in each location is not cost-effective. It would be much better to have one powerful enough tape library which could handle backups from all systems, assuming relatively fast and reliable WAN connections. In such architecture backups are done in service windows on local bacula servers, then migrated to central storage off the peak hours. If migration to different SD is working, migration to the same SD, as now, could be done the same way (i mean 'localhost') to unify the whole process
Additional Information	Migration/Copy between the same SD should keep working like it is as any network overhead is unneeded even when sending data over the loopback interface. The current code reads and writes the data in one session with the SD by providing it with the read and write storage. As is things already work in the selection phase only the code will only connect to the writing SD and that will give an error that it doesn't have the read storage.
Tags	No tags attached.

pstorz 2013-06-27 09:01 administrator ~0000482	regression tests works, documentation is written and available in the trac wiki and on the website.

~~mvwieringen adm~~ 2013-08-13 03:12 administrator ~0000623	Fix committed to bareos master branch with changesetid 841.

mvwieringen 2015-03-25 16:51 developer ~0001429	Fix committed to bareos2015 bareos-13.2 branch with changesetid 4397.

joergs 2015-03-25 19:19 developer ~0001584	Due to the reimport of the Github repository to bugs.bareos.org, the status of some tickets have been changed. These tickets will be closed again. Sorry for the noise.

bareos: master d4431a51 2013-05-04 20:37 ~~mvwieringen adm~~ Ported: N/A Details Diff	Initial working version of SD-SD replication. This code implements the replication of data between two Storage Daemons used by the migrate and copy jobs. This is an extenstion of the existing migration and copy job functionality. The old code already could select two different storage resources on two different Storage Daemons but it would then only connect to the write storage and ask there for the read storage which is ofcourse not known there. This code checks to see if the read and write storage is located on diffferent Storage Daemons by comparing the address, SDport and password of the remote storage daemon. If it detects there are two Storage Daemons involved it will use the new SD-SD communication protocol to perform a replicate of the data from the read storage to the write storage. This patch implements the following: - Refactored code for authenticating a daemon, this was refactored so that clients like the bconsole and qt-console from now on use the now working authenticate_with_director method of the bsock class so we don't duplicate code to much. Also the other authentication was refactored as part of this overhaul of the authentication system. There are now generic functions which allow to authenticate a certain daemon and a specific authentication is implemented using these functions. For the SD-SD authentication there is some extra code to be able to have one storage daemon start the challenge and the let the other response otherwise the authentication would never work as both challenge each other at the same time. - Refactored a lot of code to properly use a boolean when things are a boolean and not polute the code with assigning true to an integer variable. That by incident false is 0 and true is 1 doesn't mean that you shouldn't just call a variable of the type bool and mix and match integer and boolean all over the code. A lot done more to come. - Most protocol messages are now put into static char variables in the code instead of defining them directly in the code call. This was done in some parts of the code but not nearly enough this is a good step in the right direction. - You can bandwith limit your replication sessions in multiple ways. - set a bandwidth limit in the migration/copy job - set a bandwitdh limit for the writing SD in the SD definition in the director config. - set a bandwitdh limit for the reading SD in the SD definition in the director config. - set a bandwidth limit on the reading SD in the storage daemon config - set a bandwidth limit on the writing SD in the storage daemon config - change the bandwith via the console setbandwidth cmd. - The already allocated mig_jcr for a migration and copy Job is used to track the communication with the writing SD. The reading SD is controlled via the normal jcr structure. For this to work the mig_jcr is created without a so called Messages resource so when using the mig_jcr for controlling the writing SD we don't get those nasty mail and database logging as if the Job actually did run. - A new SD-SD communication protocol is implemented and this can be extended in the future to perform other SD-SD protocol features. For now the generic listener of the SD will understand the following incoming connections: - FD - SD backup and restore sessions which start with the following sequence: Hello Start Job <jobname> - SD - SD replication of data which start with the following sequence: Hello Start Storage Job <jobname> - DIR - SD controlling sessions for controlling the SD from the director. - The old one Storage Daemon migrate and copy is still available and when the director finds out both the read and write storage is one the same SD it will use this optimized and fastest code path where the SD reads the data and directly writes the data to other media available to the same SD. SD-SD network replication is only used when the two SDs are determined to be different this way we only use the network when really neeeded and the one SD replication using internal stuctures is way faster then any loopback network send protocol. - Comments are fixed along the way. - Parts of lib/message.c are reindented as it used the wrong indention levels. Fixes 0000007: Job migration/copy between different SDs		Affected Issues 0000007
	mod - src/win32/qt-console/bat.pro		Diff File
	mod - src/win32/console/Makefile		Diff File
	mod - src/stored/wait.c		Diff File
	mod - src/stored/stored_conf.h		Diff File
	mod - src/stored/stored_conf.c		Diff File
	mod - src/stored/stored.c		Diff File
	mod - src/stored/status.c		Diff File
	add - src/stored/sd_cmds.c		Diff File
	mod - src/stored/reserve.c		Diff File
	mod - src/stored/record.h		Diff File
	mod - src/stored/read.c		Diff File
	mod - src/stored/protos.h		Diff File
	mod - src/stored/mac.c		Diff File
	mod - src/stored/job.c		Diff File
	mod - src/stored/fd_cmds.c		Diff File
	mod - src/stored/dircmd.c		Diff File
	mod - src/stored/btape.c		Diff File
	mod - src/stored/bscan.c		Diff File
	mod - src/stored/bls.c		Diff File
	mod - src/stored/bextract.c		Diff File
	mod - src/stored/bcopy.c		Diff File
	mod - src/stored/authenticate.c		Diff File
	mod - src/stored/askdir.c		Diff File
	mod - src/stored/append.c		Diff File
	mod - src/stored/acquire.c		Diff File
	mod - src/stored/Makefile.in		Diff File
	rm - src/qt-console/bcomm/dircomm_auth.cpp		Diff File
	mod - src/qt-console/bcomm/dircomm.h		Diff File
	mod - src/qt-console/bcomm/dircomm.cpp		Diff File
	mod - src/qt-console/bat.pro.in		Diff File
	mod - src/lib/tls.c		Diff File
	mod - src/lib/protos.h		Diff File
	mod - src/lib/message.h		Diff File
	mod - src/lib/message.c		Diff File
	mod - src/lib/cram-md5.c		Diff File
	mod - src/lib/bsock.h		Diff File
	mod - src/lib/bsock.c		Diff File
	mod - src/lib/base64.c		Diff File
	mod - src/include/jcr.h		Diff File
	mod - src/filed/verify_vol.c		Diff File
	mod - src/filed/verify.c		Diff File
	mod - src/filed/status.c		Diff File
	mod - src/filed/restore.c		Diff File
	mod - src/filed/protos.h		Diff File
	mod - src/filed/job.c		Diff File
	mod - src/filed/filed.c		Diff File
	mod - src/filed/backup.c		Diff File
	mod - src/filed/authenticate.c		Diff File
	mod - src/filed/accurate.c		Diff File
	mod - src/dird/verify.c		Diff File
	mod - src/dird/vbackup.c		Diff File
	mod - src/dird/ua_status.c		Diff File
	mod - src/dird/ua_cmds.c		Diff File
	mod - src/dird/sd_cmds.c		Diff File
	mod - src/dird/restore.c		Diff File
	mod - src/dird/protos.h		Diff File
	mod - src/dird/ndmp_dma.c		Diff File
	mod - src/dird/msgchan.c		Diff File
	mod - src/dird/mountreq.c		Diff File
	mod - src/dird/migrate.c		Diff File
	mod - src/dird/job.c		Diff File
	mod - src/dird/getmsg.c		Diff File
	mod - src/dird/fd_cmds.c		Diff File
mod - src/dird/dird_conf.h		Diff File
mod - src/dird/dird_conf.c		Diff File
mod - src/dird/catreq.c		Diff File
mod - src/dird/backup.c		Diff File
mod - src/dird/authenticate.c		Diff File
mod - src/console/console.c		Diff File
rm - src/console/authenticate.c		Diff File
mod - src/console/Makefile.in		Diff File
bareos2015: bareos-13.2 edaf72d8 2013-05-04 22:37 mvwieringen Ported: N/A Details Diff	Initial working version of SD-SD replication. This code implements the replication of data between two Storage Daemons used by the migrate and copy jobs. This is an extenstion of the existing migration and copy job functionality. The old code already could select two different storage resources on two different Storage Daemons but it would then only connect to the write storage and ask there for the read storage which is ofcourse not known there. This code checks to see if the read and write storage is located on diffferent Storage Daemons by comparing the address, SDport and password of the remote storage daemon. If it detects there are two Storage Daemons involved it will use the new SD-SD communication protocol to perform a replicate of the data from the read storage to the write storage. This patch implements the following: - Refactored code for authenticating a daemon, this was refactored so that clients like the bconsole and qt-console from now on use the now working authenticate_with_director method of the bsock class so we don't duplicate code to much. Also the other authentication was refactored as part of this overhaul of the authentication system. There are now generic functions which allow to authenticate a certain daemon and a specific authentication is implemented using these functions. For the SD-SD authentication there is some extra code to be able to have one storage daemon start the challenge and the let the other response otherwise the authentication would never work as both challenge each other at the same time. - Refactored a lot of code to properly use a boolean when things are a boolean and not polute the code with assigning true to an integer variable. That by incident false is 0 and true is 1 doesn't mean that you shouldn't just call a variable of the type bool and mix and match integer and boolean all over the code. A lot done more to come. - Most protocol messages are now put into static char variables in the code instead of defining them directly in the code call. This was done in some parts of the code but not nearly enough this is a good step in the right direction. - You can bandwith limit your replication sessions in multiple ways. - set a bandwidth limit in the migration/copy job - set a bandwitdh limit for the writing SD in the SD definition in the director config. - set a bandwitdh limit for the reading SD in the SD definition in the director config. - set a bandwidth limit on the reading SD in the storage daemon config - set a bandwidth limit on the writing SD in the storage daemon config - change the bandwith via the console setbandwidth cmd. - The already allocated mig_jcr for a migration and copy Job is used to track the communication with the writing SD. The reading SD is controlled via the normal jcr structure. For this to work the mig_jcr is created without a so called Messages resource so when using the mig_jcr for controlling the writing SD we don't get those nasty mail and database logging as if the Job actually did run. - A new SD-SD communication protocol is implemented and this can be extended in the future to perform other SD-SD protocol features. For now the generic listener of the SD will understand the following incoming connections: - FD - SD backup and restore sessions which start with the following sequence: Hello Start Job <jobname> - SD - SD replication of data which start with the following sequence: Hello Start Storage Job <jobname> - DIR - SD controlling sessions for controlling the SD from the director. - The old one Storage Daemon migrate and copy is still available and when the director finds out both the read and write storage is one the same SD it will use this optimized and fastest code path where the SD reads the data and directly writes the data to other media available to the same SD. SD-SD network replication is only used when the two SDs are determined to be different this way we only use the network when really neeeded and the one SD replication using internal stuctures is way faster then any loopback network send protocol. - Comments are fixed along the way. - Parts of lib/message.c are reindented as it used the wrong indention levels. Fixes 0000007: Job migration/copy between different SDs		Affected Issues 0000007
	mod - src/console/Makefile.in		Diff File
	rm - src/console/authenticate.c		Diff File
	mod - src/console/console.c		Diff File
	mod - src/dird/authenticate.c		Diff File
	mod - src/dird/backup.c		Diff File
	mod - src/dird/catreq.c		Diff File
	mod - src/dird/dird_conf.c		Diff File
	mod - src/dird/dird_conf.h		Diff File
	mod - src/dird/fd_cmds.c		Diff File
	mod - src/dird/getmsg.c		Diff File
	mod - src/dird/job.c		Diff File
	mod - src/dird/migrate.c		Diff File
	mod - src/dird/mountreq.c		Diff File
	mod - src/dird/msgchan.c		Diff File
	mod - src/dird/ndmp_dma.c		Diff File
	mod - src/dird/protos.h		Diff File
	mod - src/dird/restore.c		Diff File
	mod - src/dird/sd_cmds.c		Diff File
	mod - src/dird/ua_cmds.c		Diff File
	mod - src/dird/ua_status.c		Diff File
	mod - src/dird/vbackup.c		Diff File
	mod - src/dird/verify.c		Diff File
	mod - src/filed/accurate.c		Diff File
	mod - src/filed/authenticate.c		Diff File
	mod - src/filed/backup.c		Diff File
	mod - src/filed/filed.c		Diff File
	mod - src/filed/job.c		Diff File
	mod - src/filed/protos.h		Diff File
	mod - src/filed/restore.c		Diff File
	mod - src/filed/status.c		Diff File
	mod - src/filed/verify.c		Diff File
	mod - src/filed/verify_vol.c		Diff File
	mod - src/include/jcr.h		Diff File
	mod - src/lib/base64.c		Diff File
	mod - src/lib/bsock.c		Diff File
	mod - src/lib/bsock.h		Diff File
	mod - src/lib/cram-md5.c		Diff File
	mod - src/lib/message.c		Diff File
	mod - src/lib/message.h		Diff File
	mod - src/lib/protos.h		Diff File
	mod - src/lib/tls.c		Diff File
	mod - src/qt-console/bat.pro.in		Diff File
	mod - src/qt-console/bcomm/dircomm.cpp		Diff File
	mod - src/qt-console/bcomm/dircomm.h		Diff File
	rm - src/qt-console/bcomm/dircomm_auth.cpp		Diff File
	mod - src/stored/Makefile.in		Diff File
	mod - src/stored/acquire.c		Diff File
	mod - src/stored/append.c		Diff File
	mod - src/stored/askdir.c		Diff File
	mod - src/stored/authenticate.c		Diff File
	mod - src/stored/bcopy.c		Diff File
	mod - src/stored/bextract.c		Diff File
	mod - src/stored/bls.c		Diff File
	mod - src/stored/bscan.c		Diff File
	mod - src/stored/btape.c		Diff File
	mod - src/stored/dircmd.c		Diff File
	mod - src/stored/fd_cmds.c		Diff File
	mod - src/stored/job.c		Diff File
	mod - src/stored/mac.c		Diff File
	mod - src/stored/protos.h		Diff File
	mod - src/stored/read.c		Diff File
	mod - src/stored/record.h		Diff File
	mod - src/stored/reserve.c		Diff File
add - src/stored/sd_cmds.c		Diff File
mod - src/stored/status.c		Diff File
mod - src/stored/stored.c		Diff File
mod - src/stored/stored_conf.c		Diff File
mod - src/stored/stored_conf.h		Diff File
mod - src/stored/wait.c		Diff File
mod - src/win32/console/Makefile		Diff File
mod - src/win32/qt-console/bat.pro		Diff File

Date Modified	Username	Field	Change
2012-12-05 14:22	mvwieringen	New Issue
2012-12-05 14:22	mvwieringen	Status	new => assigned
2012-12-05 14:22	mvwieringen	Assigned To	=> mvwieringen
2012-12-10 13:58	maik	Category	General => storage daemon
2012-12-24 19:18	mvwieringen	Product Version	12.4.0 => 13.1.0
2013-01-08 17:17	mvwieringen	Time allocated Deleted	2013-01-06: 5,00 h. => deleted
2013-02-13 18:56	mvwieringen	Changeset attached	=> bareos master 01e4409e
2013-02-13 18:56	mvwieringen	Status	assigned => closed
2013-02-13 18:56	mvwieringen	Resolution	open => fixed
2013-02-13 19:05	mvwieringen	Target Version	=> 13.1.0
2013-02-13 19:05	mvwieringen	Fixed in Version	12.4.0 => 13.1.0
2013-02-13 19:05	mvwieringen	Assigned To	mvwieringen =>
2013-03-04 10:31	mvwieringen	Product Version	=> 13.1.0
2013-03-04 10:31	mvwieringen	Fixed in Version	=> 13.1.0
2013-05-24 12:51	pstorz	Assigned To	=> pstorz
2013-05-24 12:51	pstorz	Status	closed => resolved
2013-06-27 09:01	pstorz	Note Added: 0000482
2013-07-04 13:02	pstorz	Status	resolved => closed
2013-08-13 03:12	~~mvwieringen adm~~	Changeset attached	=> bareos master d4431a51
2013-08-13 03:12	~~mvwieringen adm~~	Note Added: 0000623
2013-08-13 03:12	~~mvwieringen adm~~	Assigned To	pstorz => mvwieringen adm
2013-08-13 03:12	~~mvwieringen adm~~	Status	closed => resolved
2013-08-13 08:53	~~mvwieringen adm~~	Assigned To	mvwieringen adm =>
2013-08-13 08:53	~~mvwieringen adm~~	Status	resolved => closed
2015-03-25 16:51	mvwieringen	Changeset attached	=> bareos2015 bareos-13.2 edaf72d8
2015-03-25 16:51	mvwieringen	Note Added: 0001429
2015-03-25 16:51	mvwieringen	Status	closed => resolved
2015-03-25 19:19	joergs	Note Added: 0001584
2015-03-25 19:19	joergs	Status	resolved => closed

Reporting new Issues is disabled, please Report new Issues at https://github.com/bareos/bareos/issues

View Issue Details

Activities

Related Changesets

Issue History