Changesets: bareos

Search Browse Back to Index

master d4431a51

2013-05-04 20:37

mvwieringen adm

Ported: N/A

Details Diff		 Initial working version of SD-SD replication.

This code implements the replication of data between two Storage Daemons
used by the migrate and copy jobs. This is an extenstion of the existing
migration and copy job functionality. The old code already could select
two different storage resources on two different Storage Daemons but it
would then only connect to the write storage and ask there for the read
storage which is ofcourse not known there. This code checks to see if
the read and write storage is located on diffferent Storage Daemons by
comparing the address, SDport and password of the remote storage daemon.

If it detects there are two Storage Daemons involved it will use the
new SD-SD communication protocol to perform a replicate of the data from
the read storage to the write storage.

This patch implements the following:
- Refactored code for authenticating a daemon, this was refactored so
that clients like the bconsole and qt-console from now on use the
now working authenticate_with_director method of the bsock class so
we don't duplicate code to much. Also the other authentication was
refactored as part of this overhaul of the authentication system.
There are now generic functions which allow to authenticate a certain
daemon and a specific authentication is implemented using these
functions. For the SD-SD authentication there is some extra code to be
able to have one storage daemon start the challenge and the let the
other response otherwise the authentication would never work as both
challenge each other at the same time.
- Refactored a lot of code to properly use a boolean when things are
a boolean and not polute the code with assigning true to an integer
variable. That by incident false is 0 and true is 1 doesn't mean
that you shouldn't just call a variable of the type bool and mix
and match integer and boolean all over the code. A lot done more
to come.
- Most protocol messages are now put into static char variables in the
code instead of defining them directly in the code call. This was done
in some parts of the code but not nearly enough this is a good step
in the right direction.
- You can bandwith limit your replication sessions in multiple ways.
- set a bandwidth limit in the migration/copy job
- set a bandwitdh limit for the writing SD in the SD definition
in the director config.
- set a bandwitdh limit for the reading SD in the SD definition
in the director config.
- set a bandwidth limit on the reading SD in the storage daemon config
- set a bandwidth limit on the writing SD in the storage daemon config
- change the bandwith via the console setbandwidth cmd.
- The already allocated mig_jcr for a migration and copy Job is used
to track the communication with the writing SD. The reading SD is
controlled via the normal jcr structure. For this to work the mig_jcr
is created without a so called Messages resource so when using the
mig_jcr for controlling the writing SD we don't get those nasty mail
and database logging as if the Job actually did run.
- A new SD-SD communication protocol is implemented and this can be
extended in the future to perform other SD-SD protocol features. For
now the generic listener of the SD will understand the following
incoming connections:
- FD - SD backup and restore sessions which start with the following
sequence: Hello Start Job <jobname>
- SD - SD replication of data which start with the following
sequence: Hello Start Storage Job <jobname>
- DIR - SD controlling sessions for controlling the SD from the
director.
- The old one Storage Daemon migrate and copy is still available and
when the director finds out both the read and write storage is one
the same SD it will use this optimized and fastest code path where
the SD reads the data and directly writes the data to other media
available to the same SD. SD-SD network replication is only used when
the two SDs are determined to be different this way we only use the
network when really neeeded and the one SD replication using internal
stuctures is way faster then any loopback network send protocol.
- Comments are fixed along the way.
- Parts of lib/message.c are reindented as it used the wrong indention
levels.

Fixes 0000007: Job migration/copy between different SDs		 Affected Issues
0000007
mod - src/win32/qt-console/bat.pro Diff File
mod - src/win32/console/Makefile Diff File
mod - src/stored/wait.c Diff File
mod - src/stored/stored_conf.h Diff File
mod - src/stored/stored_conf.c Diff File
mod - src/stored/stored.c Diff File
mod - src/stored/status.c Diff File
add - src/stored/sd_cmds.c Diff File
mod - src/stored/reserve.c Diff File
mod - src/stored/record.h Diff File
mod - src/stored/read.c Diff File
mod - src/stored/protos.h Diff File
mod - src/stored/mac.c Diff File
mod - src/stored/job.c Diff File
mod - src/stored/fd_cmds.c Diff File
mod - src/stored/dircmd.c Diff File
mod - src/stored/btape.c Diff File
mod - src/stored/bscan.c Diff File
mod - src/stored/bls.c Diff File
mod - src/stored/bextract.c Diff File
mod - src/stored/bcopy.c Diff File
mod - src/stored/authenticate.c Diff File
mod - src/stored/askdir.c Diff File
mod - src/stored/append.c Diff File
mod - src/stored/acquire.c Diff File
mod - src/stored/Makefile.in Diff File
rm - src/qt-console/bcomm/dircomm_auth.cpp Diff File
mod - src/qt-console/bcomm/dircomm.h Diff File
mod - src/qt-console/bcomm/dircomm.cpp Diff File
mod - src/qt-console/bat.pro.in Diff File
mod - src/lib/tls.c Diff File
mod - src/lib/protos.h Diff File
mod - src/lib/message.h Diff File
mod - src/lib/message.c Diff File
mod - src/lib/cram-md5.c Diff File
mod - src/lib/bsock.h Diff File
mod - src/lib/bsock.c Diff File
mod - src/lib/base64.c Diff File
mod - src/include/jcr.h Diff File
mod - src/filed/verify_vol.c Diff File
mod - src/filed/verify.c Diff File
mod - src/filed/status.c Diff File
mod - src/filed/restore.c Diff File
mod - src/filed/protos.h Diff File
mod - src/filed/job.c Diff File
mod - src/filed/filed.c Diff File
mod - src/filed/backup.c Diff File
mod - src/filed/authenticate.c Diff File
mod - src/filed/accurate.c Diff File
mod - src/dird/verify.c Diff File
mod - src/dird/vbackup.c Diff File
mod - src/dird/ua_status.c Diff File
mod - src/dird/ua_cmds.c Diff File
mod - src/dird/sd_cmds.c Diff File
mod - src/dird/restore.c Diff File
mod - src/dird/protos.h Diff File
mod - src/dird/ndmp_dma.c Diff File
mod - src/dird/msgchan.c Diff File
mod - src/dird/mountreq.c Diff File
mod - src/dird/migrate.c Diff File
mod - src/dird/job.c Diff File
mod - src/dird/getmsg.c Diff File
mod - src/dird/fd_cmds.c Diff File
mod - src/dird/dird_conf.h Diff File
mod - src/dird/dird_conf.c Diff File
mod - src/dird/catreq.c Diff File
mod - src/dird/backup.c Diff File
mod - src/dird/authenticate.c Diff File
mod - src/console/console.c Diff File
rm - src/console/authenticate.c Diff File
mod - src/console/Makefile.in Diff File

master 9096704e

2013-05-04 20:34

mvwieringen adm

Ported: N/A

Details Diff		 Implementation of an allowed job command keyword

Implement an allowed Job command keyword in the filed that sets which
of the different Job commands we should accept. Currently the filed
will execute any Job command unless people specify the -b or -r option
(backup only/restore only) which makes it a serious security concern by
much of the bigger customers security officers.

This new keyword implements a per director and a global list which is used
as a fallback when a specific one for a specific director is not configured.

We won't filter all the commands issued by the director only
the important ones. Things which we allow to be filtered are:

- backup
- restore
- verify
- estimate
- runscript

Fixes 0000032: Implementation of an allowed job command keyword		 Affected Issues
0000032
mod - src/filed/job.c Diff File
mod - src/filed/filed_conf.h Diff File
mod - src/filed/filed_conf.c Diff File

master 334cfd5f

2013-05-04 20:34

mvwieringen adm

Ported: N/A

Details Diff		 Implementation of an allowed scriptdir keyword

Implement an allowed scriptdir keyword in the filed that sets the
directories in which any runscript must be located so we can limit
the attack surface of the filedaemon.

Currently the filed will execute any script in any directory which
makes it a serious security concern by much of the bigger customers
security officers.

This new keyword implemented per director and a global one which is
used as a fallback when a specific one for a specific director is not
configured.

Fixes 0000031: Implementation of an allowed scriptdir keyword		 Affected Issues
0000031
mod - src/lib/runscript.h Diff File
mod - src/lib/runscript.c Diff File
mod - src/filed/job.c Diff File
mod - src/filed/filed_conf.h Diff File
mod - src/filed/filed_conf.c Diff File

master d72cca4b

2013-05-04 20:34

mvwieringen adm

Ported: N/A

Details Diff		 Allow cancel by JobId on storage daemon.

Sometimes the Director already removed the Job from its running queue
but the Storage daemon still thinks it doing a backup (or other Job)
and you cannot cancel the Job from within a console anymore. This piece
of code allows you to cancel a Storage Daemon Job by JobId. E.g. do
a status sd on the Storage Daemon make sure what Job you want to cancel
and issue a cancel storage=<Storage Daemon> Jobid=<JobId> This way you
can also remove a Job that blocks any other Jobs from running without
the need to restart the whole Storage Daemon.

Fixes 0000013: Allow cancel by JobId on storage daemon.		 Affected Issues
0000013
mod - src/stored/dircmd.c Diff File
mod - src/dird/ua_cmds.c Diff File
mod - src/dird/sd_cmds.c Diff File
mod - src/dird/protos.h Diff File

master fb2fda7d

2013-05-04 20:34

mvwieringen adm

Ported: N/A

Details Diff		 Tweak indent.
mod - src/dird/dird_conf.c Diff File

master 30e30bfd

2013-05-04 20:34

mvwieringen adm

Ported: N/A

Details Diff		 Print BSR to memory buffer.

Print the BSR to a memory buffer and write that out in one go.
This allows you to use a printed BSR in any reporting.
mod - src/dird/protos.h Diff File
mod - src/dird/bsr.c Diff File

master 0a1711cf

2013-05-04 20:33

mvwieringen adm

Ported: N/A

Details Diff		 Switch to new version number.

Start adding new features.
mod - src/include/version.h Diff File
mod - platforms/win32/winbareos64.spec Diff File
mod - platforms/win32/winbareos32.spec Diff File
mod - platforms/win32/winbareos-nsi.spec Diff File
mod - platforms/rpms/bareos.spec Diff File
mod - platforms/deb/bareos-xUbuntu_12.04.dsc Diff File
mod - platforms/deb/bareos-Debian_6.0.dsc Diff File
mod - debian/changelog Diff File

master 3beca800

2013-05-04 13:54

Marco van Wieringen

Ported: N/A

Details Diff		 Rebuild configure.
mod - configure Diff File

master f5d7ed6f

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Tweak sql update function.

Use only one function to do the database update and pass as argument the
minimum number of affected rows that must be met.
mod - src/cats/cats.h Diff File
mod - src/cats/sql.c Diff File

master db1f55a9

2013-05-04 13:51

Joerg Steffens


Committer: Marco van Wieringen

Ported: N/A

Details Diff		 adapted for ubuntu, some improvements on packaging

Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>
mod - debian/bareos-common.dirs Diff File
mod - debian/bareos-common.postinst Diff File
mod - debian/bareos-director.install Diff File
mod - debian/bareos-filedaemon.install Diff File
add - debian/bareos-storage.dirs Diff File
mod - debian/bareos-storage.install Diff File
mod - debian/bareos-storage.postinst Diff File
mod - debian/control Diff File
mod - debian/rules Diff File
mod - platforms/deb/bareos-Debian_6.0.dsc Diff File
mod - platforms/deb/bareos-xUbuntu_12.04.dsc Diff File
mod - platforms/debian/bareos-dir.in Diff File
mod - platforms/rpms/bareos.spec Diff File
mod - platforms/ubuntu/bareos-dir.in Diff File
mod - platforms/ubuntu/bareos-fd.in Diff File
mod - platforms/ubuntu/bareos-sd.in Diff File
mod - scripts/bareos-config Diff File

master 72a68c2c

2013-05-04 13:51

Joerg Steffens


Committer: Marco van Wieringen

Ported: N/A

Details Diff		 travis-ci.org integration

Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>
add - .travis.yml Diff File
add - test/all Diff File

master 0fe2716c

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Fix typo.
mod - src/dird/ua_cmds.c Diff File

master 5a1383e3

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Don't use Zlib specific types.

Ulong and Bytef are ZLIB specific and on platforms that don't have
zlib this gives compile errors.
mod - src/filed/backup.h Diff File

master c8ce10ef

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Fix dummy function defined.
mod - src/filed/compression.c Diff File

master b74d3df5

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Fix compile error.
mod - src/filed/backup.h Diff File

master 39ac1b10

2013-05-04 13:51

Joerg Steffens


Committer: Marco van Wieringen

Ported: N/A

Details Diff		 fixed comment
mod - README.md Diff File

master 223eac9d

2013-05-04 13:51

Philipp Storz


Committer: Marco van Wieringen

Ported: N/A

Details Diff		 Send the catalog backup bootstrap file by mail

Having the bootstrap of the backupcatalog job is crucial
for disaster recovery.
Unfortunately, the default config didn't do that until now.

Now we automatically mail the bootstrap file of the backupcatalog job.

Also, we have now extended the BackupCatalog FileSet to include the
sysconfdir.

Fixes 0000024: Automatic sending of BSR (at least of the Catalog DB) via email
mod - src/dird/bareos-dir.conf.in Diff File

master 29faf185

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Set a new default for the restore dir.
mod - src/dird/bareos-dir.conf.in Diff File

master 36f40944

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Set a new default for the restore dir.
mod - src/dird/bareos-dir.conf.in Diff File

master 51e4d8ef

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Fix compiler warning.

Now that we don't use the old conversion string we can get ride of the
caps variable.
mod - src/lib/bsnprintf.c Diff File

master dde644a3

2013-05-04 13:51

Joerg Steffens


Committer: Marco van Wieringen

Ported: N/A

Details Diff		 fixes ubuntu configuration

Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>
mod - autoconf/configure.in Diff File

master 16b15624

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Fix Makefile rules for generating configure and config.h.in
mod - Makefile.in Diff File

master d786e1bd

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Set some more sane defaults.

Still its better to explicitly set the dirs to where you want
them to be and to not depend on changing defaults.
mod - autoconf/configure.in Diff File

master 6d7e2517

2013-05-04 13:51

Marco van Wieringen

Ported: N/A

Details Diff		 Drop temp file.
rm - src/qt-console/aa Diff File

master c2ec319a

2013-05-04 13:51

Radosław Korzeniewski


Committer: Marco van Wieringen

Ported: N/A

Details Diff		 Fix %f conversion in bsnprintf.

Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>
mod - src/lib/bsnprintf.c Diff File
 First  Prev  1 2 3 ... 60 ... 120 ... 180 ... 240 ... 300 ... 360 ... 420 ... 480 ... 540 ... 600 ... 627 628 629 630 631 632 633 ... 635 636 637  Next  Last