View Issue Details

IDProjectCategoryView StatusLast Update
0000253bareos-corefile daemonpublic2019-12-18 15:25
ReporterAndiH Assigned To 
PrioritynormalSeverityfeatureReproducibilityalways
Status closedResolutionfixed 
PlatformWindowsOSallOS Versionall
Product Version13.2.1 
Summary0000253: Enable concurrent jobs on Windows platforms
DescriptionOn Unix concurrent backup jobs run as expected, so it is possible to backup different volumes concurrently to get better speed in situations which might be CPU bound for example when using data encryption. With Windows FD the concurrency is limited to 1 because of global VSS lock. This is not a Windows limitation from what i can see, but a limitation of the Windows FD code.
It would be nice to get this feature to lower the pain to backup somewhat bigger Windows file servers with data encryption.

With one core saturated (Windows) we get around 5..30MByte per second, a Linux machine with similar hardware and two concurrent jobs can deliver around 12...70MBytes per second.

Thanks

Andreas
TagsNo tags attached.

Relationships

child of 0000625 closedmaik Release bareos-15.2.3 

Activities

maik

maik

2013-11-29 15:52

administrator   ~0000733

we put it into a feature request. This will induce significant code redesign.

Have you tried to run 2 separate FDs on the windows machine? I am not sure, if it works but it might be a workaround.
mvwieringen

mvwieringen

2013-11-29 16:37

developer   ~0000734

Note to self.

VSS implementation uses a global variable for keeping track of the VSS its
performing. There is a mutex lock that locks any two threads for ever entering
the critical section of the VSS session e.g. from start backup till end backup
and from start restore till end restore when there is some meta data to restore.
When there is no meta data to restore the restore will restore without VSS enabled.

Its seems that this is not easy to solve, we do initialize the COM subsystem
for multithread support (as we needed that for the MSSQL plugin.) so it might be
possible to have now two or more handles to the VSS subsystem and as the the
fileset determines what gets snapshotted (for now only drives but soon also
VMPs) I cannot imagine you can only have one set of VSS data.

Some ideas:

- If we cannot get ride of the global VSS client we might look at stuffing
  it into thread specific storage.
- First figure out if you can even have multiple sets of VSS data in one
  process (e.g. how multithreaded VSS really is.)

All in all it means yet an other serious redesign of the windows subsytem
which have to wait until we stabilize the other pending windows patches.
AndiH

AndiH

2013-12-18 18:40

reporter   ~0000763

No, we have not tried running two FDs on the same machine. If time permits i will test it.
From what i can see here

http://msdn.microsoft.com/en-us/library/windows/desktop/aa384629%28v=vs.85%29.aspx

there should be no limit regarding thread safety, but i'm not a developer at all.

As said would be nice to have, at least to be conform with documentation and Unix behaviour.

Thanks

Andreas
mvwieringen

mvwieringen

2015-01-26 15:44

developer   ~0001219

Given that the whole windows code uses a slack of global variables this means
getting at least rid of those before even being able to try instantiating
multiple VSS instances.

Over the holiday season dissected the whole problem and its one big mine field
of global variables and caches. So that needs to be cleaned to be able to not
get worse performance as one job destroys the cache of an other.

Some small progress made but needs lots of testing and a lot more work. For now
not really worth the effort.
mvwieringen

mvwieringen

2015-12-24 15:00

developer   ~0002061

Fix committed to bareos bareos-15.2 branch with changesetid 6018.

Related Changesets

bareos: bareos-15.2 37a083f5

2015-12-24 11:15

mvwieringen

Ported: N/A

Details Diff
vss: Get rid of one VSS client at a time.

The VSS code is currently one big minefield with global variables
all over the place.

These changes make it a bit better at the costs of using some so
called thread specific data pointers e.g. using
pthread_set_specific() and pthread_get_specific().

The big change is that we setup two TSD keys which control the
Thread Specific Data. There is one key which holds the so called
UTF8 to UCS2 cache which caches the last conversion done as this
conversion is rather expensive and is done multiple times. The
second key is used for registering the callback for the VSS
pathconvert functions.

In the FILED code we now keep track of the VSS instance used
using a variable in the JCR instead of using a global VSS instance.

As each Job uses one thread we could now run multiple Jobs which
shouldn't clobber each others caches, have callbacks for VSS if
they use VSS and have their own instance of the VSS class.

By removing these limits we also had to fix the following problems:
- CoInitializeSecurity may only be called once.
- Wait for Snapshotset to complete

Fixes 0000253: Enable concurrent jobs on Windows platforms

Signed-off-by: Philipp Storz <philipp.storz@bareos.com>
Affected Issues
0000253
mod - src/filed/backup.c Diff File
mod - src/filed/dir_cmd.c Diff File
mod - src/filed/fd_plugins.c Diff File
mod - src/filed/filed.h Diff File
mod - src/filed/status.c Diff File
mod - src/findlib/acl.c Diff File
mod - src/findlib/attribs.c Diff File
mod - src/findlib/create_file.c Diff File
mod - src/findlib/find.c Diff File
mod - src/findlib/find.h Diff File
mod - src/findlib/find_one.c Diff File
mod - src/findlib/match.c Diff File
mod - src/findlib/xattr.c Diff File
mod - src/include/jcr.h Diff File
mod - src/lib/jcr.c Diff File
mod - src/win32/compat/compat.c Diff File
mod - src/win32/compat/include/compat.h Diff File
mod - src/win32/compat/winapi.c Diff File
mod - src/win32/filed/Makefile Diff File
mod - src/win32/filed/vss.c Diff File
mod - src/win32/filed/vss_generic.c Diff File
mod - src/win32/filed/who.h Diff File
mod - src/win32/findlib/win32.c Diff File
mod - src/win32/generic/main.c Diff File
mod - src/win32/include/vss.h Diff File

Issue History

Date Modified Username Field Change
2013-11-29 09:57 AndiH New Issue
2013-11-29 15:52 maik Note Added: 0000733
2013-11-29 15:52 maik Status new => acknowledged
2013-11-29 16:37 mvwieringen Note Added: 0000734
2013-12-18 18:40 AndiH Note Added: 0000763
2015-01-26 15:44 mvwieringen Note Added: 0001219
2015-12-24 15:00 mvwieringen Changeset attached => bareos bareos-15.2 37a083f5
2015-12-24 15:00 mvwieringen Note Added: 0002061
2015-12-24 15:00 mvwieringen Status acknowledged => resolved
2015-12-24 15:00 mvwieringen Resolution open => fixed
2016-03-11 14:37 maik Relationship added child of 0000625
2019-12-18 15:25 arogge Status resolved => closed