View Issue Details

Related ChangesetsJump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000008bareos-corefile daemonpublic2012-12-05 14:302015-03-25 19:19
Reportermvwieringen Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
Status closedResolutionfixed 
PlatformOpenIndianaOSOpenIndianaOS Versionb151a
Product Version13.4.0 
Summary0000008: Handle Windows Encrypted Files using Win raw encryption
DescriptionMake it possible to backup and restore Encypted Files from and to
Windows systems without the need to decrypt it by using the raw
encryption functions API (see:
http://msdn2.microsoft.com/en-us/library/aa363783.aspx)
that is provided for that reason by Microsoft.
If a file is encrypted could be examined by evaluating the
FILE_ATTRIBUTE_ENCRYTED flag of the GetFileAttributes function.

For each file backed up or restored by FD on Windows, check if
the file is encrypted; if so then use OpenEncryptedFileRaw,
ReadEncryptedFileRaw, WriteEncryptedFileRaw,
CloseEncryptedFileRaw instead of BackupRead and BackupWrite
API calls.
Additional InformationWithout the usage of this interface the fd-daemon running
under the system account can't read encypted Files because
the key needed for the decrytion is missed by them. As a result
actually encrypted files are not backed up
by bacula and also no error is shown while missing these files.

Using xxxEncryptedFileRaw API would allow to backup and
restore EFS-encrypted files without decrypting their data.
Note that such files cannot be restored "portably" (at least,
easily) but they would be restoreable to a different (or
reinstalled) Win32 machine; the restore would require setup
of a EFS recovery agent in advance, of course, and this shall
be clearly reflected in the documentation, but this is the
normal Windows SysAdmin's business.

When "portable" backup is requested the EFS-encrypted files
shall be clearly reported as errors.
See MSDN on the "Backup and Restore of Encrypted Files" topic:
http://msdn.microsoft.com/en-us/library/aa363783.aspx
Maybe the EFS support requires a new flag in the database for
each file, too?

Unfortunately, the implementation is not as straightforward as
1-to-1 replacement of BackupRead with ReadEncryptedFileRaw,
requiring some FD code rewrite to work with encrypted-file-related callback functions.
TagsNo tags attached.

Relationships

child of 0000228 closed Some windows file attributes are not handled. 

Activities

mvwieringen

mvwieringen

2012-12-05 14:33

developer   ~0000014

First steps to determine if a file is encrypted on Windows are implemented
and the encryption flag is kept as the SGID bit in the permissions of the file.
This is analog to the way hidden files are implemented they have the SUID bit
set.

Also added the entry points into the library for the encryption and decryption
functions on windows. Reading the encrypted data already fits well with the
new rewritten filed code. For restoring however we need to do some more leg work.

e.g. more to come when I get to it (and feel like it).
maik

maik

2013-03-22 16:43

administrator   ~0000296

will be done, if time is available....
mvwieringen

mvwieringen

2013-12-09 15:17

developer   ~0000753

Fix committed to bareos master branch with changesetid 1361.
mvwieringen

mvwieringen

2015-03-25 16:51

developer   ~0001486

Fix committed to bareos2015 bareos-14.2 branch with changesetid 4974.
joergs

joergs

2015-03-25 19:19

developer   ~0001637

Due to the reimport of the Github repository to bugs.bareos.org, the status of some tickets have been changed. These tickets will be closed again.
Sorry for the noise.

Related Changesets

bareos: master fa9aabd5

2013-12-09 15:12

mvwieringen

Ported: N/A

Details Diff		 Bring windows support up to Windows 2012 server.

This commit bring the windows code to support all new
features that have been added to windows up till windows
2012 server. We also reindented a lot of code along the
way to be more readable and fixes most comments.

The actual added features:
- Volume Mountpoints support (VMP).

Windows for some version now supports so called Volume
MountPoint which allow you to mount a volume somewhere
in the filesystem layout analog to what UNIX has been
doing since it inception without assigning a drive letter.
We need to make sure we also add the VMPs to the VSS snapshot
list otherwise the backup will fail.

- Use GetFileInformationByHandleEx to get real changetime.

As it seems ctime in the stat struct on UNIX means for
some time now the last change time of the inode and not
the creation time. On Windows we need a special API to
get the actual change time named GetFileInformationByHandleEx().
While implementing this new feature we reformated some of
the layout and dropped some unused code.

- Windows dedup support

Windows 2012 has dedup support which needs handling.

To get rehydrated data when reading the file you should open
the file without the FILE_FLAG_OPEN_REPARSE_POINT flag set.
If you want to read the deduped data effeciently you need to
write new code which interfaces to a new API which is currently
out of scope. The original file on the filesystem is replaced
with a so called reparse point so if we open the reparse point
instead of where it points to we get data which has a pointer
to a map of all the data streams and chunks required to
“rehydrate” the file and not its actual data.

We now changed the stat emulation functions to detect the special
reparse point with type IO_REPARSE_TAG_DEDUP which indicates
a deduped file.

When that is set we set a special bit on the file so we know we
need to open the file with the O_NOFOLLOW flag which will open
the file without the FILE_FLAG_OPEN_REPARSE_POINT flag set.

More info at:

http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx

- Store all file attributes

Windows has gathered quite some special specific file
flags over the years but not all are saved during backup so
some are never restored by the restore process. The most
important ones are the ARCHIVE flag which is "misused" by
some programs for storing some special information. Others
that are known not to be stored are the COMPRESSED flag which
means that a restored file looses it and will be restored as
an uncompressed file.

As we are running out of opportunities to store the special
flags as some mode in the st_mode member of the stat struct
a redesign was needed to store the unique file flag somewhere
else. We are reusing the st_rdev field. That is already used for
reparse points in older versions but as those are never restored
anyway we can reuse them without breaking backwards compatability.

- Support for Windows EFS filesystems

Windows has for quite some time now support for a so called EFS
filesystem. This is an encrypted filesystem, to be able to backup the
data and to restore it we need to use a special API. With this API you
in essence export the data on backup and import it on restore. This
way you never have access to the unencrypted data but just import and
export the encrypted data. This is the cleanest way of handling
encryption by just seeing the data as some opaque data and not try to
do anything special with it. The problem with the restore API is
however that it kind of works in a completely different way then a
normal read/write API. You call a import function and that call the
read callback multiple times until all data is imported/restored. As
we don't want to rewrite the whole internal works of the filed for
this we now create a seperate thread that runs during the whole
restore of a filesystem and behaves as a consumer/producer problem
(well known in the pthreads world). In this solution the producer is
the filed that produces encrypted data out of the incoming data
streams from the storage daemon and the seperate thread consumes that
data by returning it as encrypted data in the callback function issues
by the special EFS import API. There is some synchronization between
the different threads so it knows when one file is done and it should
end the import of that file. We also keep some of the buffers
lingering around so we don't need to do malloc/free/malloc loops all
the time, we cleanup the whole thread and the buffers used at the end
of the whole restore process.

Fixes 0000228: Some windows file attributes are not handled.
Fixes 0000008: Handle Windows Encrypted Files using Win raw encryption

Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>		 Affected Issues
0000008, 0000228
mod - src/filed/backup.c Diff File
mod - src/filed/crypto.c Diff File
mod - src/filed/dir_cmd.c Diff File
mod - src/filed/fd_plugins.c Diff File
mod - src/filed/filed.c Diff File
mod - src/filed/fileset.c Diff File
mod - src/filed/protos.h Diff File
mod - src/filed/restore.c Diff File
mod - src/filed/verify.c Diff File
mod - src/findlib/attribs.c Diff File
mod - src/findlib/bfile.c Diff File
mod - src/findlib/bfile.h Diff File
mod - src/findlib/create_file.c Diff File
mod - src/findlib/find.c Diff File
mod - src/findlib/find.h Diff File
mod - src/findlib/find_one.c Diff File
mod - src/findlib/mkpath.c Diff File
mod - src/findlib/protos.h Diff File
mod - src/include/baconfig.h Diff File
mod - src/include/bareos.h Diff File
mod - src/include/bc_types.h Diff File
mod - src/include/jcr.h Diff File
mod - src/lib/Makefile.in Diff File
add - src/lib/cbuf.c Diff File
add - src/lib/cbuf.h Diff File
mod - src/plugins/filed/fd_common.h Diff File
mod - src/win32/Makefile Diff File
mod - src/win32/Makefile.inc Diff File
mod - src/win32/compat/compat.c Diff File
mod - src/win32/compat/include/compat.h Diff File
mod - src/win32/compat/include/mingwconfig.h Diff File
mod - src/win32/compat/print.c Diff File
mod - src/win32/compat/winapi.c Diff File
mod - src/win32/console/Makefile Diff File
mod - src/win32/filed/Makefile Diff File
mod - src/win32/filed/vss.c Diff File
mod - src/win32/filed/vss_generic.c Diff File
mod - src/win32/findlib/Makefile Diff File
add - src/win32/findlib/win32.c Diff File
mod - src/win32/include/vss.h Diff File
mod - src/win32/include/winapi.h Diff File
mod - src/win32/lib/Makefile Diff File
mod - src/win32/plugins/filed/Makefile Diff File

bareos2015: bareos-14.2 2242e405

2013-12-09 16:12

mvwieringen

Ported: N/A

Details Diff		 Bring windows support up to Windows 2012 server.

This commit bring the windows code to support all new
features that have been added to windows up till windows
2012 server. We also reindented a lot of code along the
way to be more readable and fixes most comments.

The actual added features:
- Volume Mountpoints support (VMP).

Windows for some version now supports so called Volume
MountPoint which allow you to mount a volume somewhere
in the filesystem layout analog to what UNIX has been
doing since it inception without assigning a drive letter.
We need to make sure we also add the VMPs to the VSS snapshot
list otherwise the backup will fail.

- Use GetFileInformationByHandleEx to get real changetime.

As it seems ctime in the stat struct on UNIX means for
some time now the last change time of the inode and not
the creation time. On Windows we need a special API to
get the actual change time named GetFileInformationByHandleEx().
While implementing this new feature we reformated some of
the layout and dropped some unused code.

- Windows dedup support

Windows 2012 has dedup support which needs handling.

To get rehydrated data when reading the file you should open
the file without the FILE_FLAG_OPEN_REPARSE_POINT flag set.
If you want to read the deduped data effeciently you need to
write new code which interfaces to a new API which is currently
out of scope. The original file on the filesystem is replaced
with a so called reparse point so if we open the reparse point
instead of where it points to we get data which has a pointer
to a map of all the data streams and chunks required to
“rehydrate” the file and not its actual data.

We now changed the stat emulation functions to detect the special
reparse point with type IO_REPARSE_TAG_DEDUP which indicates
a deduped file.

When that is set we set a special bit on the file so we know we
need to open the file with the O_NOFOLLOW flag which will open
the file without the FILE_FLAG_OPEN_REPARSE_POINT flag set.

More info at:

http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx

- Store all file attributes

Windows has gathered quite some special specific file
flags over the years but not all are saved during backup so
some are never restored by the restore process. The most
important ones are the ARCHIVE flag which is "misused" by
some programs for storing some special information. Others
that are known not to be stored are the COMPRESSED flag which
means that a restored file looses it and will be restored as
an uncompressed file.

As we are running out of opportunities to store the special
flags as some mode in the st_mode member of the stat struct
a redesign was needed to store the unique file flag somewhere
else. We are reusing the st_rdev field. That is already used for
reparse points in older versions but as those are never restored
anyway we can reuse them without breaking backwards compatability.

- Support for Windows EFS filesystems

Windows has for quite some time now support for a so called EFS
filesystem. This is an encrypted filesystem, to be able to backup the
data and to restore it we need to use a special API. With this API you
in essence export the data on backup and import it on restore. This
way you never have access to the unencrypted data but just import and
export the encrypted data. This is the cleanest way of handling
encryption by just seeing the data as some opaque data and not try to
do anything special with it. The problem with the restore API is
however that it kind of works in a completely different way then a
normal read/write API. You call a import function and that call the
read callback multiple times until all data is imported/restored. As
we don't want to rewrite the whole internal works of the filed for
this we now create a seperate thread that runs during the whole
restore of a filesystem and behaves as a consumer/producer problem
(well known in the pthreads world). In this solution the producer is
the filed that produces encrypted data out of the incoming data
streams from the storage daemon and the seperate thread consumes that
data by returning it as encrypted data in the callback function issues
by the special EFS import API. There is some synchronization between
the different threads so it knows when one file is done and it should
end the import of that file. We also keep some of the buffers
lingering around so we don't need to do malloc/free/malloc loops all
the time, we cleanup the whole thread and the buffers used at the end
of the whole restore process.

Fixes 0000228: Some windows file attributes are not handled.
Fixes 0000008: Handle Windows Encrypted Files using Win raw encryption

Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>		 Affected Issues
0000008, 0000228
mod - src/filed/backup.c Diff File
mod - src/filed/crypto.c Diff File
mod - src/filed/dir_cmd.c Diff File
mod - src/filed/fd_plugins.c Diff File
mod - src/filed/filed.c Diff File
mod - src/filed/fileset.c Diff File
mod - src/filed/protos.h Diff File
mod - src/filed/restore.c Diff File
mod - src/filed/verify.c Diff File
mod - src/findlib/attribs.c Diff File
mod - src/findlib/bfile.c Diff File
mod - src/findlib/bfile.h Diff File
mod - src/findlib/create_file.c Diff File
mod - src/findlib/find.c Diff File
mod - src/findlib/find.h Diff File
mod - src/findlib/find_one.c Diff File
mod - src/findlib/mkpath.c Diff File
mod - src/findlib/protos.h Diff File
mod - src/include/baconfig.h Diff File
mod - src/include/bareos.h Diff File
mod - src/include/bc_types.h Diff File
mod - src/include/jcr.h Diff File
mod - src/lib/Makefile.in Diff File
add - src/lib/cbuf.c Diff File
add - src/lib/cbuf.h Diff File
mod - src/plugins/filed/fd_common.h Diff File
mod - src/win32/Makefile Diff File
mod - src/win32/Makefile.inc Diff File
mod - src/win32/compat/compat.c Diff File
mod - src/win32/compat/include/compat.h Diff File
mod - src/win32/compat/include/mingwconfig.h Diff File
mod - src/win32/compat/print.c Diff File
mod - src/win32/compat/winapi.c Diff File
mod - src/win32/console/Makefile Diff File
mod - src/win32/filed/Makefile Diff File
mod - src/win32/filed/vss.c Diff File
mod - src/win32/filed/vss_generic.c Diff File
mod - src/win32/findlib/Makefile Diff File
add - src/win32/findlib/win32.c Diff File
mod - src/win32/include/vss.h Diff File
mod - src/win32/include/winapi.h Diff File
mod - src/win32/lib/Makefile Diff File
mod - src/win32/plugins/filed/Makefile Diff File

Issue History

Date Modified Username Field Change
2012-12-05 14:30 mvwieringen New Issue
2012-12-05 14:30 mvwieringen Status new => assigned
2012-12-05 14:30 mvwieringen Assigned To => mvwieringen
2012-12-05 14:33 mvwieringen Note Added: 0000014
2012-12-10 13:59 maik Category General => file daemon
2013-02-26 19:33 mvwieringen Status assigned => new
2013-02-26 19:33 mvwieringen Assigned To mvwieringen =>
2013-03-08 16:23 mvwieringen Product Version => 13.2.0
2013-03-22 16:43 maik Status new => confirmed
2013-03-22 16:43 maik Note Added: 0000296
2013-06-05 13:04 mvwieringen Product Version 13.2.0 => 13.4.0
2013-10-08 09:04 mvwieringen Assigned To => mvwieringen
2013-10-08 09:04 mvwieringen Status confirmed => acknowledged
2013-10-08 09:33 mvwieringen Status acknowledged => assigned
2013-12-09 15:17 mvwieringen Changeset attached => bareos master fa9aabd5
2013-12-09 15:17 mvwieringen Note Added: 0000753
2013-12-09 15:17 mvwieringen Status assigned => resolved
2013-12-09 15:17 mvwieringen Resolution open => fixed
2014-12-01 12:03 joergs Relationship added child of 0000228
2014-12-01 12:04 joergs Status resolved => closed
2014-12-01 12:04 joergs Assigned To mvwieringen =>
2015-03-25 16:51 mvwieringen Changeset attached => bareos2015 bareos-14.2 2242e405
2015-03-25 16:51 mvwieringen Note Added: 0001486
2015-03-25 16:51 mvwieringen Status closed => resolved
2015-03-25 19:19 joergs Note Added: 0001637
2015-03-25 19:19 joergs Status resolved => closed