0000228: Some windows file attributes are not handled.

ID	Project	Category	View Status	Date Submitted	Last Update

0000228	bareos-core	file daemon	public	2013-09-27 07:54	2015-03-25 19:19

Reporter	mvwieringen	Assigned To
Priority	normal	Severity	feature	Reproducibility	always
Status	closed	Resolution	fixed
Platform	Windows 7	OS	Windows	OS Version	7
Product Version	12.4.5

Summary	0000228: Some windows file attributes are not handled.
Description	Windows has gathered quite some special specific file flags over the years but not all are saved during backup so some are never restored by the restore process. The most important ones are the ARCHIVE flag which is "misused" by some programs for storing some special information. Others that are known not to be stored are the COMPRESSED flag which means that a restored file looses it and will be restored as an uncompressed file.
Additional Information	As we are running out of opportunities to store the special flags as some mode in the st_mode member of the stat struct a redesign is needed to store the unique file flag somewhere else. The idea is to start reusing the st_rdev field, but as that is already used for reparse points we need some special code to be able allow some backwards compatability with the old scheme. In essence that should be no problem as we need to be able to restore older backups anyway so we need some extra code. As this will break compatability we will need to put this under the control of the compatible option. Idea here is to have a compatible option in the windows compatible layer which is enabled by the upper level code based on the setting in the config. Then the internal code knows when it can use the new encoding or when it should use the old.
Tags	No tags attached.

mvwieringen 2013-12-09 15:17 developer ~0000752	Fix committed to bareos master branch with changesetid 1361.

joergs 2014-12-03 08:46 developer ~0001091	published and documented.

mvwieringen 2015-03-25 16:51 developer ~0001485	Fix committed to bareos2015 bareos-14.2 branch with changesetid 4974.

joergs 2015-03-25 19:19 developer ~0001632	Due to the reimport of the Github repository to bugs.bareos.org, the status of some tickets have been changed. These tickets will be closed again. Sorry for the noise.

bareos: master fa9aabd5 2013-12-09 15:12 mvwieringen Ported: N/A Details Diff	Bring windows support up to Windows 2012 server. This commit bring the windows code to support all new features that have been added to windows up till windows 2012 server. We also reindented a lot of code along the way to be more readable and fixes most comments. The actual added features: - Volume Mountpoints support (VMP). Windows for some version now supports so called Volume MountPoint which allow you to mount a volume somewhere in the filesystem layout analog to what UNIX has been doing since it inception without assigning a drive letter. We need to make sure we also add the VMPs to the VSS snapshot list otherwise the backup will fail. - Use GetFileInformationByHandleEx to get real changetime. As it seems ctime in the stat struct on UNIX means for some time now the last change time of the inode and not the creation time. On Windows we need a special API to get the actual change time named GetFileInformationByHandleEx(). While implementing this new feature we reformated some of the layout and dropped some unused code. - Windows dedup support Windows 2012 has dedup support which needs handling. To get rehydrated data when reading the file you should open the file without the FILE_FLAG_OPEN_REPARSE_POINT flag set. If you want to read the deduped data effeciently you need to write new code which interfaces to a new API which is currently out of scope. The original file on the filesystem is replaced with a so called reparse point so if we open the reparse point instead of where it points to we get data which has a pointer to a map of all the data streams and chunks required to ârehydrateâ the file and not its actual data. We now changed the stat emulation functions to detect the special reparse point with type IO_REPARSE_TAG_DEDUP which indicates a deduped file. When that is set we set a special bit on the file so we know we need to open the file with the O_NOFOLLOW flag which will open the file without the FILE_FLAG_OPEN_REPARSE_POINT flag set. More info at: http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx - Store all file attributes Windows has gathered quite some special specific file flags over the years but not all are saved during backup so some are never restored by the restore process. The most important ones are the ARCHIVE flag which is "misused" by some programs for storing some special information. Others that are known not to be stored are the COMPRESSED flag which means that a restored file looses it and will be restored as an uncompressed file. As we are running out of opportunities to store the special flags as some mode in the st_mode member of the stat struct a redesign was needed to store the unique file flag somewhere else. We are reusing the st_rdev field. That is already used for reparse points in older versions but as those are never restored anyway we can reuse them without breaking backwards compatability. - Support for Windows EFS filesystems Windows has for quite some time now support for a so called EFS filesystem. This is an encrypted filesystem, to be able to backup the data and to restore it we need to use a special API. With this API you in essence export the data on backup and import it on restore. This way you never have access to the unencrypted data but just import and export the encrypted data. This is the cleanest way of handling encryption by just seeing the data as some opaque data and not try to do anything special with it. The problem with the restore API is however that it kind of works in a completely different way then a normal read/write API. You call a import function and that call the read callback multiple times until all data is imported/restored. As we don't want to rewrite the whole internal works of the filed for this we now create a seperate thread that runs during the whole restore of a filesystem and behaves as a consumer/producer problem (well known in the pthreads world). In this solution the producer is the filed that produces encrypted data out of the incoming data streams from the storage daemon and the seperate thread consumes that data by returning it as encrypted data in the callback function issues by the special EFS import API. There is some synchronization between the different threads so it knows when one file is done and it should end the import of that file. We also keep some of the buffers lingering around so we don't need to do malloc/free/malloc loops all the time, we cleanup the whole thread and the buffers used at the end of the whole restore process. Fixes 0000228: Some windows file attributes are not handled. Fixes 0000008: Handle Windows Encrypted Files using Win raw encryption Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>	Affected Issues 0000008, 0000228
	mod - src/filed/backup.c	Diff File
	mod - src/filed/crypto.c	Diff File
	mod - src/filed/dir_cmd.c	Diff File
	mod - src/filed/fd_plugins.c	Diff File
	mod - src/filed/filed.c	Diff File
	mod - src/filed/fileset.c	Diff File
	mod - src/filed/protos.h	Diff File
	mod - src/filed/restore.c	Diff File
	mod - src/filed/verify.c	Diff File
	mod - src/findlib/attribs.c	Diff File
	mod - src/findlib/bfile.c	Diff File
	mod - src/findlib/bfile.h	Diff File
	mod - src/findlib/create_file.c	Diff File
	mod - src/findlib/find.c	Diff File
	mod - src/findlib/find.h	Diff File
	mod - src/findlib/find_one.c	Diff File
	mod - src/findlib/mkpath.c	Diff File
	mod - src/findlib/protos.h	Diff File
	mod - src/include/baconfig.h	Diff File
	mod - src/include/bareos.h	Diff File
	mod - src/include/bc_types.h	Diff File
	mod - src/include/jcr.h	Diff File
	mod - src/lib/Makefile.in	Diff File
	add - src/lib/cbuf.c	Diff File
	add - src/lib/cbuf.h	Diff File
	mod - src/plugins/filed/fd_common.h	Diff File
	mod - src/win32/Makefile	Diff File
	mod - src/win32/Makefile.inc	Diff File
	mod - src/win32/compat/compat.c	Diff File
	mod - src/win32/compat/include/compat.h	Diff File
	mod - src/win32/compat/include/mingwconfig.h	Diff File
	mod - src/win32/compat/print.c	Diff File
	mod - src/win32/compat/winapi.c	Diff File
	mod - src/win32/console/Makefile	Diff File
	mod - src/win32/filed/Makefile	Diff File
	mod - src/win32/filed/vss.c	Diff File
	mod - src/win32/filed/vss_generic.c	Diff File
	mod - src/win32/findlib/Makefile	Diff File
	add - src/win32/findlib/win32.c	Diff File
	mod - src/win32/include/vss.h	Diff File
	mod - src/win32/include/winapi.h	Diff File
	mod - src/win32/lib/Makefile	Diff File
	mod - src/win32/plugins/filed/Makefile	Diff File
bareos2015: bareos-14.2 2242e405 2013-12-09 16:12 mvwieringen Ported: N/A Details Diff	Bring windows support up to Windows 2012 server. This commit bring the windows code to support all new features that have been added to windows up till windows 2012 server. We also reindented a lot of code along the way to be more readable and fixes most comments. The actual added features: - Volume Mountpoints support (VMP). Windows for some version now supports so called Volume MountPoint which allow you to mount a volume somewhere in the filesystem layout analog to what UNIX has been doing since it inception without assigning a drive letter. We need to make sure we also add the VMPs to the VSS snapshot list otherwise the backup will fail. - Use GetFileInformationByHandleEx to get real changetime. As it seems ctime in the stat struct on UNIX means for some time now the last change time of the inode and not the creation time. On Windows we need a special API to get the actual change time named GetFileInformationByHandleEx(). While implementing this new feature we reformated some of the layout and dropped some unused code. - Windows dedup support Windows 2012 has dedup support which needs handling. To get rehydrated data when reading the file you should open the file without the FILE_FLAG_OPEN_REPARSE_POINT flag set. If you want to read the deduped data effeciently you need to write new code which interfaces to a new API which is currently out of scope. The original file on the filesystem is replaced with a so called reparse point so if we open the reparse point instead of where it points to we get data which has a pointer to a map of all the data streams and chunks required to ârehydrateâ the file and not its actual data. We now changed the stat emulation functions to detect the special reparse point with type IO_REPARSE_TAG_DEDUP which indicates a deduped file. When that is set we set a special bit on the file so we know we need to open the file with the O_NOFOLLOW flag which will open the file without the FILE_FLAG_OPEN_REPARSE_POINT flag set. More info at: http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx - Store all file attributes Windows has gathered quite some special specific file flags over the years but not all are saved during backup so some are never restored by the restore process. The most important ones are the ARCHIVE flag which is "misused" by some programs for storing some special information. Others that are known not to be stored are the COMPRESSED flag which means that a restored file looses it and will be restored as an uncompressed file. As we are running out of opportunities to store the special flags as some mode in the st_mode member of the stat struct a redesign was needed to store the unique file flag somewhere else. We are reusing the st_rdev field. That is already used for reparse points in older versions but as those are never restored anyway we can reuse them without breaking backwards compatability. - Support for Windows EFS filesystems Windows has for quite some time now support for a so called EFS filesystem. This is an encrypted filesystem, to be able to backup the data and to restore it we need to use a special API. With this API you in essence export the data on backup and import it on restore. This way you never have access to the unencrypted data but just import and export the encrypted data. This is the cleanest way of handling encryption by just seeing the data as some opaque data and not try to do anything special with it. The problem with the restore API is however that it kind of works in a completely different way then a normal read/write API. You call a import function and that call the read callback multiple times until all data is imported/restored. As we don't want to rewrite the whole internal works of the filed for this we now create a seperate thread that runs during the whole restore of a filesystem and behaves as a consumer/producer problem (well known in the pthreads world). In this solution the producer is the filed that produces encrypted data out of the incoming data streams from the storage daemon and the seperate thread consumes that data by returning it as encrypted data in the callback function issues by the special EFS import API. There is some synchronization between the different threads so it knows when one file is done and it should end the import of that file. We also keep some of the buffers lingering around so we don't need to do malloc/free/malloc loops all the time, we cleanup the whole thread and the buffers used at the end of the whole restore process. Fixes 0000228: Some windows file attributes are not handled. Fixes 0000008: Handle Windows Encrypted Files using Win raw encryption Signed-off-by: Marco van Wieringen <marco.van.wieringen@bareos.com>	Affected Issues 0000008, 0000228
	mod - src/filed/backup.c	Diff File
	mod - src/filed/crypto.c	Diff File
	mod - src/filed/dir_cmd.c	Diff File
	mod - src/filed/fd_plugins.c	Diff File
	mod - src/filed/filed.c	Diff File
	mod - src/filed/fileset.c	Diff File
	mod - src/filed/protos.h	Diff File
	mod - src/filed/restore.c	Diff File
	mod - src/filed/verify.c	Diff File
	mod - src/findlib/attribs.c	Diff File
	mod - src/findlib/bfile.c	Diff File
	mod - src/findlib/bfile.h	Diff File
	mod - src/findlib/create_file.c	Diff File
	mod - src/findlib/find.c	Diff File
	mod - src/findlib/find.h	Diff File
	mod - src/findlib/find_one.c	Diff File
	mod - src/findlib/mkpath.c	Diff File
	mod - src/findlib/protos.h	Diff File
	mod - src/include/baconfig.h	Diff File
	mod - src/include/bareos.h	Diff File
	mod - src/include/bc_types.h	Diff File
	mod - src/include/jcr.h	Diff File
	mod - src/lib/Makefile.in	Diff File
	add - src/lib/cbuf.c	Diff File
	add - src/lib/cbuf.h	Diff File
	mod - src/plugins/filed/fd_common.h	Diff File
	mod - src/win32/Makefile	Diff File
	mod - src/win32/Makefile.inc	Diff File
	mod - src/win32/compat/compat.c	Diff File
	mod - src/win32/compat/include/compat.h	Diff File
	mod - src/win32/compat/include/mingwconfig.h	Diff File
	mod - src/win32/compat/print.c	Diff File
	mod - src/win32/compat/winapi.c	Diff File
	mod - src/win32/console/Makefile	Diff File
	mod - src/win32/filed/Makefile	Diff File
	mod - src/win32/filed/vss.c	Diff File
	mod - src/win32/filed/vss_generic.c	Diff File
	mod - src/win32/findlib/Makefile	Diff File
	add - src/win32/findlib/win32.c	Diff File
	mod - src/win32/include/vss.h	Diff File
	mod - src/win32/include/winapi.h	Diff File
	mod - src/win32/lib/Makefile	Diff File
	mod - src/win32/plugins/filed/Makefile	Diff File

Date Modified	Username	Field	Change
2013-09-27 07:54	mvwieringen	New Issue
2013-10-02 16:22	mvwieringen	Assigned To	=> mvwieringen
2013-10-02 16:22	mvwieringen	Status	new => acknowledged
2013-10-08 09:34	mvwieringen	Status	acknowledged => assigned
2013-12-09 15:17	mvwieringen	Changeset attached	=> bareos master fa9aabd5
2013-12-09 15:17	mvwieringen	Note Added: 0000752
2013-12-09 15:17	mvwieringen	Status	assigned => resolved
2013-12-09 15:17	mvwieringen	Resolution	open => fixed
2014-12-01 12:03	joergs	Relationship added	parent of 0000008
2014-12-03 08:46	joergs	Note Added: 0001091
2015-01-26 16:04	~~mvwieringen adm~~	Assigned To	mvwieringen =>
2015-01-26 16:04	~~mvwieringen adm~~	Status	resolved => closed
2015-03-25 16:51	mvwieringen	Changeset attached	=> bareos2015 bareos-14.2 2242e405
2015-03-25 16:51	mvwieringen	Note Added: 0001485
2015-03-25 16:51	mvwieringen	Status	closed => resolved
2015-03-25 19:19	joergs	Note Added: 0001632
2015-03-25 19:19	joergs	Status	resolved => closed

Reporting new Issues is disabled, please Report new Issues at https://github.com/bareos/bareos/issues

View Issue Details

Relationships

Activities

Related Changesets

Issue History