View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0000197||bareos-core||[All Projects] file daemon||public||2013-06-27 10:05||2013-07-12 17:16|
|Priority||normal||Severity||feature||Reproducibility||have not tried|
|Fixed in Version|
|Summary||0000197: Implement a file change monitor to create a filelist for incremental backups during runtime|
|Description||By using inotify or similar |
we could implement a function in the fd that can create the list of altered files during the day.
When the backup then takes place, the altered files can be backed up and the filelist then can be cleared.
This would speed up the backup of a big amount of files very much as not the whole filesystems has to be scanned for changes.
|Tags||No tags attached.|
I wonder if this is a smart and scalable option. Inotify (and others on
other platforms Solaris also has something which is more or less the same
as inotify on Linux) have some severe limitations which are:
Inotify monitoring of directories is not recursive: to monitor subdirectories under a directory, additional watches must be created. This can take a significant amount time for large directory trees.
The inotify API provides no information about the user or process that triggered the inotify event. In particular, there is no easy way for a process that is monitoring events via inotify to distinguish events that it triggers itself from those that are triggered by other processes.
Note that the event queue can overflow. In this case, events are lost. Robust applications should handle the possibility of lost events gracefully.
The inotify API identifies affected files by filename. However, by the time an application processes an inotify event, the filename may already have been deleted or renamed.
If monitoring an entire directory subtree, and a new subdirectory is created in that tree, be aware that by the time you create a watch for the new subdirectory, new files may already have been created in the subdirectory. Therefore, you may want to scan the contents of the subdirectory immediately after adding the watch.
So this means that to be suitable you need to put inotify watches on the whole
backup tree (at least for all directories) which means a serious amount of memory.
|Philipp: make some tests, use recursive scripting to scan directory trees. Marco: implement as plugin.|
At least on Windows the information we are looking for should be already available in the NTFS change journal:
- Only on NTFS
- If to many changes the journal will roll-over
- All changes will be recorded
For Solaris there is ZFS diff which also allows you to see what changed since
a certain snapshot. Given the wide differences between this interfaces this
seems to be a good candidate for a FD plugin as there are already hooks for
saying what to backup (e.g. this is used by the Bacula Systems NetAPP accelerator)
|2013-06-27 10:05||pstorz||New Issue|
|2013-06-27 11:24||mvwieringen||Note Added: 0000483|
|2013-07-05 16:24||maik||Note Added: 0000498|
|2013-07-05 16:24||maik||Status||new => acknowledged|
|2013-07-12 15:15||AndiH||Note Added: 0000523|
|2013-07-12 17:16||mvwieringen||Note Added: 0000530|