View Issue Details

IDProjectCategoryView StatusLast Update
0000197bareos-core[All Projects] file daemonpublic2013-07-12 17:16
ReporterpstorzAssigned To 
PrioritynormalSeverityfeatureReproducibilityhave not tried
Status acknowledgedResolutionopen 
Product Version 
Fixed in Version 
Summary0000197: Implement a file change monitor to create a filelist for incremental backups during runtime
DescriptionBy using inotify or similar

linux:
(http://en.wikipedia.org/wiki/Inotify)

windows:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa364417(v=vs.85).aspx

we could implement a function in the fd that can create the list of altered files during the day.

When the backup then takes place, the altered files can be backed up and the filelist then can be cleared.

This would speed up the backup of a big amount of files very much as not the whole filesystems has to be scanned for changes.
TagsNo tags attached.
bareos-master: impact
bareos-master: action
bareos-19.2: impact
bareos-19.2: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action

Activities

mvwieringen

mvwieringen

2013-06-27 11:24

developer   ~0000483

I wonder if this is a smart and scalable option. Inotify (and others on
other platforms Solaris also has something which is more or less the same
as inotify on Linux) have some severe limitations which are:

Inotify monitoring of directories is not recursive: to monitor subdirectories under a directory, additional watches must be created. This can take a significant amount time for large directory trees.

The inotify API provides no information about the user or process that triggered the inotify event. In particular, there is no easy way for a process that is monitoring events via inotify to distinguish events that it triggers itself from those that are triggered by other processes.

Note that the event queue can overflow. In this case, events are lost. Robust applications should handle the possibility of lost events gracefully.

The inotify API identifies affected files by filename. However, by the time an application processes an inotify event, the filename may already have been deleted or renamed.

If monitoring an entire directory subtree, and a new subdirectory is created in that tree, be aware that by the time you create a watch for the new subdirectory, new files may already have been created in the subdirectory. Therefore, you may want to scan the contents of the subdirectory immediately after adding the watch.

So this means that to be suitable you need to put inotify watches on the whole
backup tree (at least for all directories) which means a serious amount of memory.
maik

maik

2013-07-05 16:24

administrator   ~0000498

Philipp: make some tests, use recursive scripting to scan directory trees. Marco: implement as plugin.
AndiH

AndiH

2013-07-12 15:15

reporter   ~0000523

At least on Windows the information we are looking for should be already available in the NTFS change journal:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363798%28v=vs.85%29.aspx

Limitations:
- Only on NTFS
- If to many changes the journal will roll-over
- All changes will be recorded
mvwieringen

mvwieringen

2013-07-12 17:16

developer   ~0000530

For Solaris there is ZFS diff which also allows you to see what changed since
a certain snapshot. Given the wide differences between this interfaces this
seems to be a good candidate for a FD plugin as there are already hooks for
saying what to backup (e.g. this is used by the Bacula Systems NetAPP accelerator)

Issue History

Date Modified Username Field Change
2013-06-27 10:05 pstorz New Issue
2013-06-27 11:24 mvwieringen Note Added: 0000483
2013-07-05 16:24 maik Note Added: 0000498
2013-07-05 16:24 maik Status new => acknowledged
2013-07-12 15:15 AndiH Note Added: 0000523
2013-07-12 17:16 mvwieringen Note Added: 0000530