View Issue Details

IDProjectCategoryView StatusLast Update
0001358bareos-corefile daemonpublic2024-01-16 15:25
ReporterDark_Angel Assigned Tobruno-at-bareos  
PrioritynormalSeverityblockReproducibilityalways
Status closedResolutionfixed 
Platformx86OSWindowsOS Version10
Product Version19.2.9 
Summary0001358: Windows File Daemon VSS Snapshots are corrupt
DescriptionIn short: Windows backups that are taken from system volumes are corrupt.

Longer version:
I have setup with Bareos Director and 4 Windows 10 machines of various versions. All of the Windows Bareos-fd packages are 19.2.9. Director is 19.2.9.

Common issue between them, is that I cannot get consistent Windows backup from Windows Drive.

Issues are the following:
- On older Windows version (1709), VSS snapshot is created and backup is successful. However when snapshot is inspected during backup using Shadow Explorer, hive files, i.e. registry, are corrupt. Same is observed when backup is recovered - hives are corrupt.
- On newer versions of Windows (1809+), VSS snapshots are created, however during backup I get "Cannot open "C:/Windows/SoftwareDistribution/DataStore/DataStore.edb": ERR=The process cannot access the file because it is being used by another process." on multiple files regardless of settings on director, i.e. vssenable=yes, portable=no. As result backup is corrupt and cannot be used for recovery.

At the same time, if I use System Restore to create Volume Shadow Copy and analyze it via Shadow Explorer - hives are consistent on all Windows versions.

I haven't updated to version 20.x yet, since I haven't seen anything related to changes of VSS mechanism.
Steps To ReproduceAssuming you have Director (Linux, Centos 6, v. 19.2.9) and File daemon (Windows x64 10, v19.2.9) installed:

1. Put the client configuration on Director like this:
Client {
    Name = Client
    Address = 10.0.4.21
    Password = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    # uncomment the following if using bacula
    # Catalog = "MyCatalog"
    TLS Enable = yes
    TLS Require = yes
    Maximum Concurrent Jobs = 2
    Port = 9104
    Auto Prune = yes
# File Retention =
# Enabled = no

}

FileSet {
    Name = Client-file-set

    Enable VSS = yes

    Include {

        # Exclude first
        Options {
          WildFile = "[A-Z]:/pagefile.sys"
          WildFile = "[A-Z]:/hiberfil.sys"
          WildFile = "[A-Z]:/swapfile.sys"
          WildDir = "[A-Z]:/RECYCLER"
          WildDir = "[A-Z]:/$RECYCLE.BIN"
          WildDir = "[A-Z]:/System Volume Information"

          WildFile = "[A-Z]:/VSC_Test/pagefile.sys"
          WildFile = "[A-Z]:/VSC_Test/hiberfil.sys"
          WildFile = "[A-Z]:/VSC_Test/swapfile.sys"
          WildDir = "[A-Z]:/VSC_Test/RECYCLER"
          WildDir = "[A-Z]:/VSC_Test/$RECYCLE.BIN"
          WildDir = "[A-Z]:/VSC_Test/System Volume Information"
          Exclude = yes
        }

        Options {
          RegExFile = ".*"
          Signature = MD5
          Compression = GZIP
          portable = no
          onefs = no
        }

        File = "C:/"
        File = "E:/"
        File = "F:/"

    }
}

Job {
    Name = Client-full-backup
    Type = Backup
    Level = Incremental
    Client = Client
    FileSet = Client-file-set
    Schedule = "WeeklyCycle"
    Storage = File
    Messages = Standard
    Pool = Full
    Full Backup Pool = Full
    Differential Backup Pool = Differential
    Incremental Backup Pool = Incremental
    Reschedule On Error = yes
    Reschedule Times = 120
    Reschedule Interval = 600
    # Every 40 days we force full backups. Otherwise they will be executed every 1st Saturday of the every month
    Max Full Interval = 5184000

    # Do not allow duplicates
    Allow Duplicate Jobs = no
    Cancel Lower Level Duplicates = yes

    Client Run Before Job = "C:/create_lock_file.bat"

    # There is a possibility to wake up, but if computer is asleep, then do not start it up - will backup later on
    Client Run After Job = "C:/remove_lock_file.bat"
    Write Bootstrap = "/mnt/safecopy/bareos/bootstraps/%c.bsr"
}

Job {
    Name = Client-full-restore
    Type = Restore
    Client = Client
    FileSet = Client-file-set
    Storage = File
    Pool = Full
    Full Backup Pool = Full
    Differential Backup Pool = Differential
    Incremental Backup Pool = Incremental
    Messages = Standard
    Where = "/"
}


2. Initiate backup from Director
3. Wait for VSS creation to complete on the File Daemon
4. While backup is running, Open Shadow Explorer
5. Navigate to C:\Windows\System32\config\SYSTEM
6. Copy file from Shadow Explorer on other drive than being backed up
7. Open regedit
8. Navigate to key HKEY_LOCAL_MACHINE and put the cursor there
9. Open "File -> Load Hive" and point to file copied from Shadow Copy.

Expected result:
Hive is imported correctly

Actual result:
Hive import returns error with the message that file is corrupt.

10. Wait for backup to finish
On newer (1809+) versions of Windows 10:

Expected result:
No errors related to file access during backup since VSS is used.

Actual result:
Multiple errors on files which are currently open. Example:
Cannot open "C:/Windows/SoftwareDistribution/DataStore/DataStore.edb": ERR=The process cannot access the file because it is being used by another process.
Additional InformationIn all cases VSS Writers are in stable state, i.e.:

27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "Task Scheduler Writer", State: 0x1 (VSS_WS_STABLE)
27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "VSS Metadata Store Writer", State: 0x1 (VSS_WS_STABLE)
27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "Performance Counters Writer", State: 0x1 (VSS_WS_STABLE)
27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "System Writer", State: 0x1 (VSS_WS_STABLE)
27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "ASR Writer", State: 0x1 (VSS_WS_STABLE)
27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "WMI Writer", State: 0x1 (VSS_WS_STABLE)
27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "MSSearch Service Writer", State: 0x1 (VSS_WS_STABLE)
27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "Shadow Copy Optimization Writer", State: 0x1 (VSS_WS_STABLE)
27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "Registry Writer", State: 0x1 (VSS_WS_STABLE)
27-May 23:29 Client JobId 1509: VSS Writer (BackupComplete): "COM+ REGDB Writer", State: 0x1 (VSS_WS_STABLE)
TagsNo tags attached.

Activities

Dark_Angel

Dark_Angel

2021-09-17 21:31

reporter   ~0004268

Updated to latest 20.0.1 - issue persists.
bruno-at-bareos

bruno-at-bareos

2024-01-16 15:25

manager   ~0005689

The initial root cause describe here should have already been fixed in 23 release.
The code change https://github.com/bareos/bareos/pull/1452

Issue History

Date Modified Username Field Change
2021-05-28 00:04 Dark_Angel New Issue
2021-09-17 21:31 Dark_Angel Note Added: 0004268
2024-01-16 15:25 bruno-at-bareos Assigned To => bruno-at-bareos
2024-01-16 15:25 bruno-at-bareos Status new => closed
2024-01-16 15:25 bruno-at-bareos Resolution open => fixed
2024-01-16 15:25 bruno-at-bareos Note Added: 0005689