View Issue Details

IDProjectCategoryView StatusLast Update
0001604bareos-corefile daemonpublic2024-04-30 15:10
ReporterInt Assigned Tobruno-at-bareos  
PrioritynormalSeveritycrashReproducibilityrandom
Status closedResolutionfixed 
Platformx86OSWindowsOS Version2016
Product Version23.0.1 
Summary0001604: Windows file daemon crashes sometimes during backup job
DescriptionIt happened 5 times in the last three weeks that the Windows file daemon crashed during a backup job.

It happened on two different Windows systems:
* System IGMS00: Windows Server 2016 x64 Build 14393.6709 - 2 crashes so far
* System IGMS04: Windows 10 Professional 22H2 Build 19045.4046 - 3 crashes so far

I attached the Windows event log entries.

On system IGMS04 I enabled the trace debug output but since then the bug did reappear yet.
TagsNo tags attached.

Activities

Int

Int

2024-02-26 13:31

reporter  

IGMS00 windows fd crash.txt (1,978 bytes)   
Protokollname: Application
Quelle:        Application Error
Datum:         25.02.2024 23:00:44
Ereignis-ID:   1000
Aufgabenkategorie:(100)
Ebene:         Fehler
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      IGMS00.vision.local
Beschreibung:
Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e
Name des fehlerhaften Moduls: libwinpthread-1.dll, Version: 1.0.0.0, Zeitstempel: 0x00000000
Ausnahmecode: 0xc0000005
Fehleroffset: 0x0000000000002376
ID des fehlerhaften Prozesses: 0xa7c
Startzeit der fehlerhaften Anwendung: 0x01da643f404bd7aa
Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe
Pfad des fehlerhaften Moduls: C:\Program Files\Bareos\libwinpthread-1.dll
Berichtskennung: c350063a-f31a-4b9b-9a48-05f11f66c999
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2024-02-25T22:00:44.609142500Z" />
    <EventRecordID>1135359</EventRecordID>
    <Channel>Application</Channel>
    <Computer>IGMS00.vision.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>bareos-fd.exe</Data>
    <Data>0.0.0.0</Data>
    <Data>65a68d2e</Data>
    <Data>libwinpthread-1.dll</Data>
    <Data>1.0.0.0</Data>
    <Data>00000000</Data>
    <Data>c0000005</Data>
    <Data>0000000000002376</Data>
    <Data>a7c</Data>
    <Data>01da643f404bd7aa</Data>
    <Data>C:\Program Files\Bareos\bareos-fd.exe</Data>
    <Data>C:\Program Files\Bareos\libwinpthread-1.dll</Data>
    <Data>c350063a-f31a-4b9b-9a48-05f11f66c999</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>
IGMS00 windows fd crash.txt (1,978 bytes)   
IGMS04 windows fd crash.txt (6,392 bytes)   
Protokollname: Application
Quelle:        Application Error
Datum:         08.02.2024 01:46:01
Ereignis-ID:   1000
Aufgabenkategorie:(100)
Ebene:         Fehler
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      IGMS04.vision.local
Beschreibung:
Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e
Name des fehlerhaften Moduls: ntdll.dll, Version: 10.0.19041.3636, Zeitstempel: 0x9b64aa6f
Ausnahmecode: 0xc0000005
Fehleroffset: 0x00000000000634f6
ID des fehlerhaften Prozesses: 0x4d4
Startzeit der fehlerhaften Anwendung: 0x01da59c8e7ce77e3
Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe
Pfad des fehlerhaften Moduls: C:\WINDOWS\SYSTEM32\ntdll.dll
Berichtskennung: 8b02c30d-38b7-4bbc-b62f-f0d73f20ef93
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>100</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2024-02-08T00:46:01.0288811Z" />
    <EventRecordID>216990</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>IGMS04.vision.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>bareos-fd.exe</Data>
    <Data>0.0.0.0</Data>
    <Data>65a68d2e</Data>
    <Data>ntdll.dll</Data>
    <Data>10.0.19041.3636</Data>
    <Data>9b64aa6f</Data>
    <Data>c0000005</Data>
    <Data>00000000000634f6</Data>
    <Data>4d4</Data>
    <Data>01da59c8e7ce77e3</Data>
    <Data>C:\Program Files\Bareos\bareos-fd.exe</Data>
    <Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data>
    <Data>8b02c30d-38b7-4bbc-b62f-f0d73f20ef93</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>



Protokollname: Application
Quelle:        Application Error
Datum:         09.02.2024 20:07:29
Ereignis-ID:   1000
Aufgabenkategorie:(100)
Ebene:         Fehler
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      IGMS04.vision.local
Beschreibung:
Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e
Name des fehlerhaften Moduls: ntdll.dll, Version: 10.0.19041.3636, Zeitstempel: 0x9b64aa6f
Ausnahmecode: 0xc0000005
Fehleroffset: 0x00000000000634f6
ID des fehlerhaften Prozesses: 0xb90
Startzeit der fehlerhaften Anwendung: 0x01da5a75557cd61a
Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe
Pfad des fehlerhaften Moduls: C:\WINDOWS\SYSTEM32\ntdll.dll
Berichtskennung: e33e8423-3d0d-42da-a9dc-7303dadb9aad
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>100</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2024-02-09T19:07:29.9972282Z" />
    <EventRecordID>217879</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>IGMS04.vision.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>bareos-fd.exe</Data>
    <Data>0.0.0.0</Data>
    <Data>65a68d2e</Data>
    <Data>ntdll.dll</Data>
    <Data>10.0.19041.3636</Data>
    <Data>9b64aa6f</Data>
    <Data>c0000005</Data>
    <Data>00000000000634f6</Data>
    <Data>b90</Data>
    <Data>01da5a75557cd61a</Data>
    <Data>C:\Program Files\Bareos\bareos-fd.exe</Data>
    <Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data>
    <Data>e33e8423-3d0d-42da-a9dc-7303dadb9aad</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>


Protokollname: Application
Quelle:        Application Error
Datum:         21.02.2024 01:05:43
Ereignis-ID:   1000
Aufgabenkategorie:(100)
Ebene:         Fehler
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      IGMS04.vision.local
Beschreibung:
Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e
Name des fehlerhaften Moduls: ntdll.dll, Version: 10.0.19041.3996, Zeitstempel: 0x39215800
Ausnahmecode: 0xc0000005
Fehleroffset: 0x00000000000634f6
ID des fehlerhaften Prozesses: 0xb60
Startzeit der fehlerhaften Anwendung: 0x01da63e42cdd98fc
Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe
Pfad des fehlerhaften Moduls: C:\WINDOWS\SYSTEM32\ntdll.dll
Berichtskennung: 0e0f3a78-eeee-4754-a79e-fd2cc30e2a4b
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>100</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2024-02-21T00:05:43.5443596Z" />
    <EventRecordID>222899</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>IGMS04.vision.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>bareos-fd.exe</Data>
    <Data>0.0.0.0</Data>
    <Data>65a68d2e</Data>
    <Data>ntdll.dll</Data>
    <Data>10.0.19041.3996</Data>
    <Data>39215800</Data>
    <Data>c0000005</Data>
    <Data>00000000000634f6</Data>
    <Data>b60</Data>
    <Data>01da63e42cdd98fc</Data>
    <Data>C:\Program Files\Bareos\bareos-fd.exe</Data>
    <Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data>
    <Data>0e0f3a78-eeee-4754-a79e-fd2cc30e2a4b</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>



[bareos@igms07 ~]$ bconsole <<< "setdebug level=100 trace=1 timestamp=1 client=igms04-fd"


2000 OK setdebug=100 trace=1 hangup=0 timestamp=1 tracefile=C:\WINDOWS\TEMP/igms04-fd.trace

IGMS04 windows fd crash.txt (6,392 bytes)   
Int

Int

2024-02-26 13:32

reporter   ~0005809

Made a typo - should be "On system IGMS04 I enabled the trace debug output but since then the bug did NOT reappear yet."
Int

Int

2024-03-11 09:08

reporter   ~0005828

I managed to capture a trace of the Windows file daemon crash.
But the trace file is quite large (121MB compressed) - do you have a upload where I can place it or should I provide a download link?
2024-03-08 IGMS04 windows fd crash.txt (2,060 bytes)   
Protokollname: Application
Quelle:        Application Error
Datum:         08.03.2024 20:23:40
Ereignis-ID:   1000
Aufgabenkategorie:(100)
Ebene:         Fehler
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      IGMS04.vision.local
Beschreibung:
Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e
Name des fehlerhaften Moduls: ntdll.dll, Version: 10.0.19041.3996, Zeitstempel: 0x39215800
Ausnahmecode: 0xc0000005
Fehleroffset: 0x00000000000634f6
ID des fehlerhaften Prozesses: 0x124
Startzeit der fehlerhaften Anwendung: 0x01da6e03659819c9
Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe
Pfad des fehlerhaften Moduls: C:\WINDOWS\SYSTEM32\ntdll.dll
Berichtskennung: f337364e-42ae-42f6-a098-a427168d9808
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>100</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2024-03-08T19:23:40.8211884Z" />
    <EventRecordID>226137</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>IGMS04.vision.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>bareos-fd.exe</Data>
    <Data>0.0.0.0</Data>
    <Data>65a68d2e</Data>
    <Data>ntdll.dll</Data>
    <Data>10.0.19041.3996</Data>
    <Data>39215800</Data>
    <Data>c0000005</Data>
    <Data>00000000000634f6</Data>
    <Data>124</Data>
    <Data>01da6e03659819c9</Data>
    <Data>C:\Program Files\Bareos\bareos-fd.exe</Data>
    <Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data>
    <Data>f337364e-42ae-42f6-a098-a427168d9808</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>
bruno-at-bareos

bruno-at-bareos

2024-03-12 13:28

manager   ~0005830

Hello you should have received and invitation for a shared link to upload the trace.
Int

Int

2024-03-12 14:10

reporter   ~0005831

Hi, unfortunately I did not receive an invitation to a shared link.
bruno-at-bareos

bruno-at-bareos

2024-03-13 09:52

manager   ~0005834

Thanks trace received, that maybe analyzed when free time will be allowed.
bruno-at-bareos

bruno-at-bareos

2024-03-13 15:17

manager   ~0005836

Not sure you've seen it, but 23.0.2 was released (so prev 23.0.3 for current) and that one contain fixes for certain Windows crashes
Please try to upgrade and check if it is still failing.
RobertF.

RobertF.

2024-03-27 09:30

reporter   ~0005869

I have the same bug on several Windows clients and different Windows Server versions (2012r2, 2019).
I have yesterday updated Bareos to 23. 0. 2 the error still exists.
bruno-at-bareos

bruno-at-bareos

2024-03-27 09:42

manager   ~0005870

Dear M. Robert Frische, as you are using the subscription channel, and are a paying customer, with a valid support contract, I would like to track your specific case with the dedicated portal
https://servicedesk.bareos.com

if you didn't remember your password there, you can request a new one (which can and should be different from the one use on the customer/sale one)
See you there.
Int

Int

2024-04-09 08:21

reporter   ~0005890

Same here - update to Bareos to 23.0.2 did not fix the issue.
bruno-at-bareos

bruno-at-bareos

2024-04-11 10:55

manager   ~0005891

For Intego, if you still have a bareos-fd 22 version you can try this specific build

Our developer have prepared a special version that allow them to generate better trace and crash dump report on windows.
We would really appreciate if you can deploy that bareos-fd on one of your crashing host.

Specific Bareos binaries can be found here, they are bareos-fd version 22 like the fd you reported.
https://download.bareos.org/experimental/ssura-bareos-22-fix-windows-crash/

To work correctly, you will have to install also the debughelp.dll which is part of the windows sdk.
Explanation https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/
sdk download https://go.microsoft.com/fwlink/?linkid=2261842
The sdk being very large the only needed part is Debugging Tool for Windows
Before installing you may want to check your system if debughelp.dll is not yet installed.

Please run the job as previously, with debug level set to 200.

We would like to have also the event log associated with the crash, and inside the event log you will see a path file (where the crash dump is saved), we would like to have this one too.
Also note that the windows event might have bareos-fd as a source and not Bareos in case you try to filter.
Int

Int

2024-04-11 11:11

reporter   ~0005892

Hi,

> Specific Bareos binaries can be found here, they are bareos-fd version 22 like the fd you reported.
> https://download.bareos.org/experimental/ssura-bareos-22-fix-windows-crash/

I reported the issue with bareos-fd version 23.0.1 (and now 23.0.2) - not with version 22
bruno-at-bareos

bruno-at-bareos

2024-04-11 11:16

manager   ~0005893

Yeah but you're not entitled to support, so only community effort. Not much can be done then,
Int

Int

2024-04-11 11:25

reporter   ~0005894

Well, it was you who just asked me to help you to debug this issue and I responded to your request.

> Yeah but you're not entitled to support, so only community effort. Not much can be done then,

Well, so far I supposed this was a give and take. I find the bugs and spend my time to report, debug and test for free.
If you want to play the pay me game, I can do the same. You want me to run debug tests and work as a software tester for you - ask our sales department for a support contract.
Standard rate for my time is 160€/hour
bruno-at-bareos

bruno-at-bareos

2024-04-11 12:12

manager   ~0005895

We have also an experimental build (based on master) which do the same in the following repository https://download.bareos.org/experimental/ssura-master-add-windows-trace/

I will let other answer to your remark.
bruno-at-bareos

bruno-at-bareos

2024-04-30 15:10

manager   ~0005908

All the tests and fixes are now handled into the following PR and its experimental packages
https://github.com/bareos/bareos/pull/1793

Issue History

Date Modified Username Field Change
2024-02-26 13:31 Int New Issue
2024-02-26 13:31 Int File Added: IGMS00 windows fd crash.txt
2024-02-26 13:31 Int File Added: IGMS04 windows fd crash.txt
2024-02-26 13:32 Int Note Added: 0005809
2024-03-11 09:08 Int Note Added: 0005828
2024-03-11 09:08 Int File Added: 2024-03-08 IGMS04 windows fd crash.txt
2024-03-12 13:28 bruno-at-bareos Note Added: 0005830
2024-03-12 14:10 Int Note Added: 0005831
2024-03-13 09:52 bruno-at-bareos Note Added: 0005834
2024-03-13 15:17 bruno-at-bareos Note Added: 0005836
2024-03-27 09:30 RobertF. Note Added: 0005869
2024-03-27 09:42 bruno-at-bareos Note Added: 0005870
2024-04-09 08:21 Int Note Added: 0005890
2024-04-11 10:55 bruno-at-bareos Note Added: 0005891
2024-04-11 11:11 Int Note Added: 0005892
2024-04-11 11:16 bruno-at-bareos Note Added: 0005893
2024-04-11 11:25 Int Note Added: 0005894
2024-04-11 12:12 bruno-at-bareos Note Added: 0005895
2024-04-30 15:10 bruno-at-bareos Assigned To => bruno-at-bareos
2024-04-30 15:10 bruno-at-bareos Status new => closed
2024-04-30 15:10 bruno-at-bareos Resolution open => fixed
2024-04-30 15:10 bruno-at-bareos Note Added: 0005908