View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0001604 | bareos-core | file daemon | public | 2024-02-26 13:31 | 2024-04-30 15:10 |
Reporter | Int | Assigned To | bruno-at-bareos | ||
Priority | normal | Severity | crash | Reproducibility | random |
Status | closed | Resolution | fixed | ||
Platform | x86 | OS | Windows | OS Version | 2016 |
Product Version | 23.0.1 | ||||
Summary | 0001604: Windows file daemon crashes sometimes during backup job | ||||
Description | It happened 5 times in the last three weeks that the Windows file daemon crashed during a backup job. It happened on two different Windows systems: * System IGMS00: Windows Server 2016 x64 Build 14393.6709 - 2 crashes so far * System IGMS04: Windows 10 Professional 22H2 Build 19045.4046 - 3 crashes so far I attached the Windows event log entries. On system IGMS04 I enabled the trace debug output but since then the bug did reappear yet. | ||||
Tags | No tags attached. | ||||
IGMS00 windows fd crash.txt (1,978 bytes)
Protokollname: Application Quelle: Application Error Datum: 25.02.2024 23:00:44 Ereignis-ID: 1000 Aufgabenkategorie:(100) Ebene: Fehler Schlüsselwörter:Klassisch Benutzer: Nicht zutreffend Computer: IGMS00.vision.local Beschreibung: Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e Name des fehlerhaften Moduls: libwinpthread-1.dll, Version: 1.0.0.0, Zeitstempel: 0x00000000 Ausnahmecode: 0xc0000005 Fehleroffset: 0x0000000000002376 ID des fehlerhaften Prozesses: 0xa7c Startzeit der fehlerhaften Anwendung: 0x01da643f404bd7aa Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe Pfad des fehlerhaften Moduls: C:\Program Files\Bareos\libwinpthread-1.dll Berichtskennung: c350063a-f31a-4b9b-9a48-05f11f66c999 Vollständiger Name des fehlerhaften Pakets: Anwendungs-ID, die relativ zum fehlerhaften Paket ist: Ereignis-XML: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Application Error" /> <EventID Qualifiers="0">1000</EventID> <Level>2</Level> <Task>100</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2024-02-25T22:00:44.609142500Z" /> <EventRecordID>1135359</EventRecordID> <Channel>Application</Channel> <Computer>IGMS00.vision.local</Computer> <Security /> </System> <EventData> <Data>bareos-fd.exe</Data> <Data>0.0.0.0</Data> <Data>65a68d2e</Data> <Data>libwinpthread-1.dll</Data> <Data>1.0.0.0</Data> <Data>00000000</Data> <Data>c0000005</Data> <Data>0000000000002376</Data> <Data>a7c</Data> <Data>01da643f404bd7aa</Data> <Data>C:\Program Files\Bareos\bareos-fd.exe</Data> <Data>C:\Program Files\Bareos\libwinpthread-1.dll</Data> <Data>c350063a-f31a-4b9b-9a48-05f11f66c999</Data> <Data> </Data> <Data> </Data> </EventData> </Event> IGMS04 windows fd crash.txt (6,392 bytes)
Protokollname: Application Quelle: Application Error Datum: 08.02.2024 01:46:01 Ereignis-ID: 1000 Aufgabenkategorie:(100) Ebene: Fehler Schlüsselwörter:Klassisch Benutzer: Nicht zutreffend Computer: IGMS04.vision.local Beschreibung: Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e Name des fehlerhaften Moduls: ntdll.dll, Version: 10.0.19041.3636, Zeitstempel: 0x9b64aa6f Ausnahmecode: 0xc0000005 Fehleroffset: 0x00000000000634f6 ID des fehlerhaften Prozesses: 0x4d4 Startzeit der fehlerhaften Anwendung: 0x01da59c8e7ce77e3 Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe Pfad des fehlerhaften Moduls: C:\WINDOWS\SYSTEM32\ntdll.dll Berichtskennung: 8b02c30d-38b7-4bbc-b62f-f0d73f20ef93 Vollständiger Name des fehlerhaften Pakets: Anwendungs-ID, die relativ zum fehlerhaften Paket ist: Ereignis-XML: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Application Error" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>100</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2024-02-08T00:46:01.0288811Z" /> <EventRecordID>216990</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>IGMS04.vision.local</Computer> <Security /> </System> <EventData> <Data>bareos-fd.exe</Data> <Data>0.0.0.0</Data> <Data>65a68d2e</Data> <Data>ntdll.dll</Data> <Data>10.0.19041.3636</Data> <Data>9b64aa6f</Data> <Data>c0000005</Data> <Data>00000000000634f6</Data> <Data>4d4</Data> <Data>01da59c8e7ce77e3</Data> <Data>C:\Program Files\Bareos\bareos-fd.exe</Data> <Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data> <Data>8b02c30d-38b7-4bbc-b62f-f0d73f20ef93</Data> <Data> </Data> <Data> </Data> </EventData> </Event> Protokollname: Application Quelle: Application Error Datum: 09.02.2024 20:07:29 Ereignis-ID: 1000 Aufgabenkategorie:(100) Ebene: Fehler Schlüsselwörter:Klassisch Benutzer: Nicht zutreffend Computer: IGMS04.vision.local Beschreibung: Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e Name des fehlerhaften Moduls: ntdll.dll, Version: 10.0.19041.3636, Zeitstempel: 0x9b64aa6f Ausnahmecode: 0xc0000005 Fehleroffset: 0x00000000000634f6 ID des fehlerhaften Prozesses: 0xb90 Startzeit der fehlerhaften Anwendung: 0x01da5a75557cd61a Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe Pfad des fehlerhaften Moduls: C:\WINDOWS\SYSTEM32\ntdll.dll Berichtskennung: e33e8423-3d0d-42da-a9dc-7303dadb9aad Vollständiger Name des fehlerhaften Pakets: Anwendungs-ID, die relativ zum fehlerhaften Paket ist: Ereignis-XML: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Application Error" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>100</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2024-02-09T19:07:29.9972282Z" /> <EventRecordID>217879</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>IGMS04.vision.local</Computer> <Security /> </System> <EventData> <Data>bareos-fd.exe</Data> <Data>0.0.0.0</Data> <Data>65a68d2e</Data> <Data>ntdll.dll</Data> <Data>10.0.19041.3636</Data> <Data>9b64aa6f</Data> <Data>c0000005</Data> <Data>00000000000634f6</Data> <Data>b90</Data> <Data>01da5a75557cd61a</Data> <Data>C:\Program Files\Bareos\bareos-fd.exe</Data> <Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data> <Data>e33e8423-3d0d-42da-a9dc-7303dadb9aad</Data> <Data> </Data> <Data> </Data> </EventData> </Event> Protokollname: Application Quelle: Application Error Datum: 21.02.2024 01:05:43 Ereignis-ID: 1000 Aufgabenkategorie:(100) Ebene: Fehler Schlüsselwörter:Klassisch Benutzer: Nicht zutreffend Computer: IGMS04.vision.local Beschreibung: Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e Name des fehlerhaften Moduls: ntdll.dll, Version: 10.0.19041.3996, Zeitstempel: 0x39215800 Ausnahmecode: 0xc0000005 Fehleroffset: 0x00000000000634f6 ID des fehlerhaften Prozesses: 0xb60 Startzeit der fehlerhaften Anwendung: 0x01da63e42cdd98fc Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe Pfad des fehlerhaften Moduls: C:\WINDOWS\SYSTEM32\ntdll.dll Berichtskennung: 0e0f3a78-eeee-4754-a79e-fd2cc30e2a4b Vollständiger Name des fehlerhaften Pakets: Anwendungs-ID, die relativ zum fehlerhaften Paket ist: Ereignis-XML: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Application Error" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>100</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2024-02-21T00:05:43.5443596Z" /> <EventRecordID>222899</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>IGMS04.vision.local</Computer> <Security /> </System> <EventData> <Data>bareos-fd.exe</Data> <Data>0.0.0.0</Data> <Data>65a68d2e</Data> <Data>ntdll.dll</Data> <Data>10.0.19041.3996</Data> <Data>39215800</Data> <Data>c0000005</Data> <Data>00000000000634f6</Data> <Data>b60</Data> <Data>01da63e42cdd98fc</Data> <Data>C:\Program Files\Bareos\bareos-fd.exe</Data> <Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data> <Data>0e0f3a78-eeee-4754-a79e-fd2cc30e2a4b</Data> <Data> </Data> <Data> </Data> </EventData> </Event> [bareos@igms07 ~]$ bconsole <<< "setdebug level=100 trace=1 timestamp=1 client=igms04-fd" 2000 OK setdebug=100 trace=1 hangup=0 timestamp=1 tracefile=C:\WINDOWS\TEMP/igms04-fd.trace |
|
Made a typo - should be "On system IGMS04 I enabled the trace debug output but since then the bug did NOT reappear yet." | |
I managed to capture a trace of the Windows file daemon crash. But the trace file is quite large (121MB compressed) - do you have a upload where I can place it or should I provide a download link? 2024-03-08 IGMS04 windows fd crash.txt (2,060 bytes)
Protokollname: Application Quelle: Application Error Datum: 08.03.2024 20:23:40 Ereignis-ID: 1000 Aufgabenkategorie:(100) Ebene: Fehler Schlüsselwörter:Klassisch Benutzer: Nicht zutreffend Computer: IGMS04.vision.local Beschreibung: Name der fehlerhaften Anwendung: bareos-fd.exe, Version: 0.0.0.0, Zeitstempel: 0x65a68d2e Name des fehlerhaften Moduls: ntdll.dll, Version: 10.0.19041.3996, Zeitstempel: 0x39215800 Ausnahmecode: 0xc0000005 Fehleroffset: 0x00000000000634f6 ID des fehlerhaften Prozesses: 0x124 Startzeit der fehlerhaften Anwendung: 0x01da6e03659819c9 Pfad der fehlerhaften Anwendung: C:\Program Files\Bareos\bareos-fd.exe Pfad des fehlerhaften Moduls: C:\WINDOWS\SYSTEM32\ntdll.dll Berichtskennung: f337364e-42ae-42f6-a098-a427168d9808 Vollständiger Name des fehlerhaften Pakets: Anwendungs-ID, die relativ zum fehlerhaften Paket ist: Ereignis-XML: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Application Error" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>100</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2024-03-08T19:23:40.8211884Z" /> <EventRecordID>226137</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>IGMS04.vision.local</Computer> <Security /> </System> <EventData> <Data>bareos-fd.exe</Data> <Data>0.0.0.0</Data> <Data>65a68d2e</Data> <Data>ntdll.dll</Data> <Data>10.0.19041.3996</Data> <Data>39215800</Data> <Data>c0000005</Data> <Data>00000000000634f6</Data> <Data>124</Data> <Data>01da6e03659819c9</Data> <Data>C:\Program Files\Bareos\bareos-fd.exe</Data> <Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data> <Data>f337364e-42ae-42f6-a098-a427168d9808</Data> <Data> </Data> <Data> </Data> </EventData> </Event> |
|
Hello you should have received and invitation for a shared link to upload the trace. | |
Hi, unfortunately I did not receive an invitation to a shared link. | |
Thanks trace received, that maybe analyzed when free time will be allowed. | |
Not sure you've seen it, but 23.0.2 was released (so prev 23.0.3 for current) and that one contain fixes for certain Windows crashes Please try to upgrade and check if it is still failing. |
|
I have the same bug on several Windows clients and different Windows Server versions (2012r2, 2019). I have yesterday updated Bareos to 23. 0. 2 the error still exists. |
|
Dear M. Robert Frische, as you are using the subscription channel, and are a paying customer, with a valid support contract, I would like to track your specific case with the dedicated portal https://servicedesk.bareos.com if you didn't remember your password there, you can request a new one (which can and should be different from the one use on the customer/sale one) See you there. |
|
Same here - update to Bareos to 23.0.2 did not fix the issue. | |
For Intego, if you still have a bareos-fd 22 version you can try this specific build Our developer have prepared a special version that allow them to generate better trace and crash dump report on windows. We would really appreciate if you can deploy that bareos-fd on one of your crashing host. Specific Bareos binaries can be found here, they are bareos-fd version 22 like the fd you reported. https://download.bareos.org/experimental/ssura-bareos-22-fix-windows-crash/ To work correctly, you will have to install also the debughelp.dll which is part of the windows sdk. Explanation https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/ sdk download https://go.microsoft.com/fwlink/?linkid=2261842 The sdk being very large the only needed part is Debugging Tool for Windows Before installing you may want to check your system if debughelp.dll is not yet installed. Please run the job as previously, with debug level set to 200. We would like to have also the event log associated with the crash, and inside the event log you will see a path file (where the crash dump is saved), we would like to have this one too. Also note that the windows event might have bareos-fd as a source and not Bareos in case you try to filter. |
|
Hi, > Specific Bareos binaries can be found here, they are bareos-fd version 22 like the fd you reported. > https://download.bareos.org/experimental/ssura-bareos-22-fix-windows-crash/ I reported the issue with bareos-fd version 23.0.1 (and now 23.0.2) - not with version 22 |
|
Yeah but you're not entitled to support, so only community effort. Not much can be done then, | |
Well, it was you who just asked me to help you to debug this issue and I responded to your request. > Yeah but you're not entitled to support, so only community effort. Not much can be done then, Well, so far I supposed this was a give and take. I find the bugs and spend my time to report, debug and test for free. If you want to play the pay me game, I can do the same. You want me to run debug tests and work as a software tester for you - ask our sales department for a support contract. Standard rate for my time is 160€/hour |
|
We have also an experimental build (based on master) which do the same in the following repository https://download.bareos.org/experimental/ssura-master-add-windows-trace/ I will let other answer to your remark. |
|
All the tests and fixes are now handled into the following PR and its experimental packages https://github.com/bareos/bareos/pull/1793 |
|
Date Modified | Username | Field | Change |
---|---|---|---|
2024-02-26 13:31 | Int | New Issue | |
2024-02-26 13:31 | Int | File Added: IGMS00 windows fd crash.txt | |
2024-02-26 13:31 | Int | File Added: IGMS04 windows fd crash.txt | |
2024-02-26 13:32 | Int | Note Added: 0005809 | |
2024-03-11 09:08 | Int | Note Added: 0005828 | |
2024-03-11 09:08 | Int | File Added: 2024-03-08 IGMS04 windows fd crash.txt | |
2024-03-12 13:28 | bruno-at-bareos | Note Added: 0005830 | |
2024-03-12 14:10 | Int | Note Added: 0005831 | |
2024-03-13 09:52 | bruno-at-bareos | Note Added: 0005834 | |
2024-03-13 15:17 | bruno-at-bareos | Note Added: 0005836 | |
2024-03-27 09:30 | RobertF. | Note Added: 0005869 | |
2024-03-27 09:42 | bruno-at-bareos | Note Added: 0005870 | |
2024-04-09 08:21 | Int | Note Added: 0005890 | |
2024-04-11 10:55 | bruno-at-bareos | Note Added: 0005891 | |
2024-04-11 11:11 | Int | Note Added: 0005892 | |
2024-04-11 11:16 | bruno-at-bareos | Note Added: 0005893 | |
2024-04-11 11:25 | Int | Note Added: 0005894 | |
2024-04-11 12:12 | bruno-at-bareos | Note Added: 0005895 | |
2024-04-30 15:10 | bruno-at-bareos | Assigned To | => bruno-at-bareos |
2024-04-30 15:10 | bruno-at-bareos | Status | new => closed |
2024-04-30 15:10 | bruno-at-bareos | Resolution | open => fixed |
2024-04-30 15:10 | bruno-at-bareos | Note Added: 0005908 |