0001554: bareos-sd crashes when bareos-fd can't be reached - Bareos Bug Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0001554	bareos-core	storage daemon	public	2023-09-21 15:38	2024-01-29 09:42

Reporter	oyxnaut	Assigned To	bruno-at-bareos
Priority	normal	Severity	crash	Reproducibility	always
Status	closed	Resolution	fixed
Platform	Linux	OS	Ubuntu	OS Version	22.04.3
Product Version	22.1.0

Summary	0001554: bareos-sd crashes when bareos-fd can't be reached
Description	When bareos-sd tries to connect to a bareos-fd client, and a firewall on the network path drops (didn't test with reject) packages from the storage-daemon, the bareos-sd process crashes. This happens with `22.1.1~pre88.04d51ceb5-87` from https://download.bareos.org/current/xUbuntu_22.04/\1. Since the product version dropdown doesn't list 22.1.1, I've selected 22.1.0. Sorry if this bug tracker is the wrong place to report this issue.
Steps To Reproduce	Configure a firewall that drop connections from bareos-sd to bareos-fd, and run a backup job.
Additional Information	The crash happens in __strnlen_sse2 () at ../sysdeps/x86_64/multiarch/../multiarch/strlen-vec.S:123 ``` (gdb) disas Dump of assembler code for function __strnlen_sse2: <snip> 0x00007fc4afacabb0 <+64>: and rax,0xfffffffffffffff0 => 0x00007fc4afacabb4 <+68>: pcmpeqb xmm0,XMMWORD PTR [rax] </snip> (gdb) i r rax 0x30 48 ``` You can find attached a backtrace for the crashed thread.
Tags	No tags attached.

oyxnaut 2023-09-21 15:38 reporter	bareos-sd-crash.log (6,258 bytes) #0 __strnlen_sse2 () at ../sysdeps/x86_64/multiarch/../multiarch/strlen-vec.S:123 #1 0x00007fc4afeda122 in fmtstr (max=4095, min=0, flags=0, value=0x3f <error: Cannot access memory at address 0x3f>, maxlen=4095, currlen=33, buffer=0x558f97a674b8 "3991 Bad passive client command: sk-job.2023-09-20_17.13.11_17 type=3 level=1695224374 bareos-sd JobId 185921: Fatal error: Failed to connect to File daemon: myclient.contoso.com:9102\n") at ../../../../../../../bareos_PR-1545/TARFILES/bareos-22.1.1~pre86.ca2488010/cmake-build/core/src/lib/../../../../core/src/lib/bsnprintf.cc:463 #2 Bvsnprintf ( buffer=0x558f97a674b8 "3991 Bad passive client command: sk-job.2023-09-20_17.13.11_17 type=3 level=1695224374 bareos-sd JobId 185921: Fatal error: Failed to connect to File daemon: myclient.contoso.com:9102\n", maxlen=maxlen@entry=4095, format=<optimized out>, format@entry=0x558f9690b440 <_ZN13storagedaemonL6BADcmdE.lto_priv.0> "3991 Bad %s command: %s\n", args=args@entry=0x7fc4acc59950) at ../../../../../../../bareos_PR-1545/TARFILES/bareos-22.1.1~pre86.ca2488010/cmake-build/core/src/lib/../../../../core/src/lib/bsnprintf.cc:370 #3 0x00007fc4afedda5c in BareosSocket::vfsend (this=0x558f97a3e240, fmt=0x558f9690b440 <_ZN13storagedaemonL6BADcmdE.lto_priv.0> "3991 Bad %s command: %s\n", ap=ap@entry=0x7fc4acc599b0) at ../../../../../../../bareos_PR-1545/TARFILES/bareos-22.1.1~pre86.ca2488010/cmake-build/core/src/lib/../../../../core/src/lib/bsock.cc:303 #4 0x00007fc4afeddb38 in BareosSocket::fsend (this=this@entry=0x558f97a3e240, fmt=<optimized out>) at ../../../../../../../bareos_PR-1545/TARFILES/bareos-22.1.1~pre86.ca2488010/cmake-build/core/src/lib/../../../../core/src/lib/bsock.cc:283 #5 0x0000558f968f213f in storagedaemon::PassiveCmd (jcr=0x7fc498005470) at ../../../../../../../bareos_PR-1545/TARFILES/bareos-22.1.1~pre86.ca2488010/cmake-build/core/src/stored/../../../../core/src/stored/dir_cmd.cc:1810 #6 0x0000558f968e8870 in storagedaemon::HandleDirectorConnection (dir=0x558f97a3e240) at ../../../../../../../bareos_PR-1545/TARFILES/bareos-22.1.1~pre86.ca2488010/cmake-build/core/src/stored/../../../../core/src/stored/dir_cmd.cc:312 #7 storagedaemon::HandleConnectionRequest (config=<optimized out>, arg=0x558f97a3e240) at ../../../../../../../bareos_PR-1545/TARFILES/bareos-22.1.1~pre86.ca2488010/cmake-build/core/src/stored/../../../../core/src/stored/socket_server.cc:116 #8 0x00007fc4aff3fbd2 in std::function<void* (ConfigurationParser, void)>::operator()(ConfigurationParser, void) const ( __args#1=0x558f97a3e240, __args#0=<optimized out>, this=0x558f97a64068) at /usr/include/c++/11/bits/std_function.h:590 #9 WorkerThread (l=..., ThreadInvokedHandler=..., config=0x558f97a43190, data=0x558f97a3e240, run_condition=...) at ../../../../../../../bareos_PR-1545/TARFILES/bareos-22.1.1~pre86.ca2488010/cmake-build/core/src/lib/../../../../core/src/lib/thread_list.cc:201 #10 0x00007fc4aff3b555 in std::__invoke_impl<void, void ()(std::shared_ptr<ThreadListContainer>, std::function<void (ConfigurationParser, void)> const&, ConfigurationParser, void, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser, void)>, ConfigurationParser, void, std::shared_ptr<IsRunningCondition> >(std::__invoke_other, void (&&)(std::shared_ptr<ThreadListContainer>, std::function<void (ConfigurationParser, void)> const&, ConfigurationParser, void, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>&&, std::function<void* (ConfigurationParser, void)>&&, ConfigurationParser&&, void&&, std::shared_ptr<IsRunningCondition>&&) (__f=<optimized out>, __f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:61 #11 std::__invoke<void ()(std::shared_ptr<ThreadListContainer>, std::function<void (ConfigurationParser, void)> const&, ConfigurationParser, void, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser, void)>, ConfigurationParser, void, std::shared_ptr<IsRunningCondition> >(void (&&)(std::shared_ptr<ThreadListContainer>, std::function<void (ConfigurationParser, void)> const&, ConfigurationParser, void, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>&&, std::function<void* (ConfigurationParser, void)>&&, ConfigurationParser&&, void&&, std::shared_ptr<IsRunningCondition>&&) (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96 #12 std::thread::_Invoker<std::tuple<void ()(std::shared_ptr<ThreadListContainer>, std::function<void (ConfigurationParser, void)> const&, ConfigurationParser, void, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser, void)>, ConfigurationParser, void, std::shared_ptr<IsRunningCondition> > >::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>(std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>) (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:253 #13 std::thread::_Invoker<std::tuple<void ()(std::shared_ptr<ThreadListContainer>, std::function<void (ConfigurationParser, void)> const&, ConfigurationParser, void, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser, void)>, ConfigurationParser, void, std::shared_ptr<IsRunningCondition> > >::operator()() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:260 #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void ()(std::shared_ptr<ThreadListContainer>, std::function<void (ConfigurationParser, void)> const&, ConfigurationParser, void, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser, void)>, ConfigurationParser, void, std::shared_ptr<IsRunningCondition> > > >::_M_run() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211 #15 0x00007fc4afd36253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #16 0x00007fc4afaa4b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #17 0x00007fc4afb36a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 quit bareos-sd-crash.log (6,258 bytes)

bruno-at-bareos 2023-09-21 16:51 manager ~0005444	May I ask you to check twice the version used, in the trace there's mention of which look like wired. ../../../../../../../bareos_PR-1545/TARFILES/bareos-22.1.1~pre86.ca2488010 The bugtracker is the right place if you got a backtrace ;-) Btw: You certainly want to use the passive client mode which will soon get better advertised in our documentation. https://docs.bareos.org/TasksAndConcepts/NetworkSetup.html#passive-clients

bruno-at-bareos 2023-09-21 16:53 manager ~0005445	the weird mention is the PR-1545 and the bareos-22.1.1~pre86.ca2488010 when your package mention 22.1.1~pre88.04d51ceb5-87

oyxnaut 2023-09-22 13:14 reporter ~0005446	At first I thought the version mismatch might be due to unattended updates, so I restarted the storage daemon, and tried again. Unfortunately the same happened. I won't be available to provide more information after today 16:00 GMT+2 for the next two weeks. If you need more details to assist your debugging, kindly let me know before then. P.S. I noticed I made a copy & paste mistake with the repo url. The trailing \1 should not be there.

bruno-at-bareos 2023-10-11 16:15 manager ~0005458	Could you rerun this with debug level set to 150 on both fd and sd and attach back the trace log, and the whole configuration. We need to reproduce this.

oyxnaut 2023-11-03 14:42 reporter ~0005495	Sorry for the late reply, I'm rather swamped at work right now, and I don't know when I'll be able to find a minimal config that causes the segfault. ( (We're currently fine, because we fixed our firewall rules -> no more connection timeouts -> no crash.)

bruno-at-bareos 2024-01-29 09:42 manager ~0005723	Fix in progress in https://github.com/bareos/bareos/pull/1688

Date Modified	Username	Field	Change
2023-09-21 15:38	oyxnaut	New Issue
2023-09-21 15:38	oyxnaut	File Added: bareos-sd-crash.log
2023-09-21 16:51	bruno-at-bareos	Note Added: 0005444
2023-09-21 16:53	bruno-at-bareos	Note Added: 0005445
2023-09-22 13:14	oyxnaut	Note Added: 0005446
2023-10-11 16:15	bruno-at-bareos	Assigned To	=> bruno-at-bareos
2023-10-11 16:15	bruno-at-bareos	Status	new => feedback
2023-10-11 16:15	bruno-at-bareos	Note Added: 0005458
2023-11-03 14:42	oyxnaut	Note Added: 0005495
2023-11-03 14:42	oyxnaut	Status	feedback => assigned
2024-01-29 09:42	bruno-at-bareos	Status	assigned => closed
2024-01-29 09:42	bruno-at-bareos	Resolution	open => fixed
2024-01-29 09:42	bruno-at-bareos	Note Added: 0005723