View Issue Details

IDProjectCategoryView StatusLast Update
0000996bareos-coredirectorpublic2023-07-17 16:35
Reportergnif Assigned Tobruno-at-bareos  
PriorityhighSeveritycrashReproducibilityalways
Status closedResolutionfixed 
PlatformLinuxOSDebianOS Version9
Product Version18.2.3 
Summary0000996: Reloading director configuration crashes with a double free
DescriptionWhen reloading the director's configuration it crashes with a double free, back trace provided.
Steps To Reproducesimply execute: systemctl reload bareos-dir
Additional InformationCreated /tmp//bareos-dir.core.2957 for doing postmortem debugging
[New LWP 2957]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/bareos-dir'.
#0 0x00007f6ae7f8daea in __waitpid (pid=2958, stat_loc=0x7ffff1ce980c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29 ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
$1 = 1701994850
$2 = -851886296
$3 = -851886232
$4 = 0
$5 = -393116296
$6 = -393116222
$7 = -393116196
$8 = -393116184
$9 = 1701994850
$10 = -393116272
Environment variable "TestName" not defined.
#0 0x00007f6ae7f8daea in __waitpid (pid=2958, stat_loc=0x7ffff1ce980c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
0000001 0x00007f6ae89009e8 in SignalHandler () from /usr/lib/bareos/libbareos.so.18
0000002 <signal handler called>
0000003 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
0000004 0x00007f6ae670f42a in __GI_abort () at abort.c:89
0000005 0x00007f6ae674bc00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f6ae6840d98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
0000006 0x00007f6ae6751fc6 in malloc_printerr (action=3, str=0x7f6ae6840e60 "double free or corruption (out)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
0000007 0x00007f6ae675280e in _int_free (av=0x7f6ae6a74b00 <main_arena>, p=0x5641cd3ac5b0, have_lock=0) at malloc.c:3905
0000008 0x00005641cc62b4ea in FreeResource(CommonResourceHeader*, int) ()
0000009 0x00007f6ae96450bf in ConfigurationParser::~ConfigurationParser() () from /usr/lib/bareos/libbareoscfg.so.18
0000010 0x00005641cc633bd1 in TerminateDird(int) ()
0000011 0x00005641cc63387e in main ()

Thread 1 (Thread 0x7f6ae9a66b80 (LWP 2957)):
#0 0x00007f6ae7f8daea in __waitpid (pid=2958, stat_loc=0x7ffff1ce980c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
0000001 0x00007f6ae89009e8 in SignalHandler () from /usr/lib/bareos/libbareos.so.18
0000002 <signal handler called>
0000003 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
0000004 0x00007f6ae670f42a in __GI_abort () at abort.c:89
0000005 0x00007f6ae674bc00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f6ae6840d98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
0000006 0x00007f6ae6751fc6 in malloc_printerr (action=3, str=0x7f6ae6840e60 "double free or corruption (out)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
0000007 0x00007f6ae675280e in _int_free (av=0x7f6ae6a74b00 <main_arena>, p=0x5641cd3ac5b0, have_lock=0) at malloc.c:3905
0000008 0x00005641cc62b4ea in FreeResource(CommonResourceHeader*, int) ()
0000009 0x00007f6ae96450bf in ConfigurationParser::~ConfigurationParser() () from /usr/lib/bareos/libbareoscfg.so.18
0000010 0x00005641cc633bd1 in TerminateDird(int) ()
0000011 0x00005641cc63387e in main ()
#0 0x00007f6ae7f8daea in __waitpid (pid=2958, stat_loc=0x7ffff1ce980c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29 in ../sysdeps/unix/sysv/linux/waitpid.c
resultvar = 18446744073709551104
0000001 0x00007f6ae89009e8 in SignalHandler () from /usr/lib/bareos/libbareos.so.18
No symbol table info available.
0000002 <signal handler called>
No locals.
0000003 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
set = {__val = {0, 3615882721364815733, 3919881043631615540, 3472384402208154977, 4135818719080688944, 8223625903106963297, 3472328295963438455, 4192904167887482928, 7365366848950382640, 3486176864243181158, 7378697628691542064, 2319406792463180593, 3472328228581767026, 3472310704041635888, 2314885599538982970, 2314885530818453536}}
pid = <optimized out>
tid = <optimized out>
0000004 0x00007f6ae670f42a in __GI_abort () at abort.c:89
89 abort.c: No such file or directory.
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x2020202020373133, sa_sigaction = 0x2020202020373133}, sa_mask = {__val = {2314885530818453536, 2314885530818453536, 3907004821653777455, 8461816663211521631, 7236210882687479160, 8030813362246791725, 4051380807995373322, 7365405400577882211, 3474917680625312054, 2337418197644357680, 3472328296227680304, 3467824696768081952, 4135818719080679968, 3975887029563646776, 3762301437856527974, 140737250238960}}, sa_flags = 93, sa_restorer = 0x7ffff1cea1f0}
sigs = {__val = {32, 0 <repeats 15 times>}}
0000005 0x00007f6ae674bc00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f6ae6840d98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
175 ../sysdeps/posix/libc_fatal.c: No such file or directory.
ap = {{gp_offset = 40, fp_offset = 0, overflow_arg_area = 0x7ffff1cea200, reg_save_area = 0x7ffff1cea190}}
fd = 2
on_2 = <optimized out>
list = <optimized out>
nlist = <optimized out>
cp = <optimized out>
written = <optimized out>
0000006 0x00007f6ae6751fc6 in malloc_printerr (action=3, str=0x7f6ae6840e60 "double free or corruption (out)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
5049 malloc.c: No such file or directory.
buf = "00005641cd3ac5c0"
cp = <optimized out>
ar_ptr = <optimized out>
ptr = <optimized out>
str = 0x7f6ae6840e60 "double free or corruption (out)"
action = 3
0000007 0x00007f6ae675280e in _int_free (av=0x7f6ae6a74b00 <main_arena>, p=0x5641cd3ac5b0, have_lock=0) at malloc.c:3905
3905 in malloc.c
size = <optimized out>
fb = <optimized out>
nextchunk = <optimized out>
nextsize = <optimized out>
nextinuse = <optimized out>
prevsize = <optimized out>
bck = <optimized out>
fwd = <optimized out>
errstr = <optimized out>
locked = <optimized out>
__func__ = "_int_free"
TagsNo tags attached.

Activities

gnif

gnif

2018-08-06 08:52

reporter   ~0003087

It should be noted that this is occurring with the latest nightly, 18.2.3-1388.1
gnif

gnif

2018-08-06 09:12

reporter   ~0003088

Just realized I attached the wrong back trace. Even still it looks like the same problem. Here is the correct back trace.

Created //var/lib/bareos/bareos-dir.core.2779 for doing postmortem debugging
[New LWP 2779]
[New LWP 2781]
[New LWP 2782]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/bareos-dir'.
#0 0x00007f7760d1cb3a in __waitpid (pid=2915, stat_loc=0x7ffd20f76dcc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29 ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
[Current thread is 1 (Thread 0x7f77627f5b80 (LWP 2779))]
$1 = 1701994850
$2 = 1300803368
$3 = 1300803432
$4 = 1301118776
$5 = 1634366840
$6 = 1634366914
$7 = 1634366940
$8 = 1634366952
$9 = 1701994850
$10 = 1634366864
Environment variable "TestName" not defined.
#0 0x00007f7760d1cb3a in __waitpid (pid=2915, stat_loc=0x7ffd20f76dcc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
0000001 0x00007f776168f9e8 in SignalHandler () from /usr/lib/bareos/libbareos.so.18
0000002 <signal handler called>
0000003 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
0000004 0x00007f775f49e42a in __GI_abort () at abort.c:89
0000005 0x00007f775f4dac00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f775f5cfd98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
0000006 0x00007f775f4e0fc6 in malloc_printerr (action=3, str=0x7f775f5cfe60 "double free or corruption (out)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
0000007 0x00007f775f4e180e in _int_free (av=0x7f775f803b00 <main_arena>, p=0x55634d89f970, have_lock=0) at malloc.c:3905
0000008 0x000055634c6fe4ea in FreeResource(CommonResourceHeader*, int) ()
0000009 0x000055634c705d8b in FreeSavedResources(resource_table_reference*) ()
0000010 0x000055634c707140 in DoReloadConfig() ()
0000011 0x000055634c706c65 in SighandlerReloadConfig ()
0000012 <signal handler called>
0000013 0x00007f7760d1c7dd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000014 0x00007f776166733b in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000015 0x000055634c73b315 in wait_for_next_job(char*) ()
0000016 0x000055634c7069f1 in main ()

Thread 3 (Thread 0x7f775de3c700 (LWP 2782)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
0000001 0x00007f77616a00e5 in watchdog_thread () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f7760d13494 in start_thread (arg=0x7f775de3c700) at pthread_create.c:333
0000003 0x00007f775f552acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7f775e63d700 (LWP 2781)):
#0 0x00007f775f54967d in poll () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f77616572ed in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(void*)) () from /usr/lib/bareos/libbareos.so.18
0000002 0x000055634c733f12 in connect_thread ()
0000003 0x00007f7760d13494 in start_thread (arg=0x7f775e63d700) at pthread_create.c:333
0000004 0x00007f775f552acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7f77627f5b80 (LWP 2779)):
#0 0x00007f7760d1cb3a in __waitpid (pid=2915, stat_loc=0x7ffd20f76dcc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
0000001 0x00007f776168f9e8 in SignalHandler () from /usr/lib/bareos/libbareos.so.18
0000002 <signal handler called>
0000003 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
0000004 0x00007f775f49e42a in __GI_abort () at abort.c:89
0000005 0x00007f775f4dac00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f775f5cfd98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
0000006 0x00007f775f4e0fc6 in malloc_printerr (action=3, str=0x7f775f5cfe60 "double free or corruption (out)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
0000007 0x00007f775f4e180e in _int_free (av=0x7f775f803b00 <main_arena>, p=0x55634d89f970, have_lock=0) at malloc.c:3905
0000008 0x000055634c6fe4ea in FreeResource(CommonResourceHeader*, int) ()
0000009 0x000055634c705d8b in FreeSavedResources(resource_table_reference*) ()
0000010 0x000055634c707140 in DoReloadConfig() ()
0000011 0x000055634c706c65 in SighandlerReloadConfig ()
0000012 <signal handler called>
0000013 0x00007f7760d1c7dd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000014 0x00007f776166733b in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000015 0x000055634c73b315 in wait_for_next_job(char*) ()
0000016 0x000055634c7069f1 in main ()
#0 0x00007f7760d1cb3a in __waitpid (pid=2915, stat_loc=0x7ffd20f76dcc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29 in ../sysdeps/unix/sysv/linux/waitpid.c
resultvar = 18446744073709551104
sc_cancel_oldtype = 2
0000001 0x00007f776168f9e8 in SignalHandler () from /usr/lib/bareos/libbareos.so.18
No symbol table info available.
0000002 <signal handler called>
No locals.
0000003 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
set = {__val = {1, 0, 93884991526184, 140150730437832, 140725156541568, 140150730355432, 5, 0, 140725156541608, 140150728229057, 17, 140150730355432, 5, 0, 1, 140150730436976}}
pid = <optimized out>
tid = <optimized out>
0000004 0x00007f775f49e42a in __GI_abort () at abort.c:89
89 abort.c: No such file or directory.
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x55634d91d528, sa_sigaction = 0x55634d91d528}, sa_mask = {__val = {140150685565744, 140725156542560, 140150728278874, 93884990994408, 93884990969280, 4294967295, 0, 93884991004336, 93884990998160, 0, 93884991245504, 140725156541728, 140150663855075, 0, 93, 140725156542384}}, sa_flags = -9, sa_restorer = 0x7ffd20f77550}
sigs = {__val = {32, 0 <repeats 15 times>}}
0000005 0x00007f775f4dac00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f775f5cfd98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
175 ../sysdeps/posix/libc_fatal.c: No such file or directory.
ap = {{gp_offset = 40, fp_offset = 0, overflow_arg_area = 0x7ffd20f777c0, reg_save_area = 0x7ffd20f77750}}
fd = 2
on_2 = <optimized out>
list = <optimized out>
nlist = <optimized out>
cp = <optimized out>
written = <optimized out>
0000006 0x00007f775f4e0fc6 in malloc_printerr (action=3, str=0x7f775f5cfe60 "double free or corruption (out)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
5049 malloc.c: No such file or directory.
buf = "000055634d89f980"
cp = <optimized out>
ar_ptr = <optimized out>
ptr = <optimized out>
str = 0x7f775f5cfe60 "double free or corruption (out)"
action = 3
0000007 0x00007f775f4e180e in _int_free (av=0x7f775f803b00 <main_arena>, p=0x55634d89f970, have_lock=0) at malloc.c:3905
3905 in malloc.c
size = <optimized out>
fb = <optimized out>
nextchunk = <optimized out>
nextsize = <optimized out>
nextinuse = <optimized out>
prevsize = <optimized out>
bck = <optimized out>
fwd = <optimized out>
errstr = <optimized out>
locked = <optimized out>
__func__ = "_int_free"
gnif

gnif

2018-08-06 12:05

reporter   ~0003092

I found the error, FreeResource is attempting to free dhfile three times in a row, looks like a C&P bug.

Please see https://github.com/bareos/bareos/pull/104
gnif

gnif

2018-08-06 12:11

reporter   ~0003093

0000104 is not complete, please see:

https://github.com/bareos/bareos/pull/105
bruno-at-bareos

bruno-at-bareos

2023-07-17 16:35

manager   ~0005199

Fixed in master

Issue History

Date Modified Username Field Change
2018-08-06 08:51 gnif New Issue
2018-08-06 08:52 gnif Note Added: 0003087
2018-08-06 09:12 gnif Note Added: 0003088
2018-08-06 12:05 gnif Note Added: 0003092
2018-08-06 12:11 gnif Note Added: 0003093
2023-07-17 16:35 bruno-at-bareos Assigned To => bruno-at-bareos
2023-07-17 16:35 bruno-at-bareos Status new => closed
2023-07-17 16:35 bruno-at-bareos Resolution open => fixed
2023-07-17 16:35 bruno-at-bareos Note Added: 0005199