View Issue Details

IDProjectCategoryView StatusLast Update
0001084bareos-corestorage daemonpublic2019-07-22 10:10
Reporterdh-lin Assigned Toarogge  
PriorityhighSeveritymajorReproducibilityrandom
Status closedResolutionno change required 
PlatformLinuxOSDebianOS Version9
Product Version17.2.7 
Summary0001084: bareos-sd hangs ramdomly on backups
DescriptionOn doing some bigger backups with lot of data bareos-sd stop working. No message in syslog. No backtrace.
All connections are still there and some packats in RECV Queue. One tape is open but bareos-sd does not write in this tape anymore.
strace on process see lot of 'futex' and some 'ressource unaviable'.
Let me know which informations (logfiles etc or strace on next hand or something else) you need
Steps To ReproduceIs happened randomly
Additional InformationWe run with 100 concurent jobs on bareos-sd and use one tape (fila based) for each job.
TagsNo tags attached.

Activities

dh-lin

dh-lin

2019-05-16 09:28

reporter   ~0003374

Here is strace from hanging process:

strace: Process 10405 attached with 51 threads
[pid 12582] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12581] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1441, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12579] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1440, NULL <unfinished ...>
[pid 12581] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12579] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12579] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12578] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1439, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12577] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1438, NULL <unfinished ...>
[pid 12578] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12577] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12577] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12570] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1437, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12569] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1436, NULL <unfinished ...>
[pid 12570] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12569] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12569] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12568] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1435, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12567] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1434, NULL <unfinished ...>
[pid 12568] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12567] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12567] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12566] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1433, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12564] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1432, NULL <unfinished ...>
[pid 12566] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12564] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12564] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12563] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1430, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12562] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1429, NULL <unfinished ...>
[pid 12563] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12562] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12562] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12561] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1431, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12560] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1428, NULL <unfinished ...>
[pid 12561] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12560] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12560] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12559] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1427, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12558] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1425, NULL <unfinished ...>
[pid 12559] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12558] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12558] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12557] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1426, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12554] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1424, NULL <unfinished ...>
[pid 12557] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12554] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12554] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12553] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1422, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12552] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1423, NULL <unfinished ...>
[pid 12553] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12552] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12552] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12551] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1421, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12550] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1420, NULL <unfinished ...>
[pid 12551] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12550] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12550] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12549] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1419, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12548] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1418, NULL <unfinished ...>
[pid 12549] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12548] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12548] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12547] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1415, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12546] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1414, NULL <unfinished ...>
[pid 12547] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12546] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12546] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12545] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1417, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12544] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1416, NULL <unfinished ...>
[pid 12545] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12544] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12544] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12543] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1413, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 12542] restart_syscall(<... resuming interrupted futex ...> <unfinished ...>
[pid 12543] futex(0x7fce320d8c5c, FUTEX_WAIT_PRIVATE, 1442, NULL <unfinished ...>
[pid 12541] futex(0x7fce31c719c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12540] futex(0x7fce31c719c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12539] futex(0x7fce31c719c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12538] futex(0x7fce320d8ae0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12537] futex(0x7fce320d8ae0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12536] futex(0x7fce320d8ae0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12535] futex(0x7fce320d8ae0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12533] futex(0x7fce31c719c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12532] futex(0x7fce320d8ae0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12530] futex(0x7fce31c719c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12529] futex(0x7fce31c719c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12527] futex(0x7fce31c719c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 12526] read(59, <unfinished ...>
[pid 12523] futex(0x7fcdd4000020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 10527] read(10, <unfinished ...>
[pid 10464] read(6, <unfinished ...>
[pid 10462] read(27, <unfinished ...>
[pid 10408] restart_syscall(<... resuming interrupted futex ...> <unfinished ...>
[pid 10406] restart_syscall(<... resuming interrupted nanosleep ...> <unfinished ...>
[pid 10405] futex(0x7fce31c719c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 10406] <... restart_syscall resumed> ) = 0
[pid 10406] nanosleep({tv_sec=30, tv_nsec=0}, <unfinished ...>
[pid 12526] <... read resumed> ""..., 4) = 4
[pid 12526] read(59, ""..., 20) = 20
[pid 12526] write(59, ""..., 29) = 29
[pid 12526] read(59, ""..., 4) = 4
[pid 12526] read(59, ""..., 15) = 15
[pid 12526] futex(0x7fce31c719c0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 10408] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 10408] futex(0x7fce31c726a0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10408] futex(0x7fce31c72664, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 575, {tv_sec=1557991709, tv_nsec=443233000}, 0xffffffff <unfinished ...>
[pid 10406] <... nanosleep resumed> NULL) = 0
[pid 10406] nanosleep({tv_sec=30, tv_nsec=0}, NULL) = 0
[pid 10406] nanosleep({tv_sec=30, tv_nsec=0},
dh-lin

dh-lin

2019-05-16 11:03

reporter   ~0003375

With -d 200 we got this message:

backup2-de1-sd (100): dev.c:487-190398 created new block of buf_len: 1048576 on device "pl2_tape2_incr" (/var/lib/bareos/storage/tapes_ng//standard/incr)
BAREOS interrupted by signal 11: Segmentation violation

And bareos-sd stopped working. No trace or something else. Just freeze
nwhisper

nwhisper

2019-05-21 12:04

reporter   ~0003378

Developers, please start fixing your bugs! Or stop telling everyone that your product works somehow.
dh-lin

dh-lin

2019-05-23 02:22

reporter   ~0003379

Another one outage from bareos-sd.
Here postmortem trace:


Created /var/lib/bareos/bareos-sd.core.20143 for doing postmortem debugging
[New LWP 20143]
[New LWP 20144]
[New LWP 20146]
[New LWP 1435]
[New LWP 12742]
[New LWP 25181]
[New LWP 25211]
[New LWP 25215]
[New LWP 25387]
[New LWP 25388]
[New LWP 25390]
[New LWP 25392]
[New LWP 25393]
[New LWP 25397]
[New LWP 25400]
[New LWP 25401]
[New LWP 25402]
[New LWP 25403]
[New LWP 25404]
[New LWP 25405]
[New LWP 25406]
[New LWP 25407]
[New LWP 25408]
[New LWP 25409]
[New LWP 25410]
[New LWP 25411]
[New LWP 25412]
[New LWP 25413]
[New LWP 25414]
[New LWP 25415]
[New LWP 25416]
[New LWP 25417]
[New LWP 25418]
[New LWP 25419]
[New LWP 25420]
[New LWP 25421]
[New LWP 25422]
[New LWP 25423]
[New LWP 25424]
[New LWP 25425]
[New LWP 25427]
[New LWP 25460]
[New LWP 25461]
[New LWP 25465]
[New LWP 25501]
[New LWP 25502]
[New LWP 25503]
[New LWP 25504]
[New LWP 25573]
[New LWP 25575]
[New LWP 25576]
[New LWP 25577]
[New LWP 25643]
[New LWP 25645]
[New LWP 25647]
[New LWP 25843]
[New LWP 26038]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/bareos-sd'.
#0 0x00007f4f914978bd in poll () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
[Current thread is 1 (Thread 0x7f4f940e5cc0 (LWP 20143))]
From To Syms Read Shared Object Library
0x00007f4f93c79fd0 0x00007f4f93cae315 Yes /usr/lib/bareos/libbareosndmp-17.2.7.so
0x00007f4f93a08650 0x00007f4f93a31469 Yes /usr/lib/bareos/libbareossd-17.2.7.so
0x00007f4f937e7830 0x00007f4f937f3efd Yes /usr/lib/bareos/libbareoscfg-17.2.7.so
0x00007f4f9358ca00 0x00007f4f935c59be Yes /usr/lib/bareos/libbareos-17.2.7.so
0x00007f4f9335b1c0 0x00007f4f9336bafe Yes (*) /lib/x86_64-linux-gnu/libz.so.1
0x00007f4f931395d0 0x00007f4f931546cd Yes (*) /lib/x86_64-linux-gnu/liblzo2.so.2
0x00007f4f92f2c1d0 0x00007f4f92f330f1 Yes (*) /usr/lib/x86_64-linux-gnu/libjansson.so.4
0x00007f4f92cd6ab0 0x00007f4f92d0e356 Yes (*) /usr/lib/x86_64-linux-gnu/libssl.so.1.1
0x00007f4f928a4000 0x00007f4f92a138e9 Yes (*) /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
0x00007f4f9260dab0 0x00007f4f9261a831 Yes /lib/x86_64-linux-gnu/libpthread.so.0
0x00007f4f92404d80 0x00007f4f9240594e Yes /lib/x86_64-linux-gnu/libdl.so.2
0x00007f4f921fcd00 0x00007f4f92200864 Yes (*) /lib/x86_64-linux-gnu/libwrap.so.0
0x00007f4f91ff58c0 0x00007f4f91ff72a6 Yes (*) /lib/x86_64-linux-gnu/libcap.so.2
0x00007f4f91cfd7e0 0x00007f4f91da5d49 Yes (*) /usr/lib/x86_64-linux-gnu/libstdc++.so.6
0x00007f4f91973680 0x00007f4f919df8da Yes /lib/x86_64-linux-gnu/libm.so.6
0x00007f4f91759a90 0x00007f4f91769895 Yes (*) /lib/x86_64-linux-gnu/libgcc_s.so.1
0x00007f4f913d7940 0x00007f4f91501613 Yes /lib/x86_64-linux-gnu/libc.so.6
0x00007f4f93ed6aa0 0x00007f4f93ef1fb0 Yes /lib64/ld-linux-x86-64.so.2
0x00007f4f911a3fd0 0x00007f4f911af271 Yes /lib/x86_64-linux-gnu/libnsl.so.1
0x00007f4f885e91d0 0x00007f4f885eee91 Yes /lib/x86_64-linux-gnu/libnss_files.so.2
(*): Shared library is missing debugging information.
$1 = 0x55e6cd82bd80 <my_name> "backup7-de1-sd"
$2 = 0x55e6cee41c98 "bareos-sd"
$3 = 0x55e6cee41cd8 "/usr/sbin/bareos-sd"
$4 = 0x0
$5 = 0x7f4f935cb008 "17.2.7 (16 Jul 2018)"
$6 = 0x7f4f935caff4 "x86_64-pc-linux-gnu"
$7 = 0x7f4f935cafed "debian"
$8 = 0x7f4f935cb4e8 "Debian GNU/Linux 9.5 (stretch)"
$9 = "backup7-de1", '\000' <repeats 244 times>
$10 = 0x7f4f935cb508 "debian Debian GNU/Linux 9.5 (stretch)"
Environment variable "TestName" not defined.
#0 0x00007f4f914978bd in poll () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f9358ffe3 in bnet_thread_server_tcp (addr_list=<optimized out>, max_clients=<optimized out>, sockfds=<optimized out>, client_wq=<optimized out>, nokeepalive=<optimized out>, handle_client_request=<optimized out>) at bnet_server_tcp.c:306
0000002 0x000055e6cd60d88f in main (argc=<optimized out>, argv=<optimized out>) at stored.c:322

Thread 57 (Thread 0x7f4f267fc700 (LWP 26038)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06d1e8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06d1e8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf059938) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f267fc700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 56 (Thread 0x7f4f8affd700 (LWP 25843)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06d498) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06d498) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05ea98) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f8affd700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 55 (Thread 0x7f4ef1fe3700 (LWP 25647)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05eae8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05eae8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05ecf8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef1fe3700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 54 (Thread 0x7f4f9099e700 (LWP 25645)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05e6c8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05e6c8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05e8d8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f9099e700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 53 (Thread 0x7f4f467fc700 (LWP 25643)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cee422a8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cee422a8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08f438) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f467fc700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 52 (Thread 0x7f4f27fff700 (LWP 25577)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05e1c8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05e1c8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05e3d8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f27fff700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 51 (Thread 0x7f4ef27e4700 (LWP 25576)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05e008) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05e008) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05e1c8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef27e4700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 50 (Thread 0x7f4ef5feb700 (LWP 25575)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05de48) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05de48) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08dc98) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef5feb700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 49 (Thread 0x7f4ef37e6700 (LWP 25573)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05bfb8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05bfb8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05dcc8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef37e6700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 48 (Thread 0x7f4ef3fe7700 (LWP 25504)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08f278) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08f278) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05d988) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef3fe7700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 47 (Thread 0x7f4ef47e8700 (LWP 25503)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08f0a8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08f0a8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08f278) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef47e8700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 46 (Thread 0x7f4f44ff9700 (LWP 25502)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05b208) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05b208) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08cde8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f44ff9700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 45 (Thread 0x7f4f45ffb700 (LWP 25501)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05d648) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05d648) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05d858) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f45ffb700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 44 (Thread 0x7f4ef2fe5700 (LWP 25465)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05c8d8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05c8d8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05c178) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef2fe5700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 43 (Thread 0x7f4ef4fe9700 (LWP 25461)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05bc78) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05bc78) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05be38) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef4fe9700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 42 (Thread 0x7f4ef57ea700 (LWP 25460)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05f6d8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05f6d8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05f898) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef57ea700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 41 (Thread 0x7f4ef67ec700 (LWP 25427)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf0799b8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf0799b8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf079838) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef67ec700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 40 (Thread 0x7f4ef77ee700 (LWP 25425)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf079678) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf079678) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf079838) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef77ee700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 39 (Thread 0x7f4ef7fef700 (LWP 25424)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf079338) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf079338) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf0794f8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef7fef700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 38 (Thread 0x7f4ef87f0700 (LWP 25423)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf078ff8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf078ff8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf0791b8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef87f0700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 37 (Thread 0x7f4ef8ff1700 (LWP 25422)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf078cb8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf078cb8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf078e78) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef8ff1700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 36 (Thread 0x7f4ef97f2700 (LWP 25421)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf078978) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf078978) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf078b38) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef97f2700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 35 (Thread 0x7f4ef9ff3700 (LWP 25420)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf078638) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf078638) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf0787f8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4ef9ff3700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 34 (Thread 0x7f4efa7f4700 (LWP 25419)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf0782f8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf0782f8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf0784b8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efa7f4700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 33 (Thread 0x7f4efaff5700 (LWP 25418)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf077ff8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf077ff8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf078178) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efaff5700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 32 (Thread 0x7f4efb7f6700 (LWP 25417)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf070508) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf070508) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf070718) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efb7f6700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 31 (Thread 0x7f4efbff7700 (LWP 25416)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06f588) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06f588) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06f748) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efbff7700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 30 (Thread 0x7f4efc7f8700 (LWP 25415)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06f248) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06f248) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06f408) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efc7f8700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 29 (Thread 0x7f4efcff9700 (LWP 25414)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06ef08) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06ef08) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06f0c8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efcff9700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 28 (Thread 0x7f4efd7fa700 (LWP 25413)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08caa8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08caa8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08cc68) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efd7fa700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 27 (Thread 0x7f4efdffb700 (LWP 25412)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08c768) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08c768) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08c928) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efdffb700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 26 (Thread 0x7f4efe7fc700 (LWP 25411)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08c428) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08c428) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08c5e8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efe7fc700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 25 (Thread 0x7f4efeffd700 (LWP 25410)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08c0e8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08c0e8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08c2a8) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efeffd700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 24 (Thread 0x7f4eff7fe700 (LWP 25409)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08bda8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08bda8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08bf68) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4eff7fe700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 23 (Thread 0x7f4efffff700 (LWP 25408)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08ba68) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08ba68) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08bc28) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4efffff700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 22 (Thread 0x7f4f04ff9700 (LWP 25407)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06e8a8) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06e8a8) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06ea68) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f04ff9700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 21 (Thread 0x7f4f057fa700 (LWP 25406)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f4f935b8906 in rwl_writelock_p (rwl=rwl@entry=0x7f4f93c47c00 <reservation_lock>, file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at rwlock.c:236
0000002 0x00007f4f93a2335e in _lock_reservations (file=file@entry=0x7f4f93a3939f "reserve.c", line=line@entry=303) at reserve.c:113
0000003 0x00007f4f93a25d33 in use_device_cmd (jcr=<optimized out>) at reserve.c:303
0000004 use_cmd (jcr=<optimized out>) at reserve.c:76
0000005 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06e568) at dir_cmd.c:315
0000006 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06e568) at socket_server.c:100
0000007 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000008 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06e728) at lockmgr.c:928
0000009 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f057fa700) at pthread_create.c:456
0000010 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 20 (Thread 0x7f4f05ffb700 (LWP 25405)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
0000001 0x00007f4f92611bb5 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f4f937e09c0 <_ZL5mutex>) at ../nptl/pthread_mutex_lock.c:80
0000002 0x00007f4f935aac1b in lmgr_p (m=m@entry=0x7f4f937e09c0 <_ZL5mutex>) at lockmgr.c:91
0000003 0x00007f4f935ab9dd in bthread_mutex_lock_p (m=m@entry=0x7f4f937e09c0 <_ZL5mutex>, file=file@entry=0x7f4f935caa45 "mem_pool.c", line=line@entry=145) at lockmgr.c:770
0000004 0x00007f4f935ac7fb in sm_get_pool_memory (fname=fname@entry=0x7f4f93a384b2 "record.c", lineno=lineno@entry=493, pool=pool@entry=6) at mem_pool.c:145
0000005 0x00007f4f93a2208e in new_record (with_data=with_data@entry=true) at record.c:493
0000006 0x00007f4f93a08f8a in setup_new_dcr_device (jcr=0x7f4f58017ae8, dcr=0x7f4f58016328, dev=0x7f4f8433a268, blocksizes=blocksizes@entry=0x0) at acquire.c:743
0000007 0x00007f4f93a2399e in reserve_device (rctx=...) at reserve.c:745
0000008 0x00007f4f93a252e3 in find_suitable_device_for_job (jcr=jcr@entry=0x7f4f58017ae8, rctx=...) at reserve.c:525
0000009 0x00007f4f93a25e65 in use_device_cmd (jcr=<optimized out>) at reserve.c:347
0000010 use_cmd (jcr=<optimized out>) at reserve.c:76
0000011 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06e228) at dir_cmd.c:315
0000012 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06e228) at socket_server.c:100
0000013 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000014 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06e3e8) at lockmgr.c:928
0000015 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f05ffb700) at pthread_create.c:456
0000016 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 19 (Thread 0x7f4f067fc700 (LWP 25404)):
#0 0x00007f4f9261822d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f935991e7 in BSOCK_TCP::read_nbytes (this=0x55e6cf05d108, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f4f93598b25 in BSOCK_TCP::recv (this=0x55e6cf05d108) at bsock_tcp.c:550
0000003 0x000055e6cd615a65 in do_fd_commands (jcr=0x7f4f5c006398) at fd_cmds.c:213
0000004 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f5c006398) at fd_cmds.c:183
0000005 0x000055e6cd61678a in do_job_run (jcr=0x7f4f5c006398) at job.c:238
0000006 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06dee8) at dir_cmd.c:315
0000007 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06dee8) at socket_server.c:100
0000008 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000009 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06e0a8) at lockmgr.c:928
0000010 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f067fc700) at pthread_create.c:456
0000011 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 18 (Thread 0x7f4f077fe700 (LWP 25403)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
0000001 0x00007f4f92611bb5 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f4f93c47ae0 <mount_mutex>) at ../nptl/pthread_mutex_lock.c:80
0000002 0x00007f4f935aac1b in lmgr_p (m=m@entry=0x7f4f93c47ae0 <mount_mutex>) at lockmgr.c:91
0000003 0x00007f4f935ab9dd in bthread_mutex_lock_p (m=m@entry=0x7f4f93c47ae0 <mount_mutex>, file=file@entry=0x7f4f93a38194 "mount.c", line=line@entry=77) at lockmgr.c:770
0000004 0x00007f4f93a1eed2 in DCR::mount_next_write_volume (this=this@entry=0x7f4f50006918) at mount.c:77
0000005 0x00007f4f93a089f7 in acquire_device_for_append (dcr=dcr@entry=0x7f4f50006918) at acquire.c:441
0000006 0x000055e6cd60dee1 in do_append_data (jcr=jcr@entry=0x7f4f50001058, bs=bs@entry=0x55e6cf08cfe8, what=what@entry=0x55e6cd623b28 "FD") at append.c:77
0000007 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f50001058) at fd_cmds.c:271
0000008 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f50001058) at fd_cmds.c:227
0000009 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f50001058) at fd_cmds.c:183
0000010 0x000055e6cd61678a in do_job_run (jcr=0x7f4f50001058) at job.c:238
0000011 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06dba8) at dir_cmd.c:315
0000012 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06dba8) at socket_server.c:100
0000013 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000014 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06dd68) at lockmgr.c:928
0000015 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f077fe700) at pthread_create.c:456
0000016 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 17 (Thread 0x7f4f07fff700 (LWP 25402)):
#0 0x00007f4f9261822d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f935991e7 in BSOCK_TCP::read_nbytes (this=0x55e6cf08dad8, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f4f93598b25 in BSOCK_TCP::recv (this=0x55e6cf08dad8) at bsock_tcp.c:550
0000003 0x000055e6cd615a65 in do_fd_commands (jcr=0x7f4f54107578) at fd_cmds.c:213
0000004 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f54107578) at fd_cmds.c:183
0000005 0x000055e6cd61678a in do_job_run (jcr=0x7f4f54107578) at job.c:238
0000006 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf06d868) at dir_cmd.c:315
0000007 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf06d868) at socket_server.c:100
0000008 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000009 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06da28) at lockmgr.c:928
0000010 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f07fff700) at pthread_create.c:456
0000011 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 16 (Thread 0x7f4f24ff9700 (LWP 25401)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
0000001 0x00007f4f92611bb5 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f4f93c47ae0 <mount_mutex>) at ../nptl/pthread_mutex_lock.c:80
0000002 0x00007f4f935aac1b in lmgr_p (m=m@entry=0x7f4f93c47ae0 <mount_mutex>) at lockmgr.c:91
0000003 0x00007f4f935ab9dd in bthread_mutex_lock_p (m=m@entry=0x7f4f93c47ae0 <mount_mutex>, file=file@entry=0x7f4f93a38194 "mount.c", line=line@entry=77) at lockmgr.c:770
0000004 0x00007f4f93a1eed2 in DCR::mount_next_write_volume (this=this@entry=0x7f4f380008e8) at mount.c:77
0000005 0x00007f4f93a089f7 in acquire_device_for_append (dcr=dcr@entry=0x7f4f380008e8) at acquire.c:441
0000006 0x000055e6cd60dee1 in do_append_data (jcr=jcr@entry=0x7f4f38001818, bs=bs@entry=0x55e6cf05c5e8, what=what@entry=0x55e6cd623b28 "FD") at append.c:77
0000007 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f38001818) at fd_cmds.c:271
0000008 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f38001818) at fd_cmds.c:227
0000009 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f38001818) at fd_cmds.c:183
0000010 0x000055e6cd61678a in do_job_run (jcr=0x7f4f38001818) at job.c:238
0000011 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08e178) at dir_cmd.c:315
0000012 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08e178) at socket_server.c:100
0000013 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000014 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08e338) at lockmgr.c:928
0000015 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f24ff9700) at pthread_create.c:456
0000016 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 15 (Thread 0x7f4f26ffd700 (LWP 25400)):
#0 0x00007f4f92618b5a in __waitpid (pid=pid@entry=26103, stat_loc=stat_loc@entry=0x7f4f26ffbc8c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
0000001 0x00007f4f935bb3f4 in signal_handler (sig=11) at signal.c:240
0000002 <signal handler called>
0000003 smart_alloc_msg (file=0x7f4f935cdf08 "smartall.c", line=line@entry=229, fmt=0x7f4f935ce048 "Overrun buffer: len=%d addr=%p allocated: %s:%d called from %s:%d\n", file=0x7f4f935cdf08 "smartall.c") at smartall.c:113
0000004 0x00007f4f935bc1a9 in sm_free (file=file@entry=0x7f4f935caa45 "mem_pool.c", line=line@entry=257, fp=0x7f4f58016938) at smartall.c:230
0000005 0x00007f4f935acd31 in sm_free_pool_memory (fname=fname@entry=0x7f4f93a33439 "block.c", lineno=lineno@entry=178, obuf=0x7f4f58016950 "") at mem_pool.c:257
0000006 0x00007f4f93a109aa in free_block (block=<optimized out>) at block.c:178
0000007 0x00007f4f93a16513 in DEVICE::set_blocksizes (this=0x7f4f8433a268, dcr=dcr@entry=0x7f4f58017568) at dev.c:488
0000008 0x00007f4f93a1b58a in write_new_volume_label_to_dev (dcr=dcr@entry=0x7f4f58017568, VolName=VolName@entry=0x7f4f58017680 "file-pl2-tape7-xaxis_turbine-incr-37161", PoolName=PoolName@entry=0x7f4f58017700 "pool-pl2-tape7-xaxis_turbine-incr", relabel=relabel@entry=false) at label.c:463
0000009 0x00007f4f93a1e38c in DCR::try_autolabel (this=0x7f4f58017568, opened=<optimized out>) at mount.c:781
0000010 0x00007f4f93a1f58a in DCR::mount_next_write_volume (this=this@entry=0x7f4f58017568) at mount.c:245
0000011 0x00007f4f93a089f7 in acquire_device_for_append (dcr=dcr@entry=0x7f4f58017568) at acquire.c:441
0000012 0x000055e6cd60dee1 in do_append_data (jcr=jcr@entry=0x7f4f58021518, bs=bs@entry=0x55e6cf05cbc8, what=what@entry=0x55e6cd623b28 "FD") at append.c:77
0000013 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f58021518) at fd_cmds.c:271
0000014 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f58021518) at fd_cmds.c:227
0000015 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f58021518) at fd_cmds.c:183
0000016 0x000055e6cd61678a in do_job_run (jcr=0x7f4f58021518) at job.c:238
0000017 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08de88) at dir_cmd.c:315
#18 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08de88) at socket_server.c:100
#19 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000020 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05b918) at lockmgr.c:928
0000021 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f26ffd700) at pthread_create.c:456
0000022 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 14 (Thread 0x7f4f6a7fc700 (LWP 25397)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
0000001 0x00007f4f92611bb5 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f4f93c47ae0 <mount_mutex>) at ../nptl/pthread_mutex_lock.c:80
0000002 0x00007f4f935aac1b in lmgr_p (m=m@entry=0x7f4f93c47ae0 <mount_mutex>) at lockmgr.c:91
0000003 0x00007f4f935ab9dd in bthread_mutex_lock_p (m=m@entry=0x7f4f93c47ae0 <mount_mutex>, file=file@entry=0x7f4f93a38194 "mount.c", line=line@entry=77) at lockmgr.c:770
0000004 0x00007f4f93a1eed2 in DCR::mount_next_write_volume (this=this@entry=0x7f4f5c008c08) at mount.c:77
0000005 0x00007f4f93a089f7 in acquire_device_for_append (dcr=dcr@entry=0x7f4f5c008c08) at acquire.c:441
0000006 0x000055e6cd60dee1 in do_append_data (jcr=jcr@entry=0x7f4f5c0081a8, bs=bs@entry=0x55e6cf05cd88, what=what@entry=0x55e6cd623b28 "FD") at append.c:77
0000007 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f5c0081a8) at fd_cmds.c:271
0000008 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f5c0081a8) at fd_cmds.c:227
0000009 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f5c0081a8) at fd_cmds.c:183
0000010 0x000055e6cd61678a in do_job_run (jcr=0x7f4f5c0081a8) at job.c:238
0000011 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08d2e8) at dir_cmd.c:315
0000012 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08d2e8) at socket_server.c:100
0000013 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000014 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08d4a8) at lockmgr.c:928
0000015 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f6a7fc700) at pthread_create.c:456
0000016 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 13 (Thread 0x7f4f88ff9700 (LWP 25393)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
0000001 0x00007f4f92611bb5 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f4f937e09c0 <_ZL5mutex>) at ../nptl/pthread_mutex_lock.c:80
0000002 0x00007f4f935aac1b in lmgr_p (m=m@entry=0x7f4f937e09c0 <_ZL5mutex>) at lockmgr.c:91
0000003 0x00007f4f935ab9dd in bthread_mutex_lock_p (m=m@entry=0x7f4f937e09c0 <_ZL5mutex>, file=file@entry=0x7f4f935caa45 "mem_pool.c", line=line@entry=145) at lockmgr.c:770
0000004 0x00007f4f935ac7fb in sm_get_pool_memory (fname=fname@entry=0x7f4f935c5c81 "../lib/mem_pool.h", lineno=lineno@entry=93, pool=pool@entry=4) at mem_pool.c:145
0000005 0x00007f4f935b1854 in POOL_MEM::POOL_MEM (pool=4, this=0x7f4f88ff8948) at ../lib/mem_pool.h:93
0000006 Jmsg (jcr=jcr@entry=0x7f4f14012c28, type=type@entry=6, mtime=mtime@entry=0, fmt=0x55e6cd621908 "Elapsed time=%02d:%02d:%02d, Transfer rate=%s Bytes/second\n") at message.c:1554
0000007 0x000055e6cd60e410 in do_append_data (jcr=jcr@entry=0x7f4f14012c28, bs=bs@entry=0x55e6cf05e3d8, what=what@entry=0x55e6cd623b28 "FD") at append.c:334
0000008 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f14012c28) at fd_cmds.c:271
0000009 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f14012c28) at fd_cmds.c:227
0000010 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f14012c28) at fd_cmds.c:183
0000011 0x000055e6cd61678a in do_job_run (jcr=0x7f4f14012c28) at job.c:238
0000012 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08b628) at dir_cmd.c:315
0000013 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08b628) at socket_server.c:100
0000014 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000015 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf06d318) at lockmgr.c:928
0000016 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f88ff9700) at pthread_create.c:456
0000017 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 12 (Thread 0x7f4f89ffb700 (LWP 25392)):
#0 0x00007f4f9261822d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f935991e7 in BSOCK_TCP::read_nbytes (this=0x55e6cf05db08, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f4f93598b25 in BSOCK_TCP::recv (this=0x55e6cf05db08) at bsock_tcp.c:550
0000003 0x00007f4f9358f36d in bget_msg (sock=sock@entry=0x55e6cf05db08) at bget_msg.c:53
0000004 0x000055e6cd60e158 in do_append_data (jcr=jcr@entry=0x7f4f40001e58, bs=bs@entry=0x55e6cf05db08, what=what@entry=0x55e6cd623b28 "FD") at append.c:209
0000005 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f40001e58) at fd_cmds.c:271
0000006 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f40001e58) at fd_cmds.c:227
0000007 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f40001e58) at fd_cmds.c:183
0000008 0x000055e6cd61678a in do_job_run (jcr=0x7f4f40001e58) at job.c:238
0000009 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05aa38) at dir_cmd.c:315
0000010 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05aa38) at socket_server.c:100
0000011 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000012 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08b4a8) at lockmgr.c:928
0000013 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f89ffb700) at pthread_create.c:456
0000014 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 11 (Thread 0x7f4f69ffb700 (LWP 25390)):
#0 0x00007f4f9261822d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f935991e7 in BSOCK_TCP::read_nbytes (this=0x55e6cf05d488, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f4f93598b25 in BSOCK_TCP::recv (this=0x55e6cf05d488) at bsock_tcp.c:550
0000003 0x000055e6cd615a65 in do_fd_commands (jcr=0x7f4f3c0008e8) at fd_cmds.c:213
0000004 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f3c0008e8) at fd_cmds.c:183
0000005 0x000055e6cd61678a in do_job_run (jcr=0x7f4f3c0008e8) at job.c:238
0000006 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf059cb8) at dir_cmd.c:315
0000007 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf059cb8) at socket_server.c:100
0000008 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000009 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cee45ae8) at lockmgr.c:928
0000010 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f69ffb700) at pthread_create.c:456
0000011 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 10 (Thread 0x7f4f477fe700 (LWP 25388)):
#0 0x00007f4f9261822d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f935991e7 in BSOCK_TCP::read_nbytes (this=0x55e6cf05c178, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f4f93598b25 in BSOCK_TCP::recv (this=0x55e6cf05c178) at bsock_tcp.c:550
0000003 0x00007f4f9358f36d in bget_msg (sock=sock@entry=0x55e6cf05c178) at bget_msg.c:53
0000004 0x000055e6cd60e0c5 in do_append_data (jcr=jcr@entry=0x7f4f844a5688, bs=bs@entry=0x55e6cf05c178, what=what@entry=0x55e6cd623b28 "FD") at append.c:158
0000005 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f844a5688) at fd_cmds.c:271
0000006 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f844a5688) at fd_cmds.c:227
0000007 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f844a5688) at fd_cmds.c:183
0000008 0x000055e6cd61678a in do_job_run (jcr=0x7f4f844a5688) at job.c:238
0000009 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08ad48) at dir_cmd.c:315
0000010 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08ad48) at socket_server.c:100
0000011 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000012 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cee45ae8) at lockmgr.c:928
0000013 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f477fe700) at pthread_create.c:456
0000014 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 9 (Thread 0x7f4f8b7fe700 (LWP 25387)):
#0 0x00007f4f9261822d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f935991e7 in BSOCK_TCP::read_nbytes (this=0x55e6cf05d2c8, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f4f93598b25 in BSOCK_TCP::recv (this=0x55e6cf05d2c8) at bsock_tcp.c:550
0000003 0x000055e6cd615a65 in do_fd_commands (jcr=0x7f4f4c0050d8) at fd_cmds.c:213
0000004 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f4c0050d8) at fd_cmds.c:183
0000005 0x000055e6cd61678a in do_job_run (jcr=0x7f4f4c0050d8) at job.c:238
0000006 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05faa8) at dir_cmd.c:315
0000007 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05faa8) at socket_server.c:100
0000008 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000009 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cee45a98) at lockmgr.c:928
0000010 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f8b7fe700) at pthread_create.c:456
0000011 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 8 (Thread 0x7f4f25ffb700 (LWP 25215)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
0000001 0x00007f4f92611bb5 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f4f937e09c0 <_ZL5mutex>) at ../nptl/pthread_mutex_lock.c:80
0000002 0x00007f4f935aac1b in lmgr_p (m=m@entry=0x7f4f937e09c0 <_ZL5mutex>) at lockmgr.c:91
0000003 0x00007f4f935ab9dd in bthread_mutex_lock_p (m=m@entry=0x7f4f937e09c0 <_ZL5mutex>, file=file@entry=0x7f4f935caa45 "mem_pool.c", line=line@entry=145) at lockmgr.c:770
0000004 0x00007f4f935ac7fb in sm_get_pool_memory (fname=fname@entry=0x7f4f93a384b2 "record.c", lineno=lineno@entry=493, pool=pool@entry=6) at mem_pool.c:145
0000005 0x00007f4f93a2208e in new_record (with_data=with_data@entry=true) at record.c:493
0000006 0x00007f4f93a1a21c in write_session_label (dcr=dcr@entry=0x7f4f780020a8, label=label@entry=-5) at label.c:799
0000007 0x000055e6cd60e5e5 in do_append_data (jcr=jcr@entry=0x7f4f78006db8, bs=bs@entry=0x55e6cf05cf48, what=what@entry=0x55e6cd623b28 "FD") at append.c:277
0000008 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f78006db8) at fd_cmds.c:271
0000009 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f78006db8) at fd_cmds.c:227
0000010 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f78006db8) at fd_cmds.c:183
0000011 0x000055e6cd61678a in do_job_run (jcr=0x7f4f78006db8) at job.c:238
0000012 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08b0c8) at dir_cmd.c:315
0000013 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08b0c8) at socket_server.c:100
0000014 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000015 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08b2d8) at lockmgr.c:928
0000016 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f25ffb700) at pthread_create.c:456
0000017 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 7 (Thread 0x7f4f06ffd700 (LWP 25211)):
#0 0x00007f4f9261822d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f935991e7 in BSOCK_TCP::read_nbytes (this=0x55e6cf05c2f8, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f4f93598b25 in BSOCK_TCP::recv (this=0x55e6cf05c2f8) at bsock_tcp.c:550
0000003 0x000055e6cd615a65 in do_fd_commands (jcr=0x7f4f740044a8) at fd_cmds.c:213
0000004 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f740044a8) at fd_cmds.c:183
0000005 0x000055e6cd61678a in do_job_run (jcr=0x7f4f740044a8) at job.c:238
0000006 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf071b18) at dir_cmd.c:315
0000007 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf071b18) at socket_server.c:100
0000008 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000009 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08e508) at lockmgr.c:928
0000010 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f06ffd700) at pthread_create.c:456
0000011 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 6 (Thread 0x7f4f277fe700 (LWP 25181)):
#0 0x00007f4f9261822d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f935991e7 in BSOCK_TCP::read_nbytes (this=0x55e6cf079ca8, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f4f93598b25 in BSOCK_TCP::recv (this=0x55e6cf079ca8) at bsock_tcp.c:550
0000003 0x000055e6cd615a65 in do_fd_commands (jcr=0x7f4f18002438) at fd_cmds.c:213
0000004 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f18002438) at fd_cmds.c:183
0000005 0x000055e6cd61678a in do_job_run (jcr=0x7f4f18002438) at job.c:238
0000006 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf08f568) at dir_cmd.c:315
0000007 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf08f568) at socket_server.c:100
0000008 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000009 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cee45a98) at lockmgr.c:928
0000010 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f277fe700) at pthread_create.c:456
0000011 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 5 (Thread 0x7f4f457fa700 (LWP 12742)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
0000001 0x00007f4f92611bb5 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f4f93c47ae0 <mount_mutex>) at ../nptl/pthread_mutex_lock.c:80
0000002 0x00007f4f935aac1b in lmgr_p (m=m@entry=0x7f4f93c47ae0 <mount_mutex>) at lockmgr.c:91
0000003 0x00007f4f935ab9dd in bthread_mutex_lock_p (m=m@entry=0x7f4f93c47ae0 <mount_mutex>, file=file@entry=0x7f4f93a38194 "mount.c", line=line@entry=77) at lockmgr.c:770
0000004 0x00007f4f93a1eed2 in DCR::mount_next_write_volume (this=this@entry=0x7f4f54107c48) at mount.c:77
0000005 0x00007f4f93a18c3a in fixup_device_block_write_error (dcr=dcr@entry=0x7f4f54107c48, retries=retries@entry=4) at device.c:120
0000006 0x00007f4f93a137a8 in DCR::write_block_to_device (this=this@entry=0x7f4f54107c48) at block.c:410
0000007 0x00007f4f93a227a8 in DCR::write_record (this=this@entry=0x7f4f54107c48) at record.c:668
0000008 0x000055e6cd60e20f in do_append_data (jcr=jcr@entry=0x7f4f54106418, bs=bs@entry=0x55e6cee45918, what=what@entry=0x55e6cd623b28 "FD") at append.c:223
0000009 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f54106418) at fd_cmds.c:271
0000010 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f54106418) at fd_cmds.c:227
0000011 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f54106418) at fd_cmds.c:183
0000012 0x000055e6cd61678a in do_job_run (jcr=0x7f4f54106418) at job.c:238
0000013 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf05ae68) at dir_cmd.c:315
0000014 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf05ae68) at socket_server.c:100
0000015 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000016 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08d948) at lockmgr.c:928
0000017 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f457fa700) at pthread_create.c:456
#18 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 4 (Thread 0x7f4f897fa700 (LWP 1435)):
#0 0x00007f4f9261822d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f935991e7 in BSOCK_TCP::read_nbytes (this=0x55e6cf08d788, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f4f93598b25 in BSOCK_TCP::recv (this=0x55e6cf08d788) at bsock_tcp.c:550
0000003 0x00007f4f9358f36d in bget_msg (sock=sock@entry=0x55e6cf08d788) at bget_msg.c:53
0000004 0x000055e6cd60e0c5 in do_append_data (jcr=jcr@entry=0x7f4f301193c8, bs=bs@entry=0x55e6cf08d788, what=what@entry=0x55e6cd623b28 "FD") at append.c:158
0000005 0x000055e6cd6156a6 in append_data_cmd (jcr=0x7f4f301193c8) at fd_cmds.c:271
0000006 0x000055e6cd615afc in do_fd_commands (jcr=0x7f4f301193c8) at fd_cmds.c:227
0000007 0x000055e6cd615d08 in run_job (jcr=jcr@entry=0x7f4f301193c8) at fd_cmds.c:183
0000008 0x000055e6cd61678a in do_job_run (jcr=0x7f4f301193c8) at job.c:238
0000009 0x000055e6cd614e70 in handle_director_connection (dir=dir@entry=0x55e6cf071538) at dir_cmd.c:315
0000010 0x000055e6cd61db7c in handle_connection_request (arg=0x55e6cf071538) at socket_server.c:100
0000011 0x00007f4f935c4e65 in workq_server (arg=arg@entry=0x55e6cd82c540 <socket_workq>) at workq.c:336
0000012 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf08d788) at lockmgr.c:928
0000013 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f897fa700) at pthread_create.c:456
0000014 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7f4f8bfff700 (LWP 20146)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
0000001 0x00007f4f935abb2c in bthread_cond_timedwait_p (cond=cond@entry=0x7f4f937e1660 <_ZL5timer>, m=m@entry=0x7f4f937e16a0 <_ZL11timer_mutex>, abstime=abstime@entry=0x7f4f8bffee20, file=file@entry=0x7f4f935cf89a "watchdog.c", line=line@entry=313) at lockmgr.c:813
0000002 0x00007f4f935c448f in watchdog_thread (arg=arg@entry=0x0) at watchdog.c:313
0000003 0x00007f4f935ab68f in lmgr_thread_launcher (x=0x55e6cf05ac48) at lockmgr.c:928
0000004 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f8bfff700) at pthread_create.c:456
0000005 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7f4f9119f700 (LWP 20144)):
#0 0x00007f4f926187fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f9359b024 in bmicrosleep (sec=sec@entry=30, usec=usec@entry=0) at bsys.c:171
0000002 0x00007f4f935ab5fc in check_deadlock () at lockmgr.c:568
0000003 0x00007f4f9260f4a4 in start_thread (arg=0x7f4f9119f700) at pthread_create.c:456
0000004 0x00007f4f914a0d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7f4f940e5cc0 (LWP 20143)):
#0 0x00007f4f914978bd in poll () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4f9358ffe3 in bnet_thread_server_tcp (addr_list=<optimized out>, max_clients=<optimized out>, sockfds=<optimized out>, client_wq=<optimized out>, nokeepalive=<optimized out>, handle_client_request=<optimized out>) at bnet_server_tcp.c:306
0000002 0x000055e6cd60d88f in main (argc=<optimized out>, argv=<optimized out>) at stored.c:322
#0 0x00007f4f914978bd in poll () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
No locals.
0000001 0x00007f4f9358ffe3 in bnet_thread_server_tcp (addr_list=<optimized out>, max_clients=<optimized out>, sockfds=<optimized out>, client_wq=<optimized out>, nokeepalive=<optimized out>, handle_client_request=<optimized out>) at bnet_server_tcp.c:306
306 bnet_server_tcp.c: No such file or directory.
cnt = <optimized out>
newsockfd = <optimized out>
status = <optimized out>
clilen = 16
cli_addr = {sa_family = 2, sa_data = "\236\b\nGr\217\000\000\000\000\000\000\000"}
tlog = <optimized out>
tmax = <optimized out>
value = 1
request = {fd = 32, user = '\000' <repeats 127 times>, daemon = "backup7-de1-sd", '\000' <repeats 113 times>, pid = "20143\000\000\000\000", client = {{name = '\000' <repeats 127 times>, addr = '\000' <repeats 127 times>, sin = 0x7f4f924037e0, unit = 0x0, request = 0x7ffe330d2700}}, server = {{name = '\000' <repeats 127 times>, addr = '\000' <repeats 127 times>, sin = 0x7f4f92403760, unit = 0x0, request = 0x7ffe330d2700}}, sink = 0x0, hostname = 0x7f4f921ffb10 <sock_hostname>, hostaddr = 0x7f4f921ffad0 <sock_hostaddr>, cleanup = 0x0, config = 0x0}
ipaddr = 0x0
next = <optimized out>
to_free = <optimized out>
fd_ptr = <optimized out>
buf = "10.71.114.143", '\000' <repeats 11 times>, "\350\313\016\224O\177\000\000\000\000\000\000\000\000\000\000\005\000\000\000O\177\000\000\020\255\016\224O\177\000\000\000\066\r3\376\177\000\000@\003\000\000\000\000\000\000\177ELF\002\001\001\000\000\000\000\000\000\000\000\000\003\000>\000\001\000\000\000\320?\000\000\000\000\000\000@\000\000\000\000\000\000\000hT\001\000\000\000\000\000\000\000\000\000@\000\070"
nfds = <optimized out>
events = 195
pfds = <optimized out>
allbuf = "\376\002\000\000\000\000\000\000\070\220<\221O\177\000\000\270\243\016\224O\177\000\000\030+\r3\376\177\000\000\024+\r3\376\177\000\000\000\000\000\000\000\000\000\000\026\241<\221O\177\000\000`\005<\221O\177\000\000\030+\r3\376\177\000\000\363\026\346\365\000\000\000\000[\230\327\003\000\000\000\000\070\220<\221O\177\000\000\360+\r3\376\177\000\000\220\275;\221O\177\000\000\024+\r3\376\177\000\000\340+\r3\376\177\000\000\330v\016\224O\177\000\000\024\000\000\000\000\000\000\000\001", '\000' <repeats 15 times>, "\220\220\005\317", '\000' <repeats 12 times>, "P\220\005\317\346U\000\000\020\247\016\224O\177\000\000\200,\r3\376\177\000\000"...
0000002 0x000055e6cd60d88f in main (argc=<optimized out>, argv=<optimized out>) at stored.c:322
322 stored.c: No such file or directory.
ch = <optimized out>
no_signals = <optimized out>
test_config = false
export_config = false
export_config_schema = false
thid = 139979705149184
uid = 0x0
gid = 0x0
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
dh-lin

dh-lin

2019-05-26 09:50

reporter   ~0003383

Another one outage today:

????backup1-de1-sd: ABORTING due to ERROR in smartall.c:229
Overrun buffer: len=145 addr=7f9cdc006568 allocated: block.c:123 called from mem_pool.c:257
BAREOS interrupted by signal 11: Segmentation violation
dh-lin

dh-lin

2019-05-31 09:57

reporter   ~0003386

After some days on stable operations with bareos 18.2.6 got today again outage with signal 11 on one SD. Unfortunately no further information.

Dimitrij
dh-lin

dh-lin

2019-06-01 00:42

reporter   ~0003387

another one outage:
Created /var/lib/bareos/bareos-sd.core.14418 for doing postmortem debugging
[New LWP 14418]
[New LWP 14420]
[New LWP 19223]
[New LWP 19224]
[New LWP 19225]
[New LWP 19226]
[New LWP 19227]
[New LWP 19228]
[New LWP 19229]
[New LWP 19230]
[New LWP 19231]
[New LWP 19232]
[New LWP 19233]
[New LWP 19234]
[New LWP 19235]
[New LWP 19236]
[New LWP 19237]
[New LWP 19238]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/bareos-sd'.
#0 0x00007f3fa1b218bd in poll () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
[Current thread is 1 (Thread 0x7f3fa4609040 (LWP 14418))]
$1 = 1801675106
$2 = -1742040072
$3 = -1742040008
$4 = 0
$5 = -1549071688
$6 = -1549071606
$7 = -1549071586
$8 = -1549071576
$9 = 1801675106
$10 = -1549071656
Environment variable "TestName" not defined.
#0 0x00007f3fa1b218bd in poll () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f3fa3a45f46 in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(ConfigurationParser*, void*), ConfigurationParser*, std::atomic<BnetServerState>*) () from /usr/lib/bareos/libbareos.so.18
0000002 0x0000557f97f623b9 in storagedaemon::StartSocketServer(dlist*) ()
0000003 0x0000557f97f57256 in main ()

Thread 18 (Thread 0x7f3f5bfff700 (LWP 19238)):
#0 0x00007f3fa41b664b in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f5bfff700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 17 (Thread 0x7f3f78ff9700 (LWP 19237)):
#0 0x00007f3fa41b664e in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f78ff9700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 16 (Thread 0x7f3f797fa700 (LWP 19236)):
#0 0x00007f3fa30e622d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f3fa3a58abf in BareosSocketTCP::read_nbytes(char*, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f3fa3a57c52 in BareosSocketTCP::recv() () from /usr/lib/bareos/libbareos.so.18
0000003 0x00007f3fa3a44e57 in BgetMsg(BareosSocket*) () from /usr/lib/bareos/libbareos.so.18
0000004 0x0000557f97f71cb4 in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000005 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000006 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000007 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000008 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000009 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000010 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000011 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000012 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000013 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f797fa700) at pthread_create.c:456
0000014 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 15 (Thread 0x7f3f79ffb700 (LWP 19235)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
0000001 0x00007f3fa30dfbb5 in __GI___pthread_mutex_lock (mutex=0x7f3fa43f9880 <storagedaemon::mount_mutex>) at ../nptl/pthread_mutex_lock.c:80
0000002 0x00007f3fa3a6db7f in Lmgr_p(pthread_mutex_t*) () from /usr/lib/bareos/libbareos.so.18
0000003 0x00007f3fa41c0ed0 in storagedaemon::DeviceControlRecord::MountNextWriteVolume() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41bab5c in storagedaemon::FixupDeviceBlockWriteError(storagedaemon::DeviceControlRecord*, int) () from /usr/lib/bareos/libbareossd.so
0000005 0x00007f3fa41b1ad0 in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000006 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000007 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000008 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000009 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000010 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000011 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000012 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000013 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000014 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000015 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000016 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f79ffb700) at pthread_create.c:456
0000017 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 14 (Thread 0x7f3f7a7fc700 (LWP 19234)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
0000001 0x00007f3fa30dfbb5 in __GI___pthread_mutex_lock (mutex=0x7f3fa43f9880 <storagedaemon::mount_mutex>) at ../nptl/pthread_mutex_lock.c:80
0000002 0x00007f3fa3a6db7f in Lmgr_p(pthread_mutex_t*) () from /usr/lib/bareos/libbareos.so.18
0000003 0x00007f3fa41c0ed0 in storagedaemon::DeviceControlRecord::MountNextWriteVolume() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41bab5c in storagedaemon::FixupDeviceBlockWriteError(storagedaemon::DeviceControlRecord*, int) () from /usr/lib/bareos/libbareossd.so
0000005 0x00007f3fa41b1ad0 in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000006 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000007 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000008 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000009 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000010 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000011 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000012 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000013 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000014 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000015 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000016 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f7a7fc700) at pthread_create.c:456
0000017 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 13 (Thread 0x7f3f7affd700 (LWP 19233)):
#0 0x00007f3fa41b6668 in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f7affd700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 12 (Thread 0x7f3f7b7fe700 (LWP 19232)):
#0 0x00007f3fa41b664b in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f7b7fe700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 11 (Thread 0x7f3f7bfff700 (LWP 19231)):
#0 0x00007f3fa41b665e in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f7bfff700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 10 (Thread 0x7f3f98ff9700 (LWP 19230)):
#0 0x00007f3fa41b668b in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f98ff9700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 9 (Thread 0x7f3f997fa700 (LWP 19229)):
#0 0x00007f3fa30e622d in read () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f3fa3a58abf in BareosSocketTCP::read_nbytes(char*, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f3fa3a57c52 in BareosSocketTCP::recv() () from /usr/lib/bareos/libbareos.so.18
0000003 0x00007f3fa3a44e57 in BgetMsg(BareosSocket*) () from /usr/lib/bareos/libbareos.so.18
0000004 0x0000557f97f71cb4 in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000005 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000006 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000007 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000008 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000009 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000010 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000011 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000012 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000013 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f997fa700) at pthread_create.c:456
0000014 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 8 (Thread 0x7f3f99ffb700 (LWP 19228)):
#0 0x00007f3fa41b665e in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f99ffb700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 7 (Thread 0x7f3f9a7fc700 (LWP 19227)):
#0 0x00007f3fa41b668f in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f9a7fc700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 6 (Thread 0x7f3f9affd700 (LWP 19226)):
#0 0x00007f3fa30e6b5a in __waitpid (pid=20939, stat_loc=0x7f3f9affbf0c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
0000001 0x00007f3fa3a813a4 in SignalHandler () from /usr/lib/bareos/libbareos.so.18
0000002 <signal handler called>
0000003 0x00007f3fa41c7743 in storagedaemon::WriteRecordToBlock(storagedaemon::DeviceControlRecord*, storagedaemon::DeviceRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41bce92 in storagedaemon::WriteNewVolumeLabelToDev(storagedaemon::DeviceControlRecord*, char const*, char const*, bool) () from /usr/lib/bareos/libbareossd.so
0000005 0x00007f3fa41c382a in storagedaemon::DeviceControlRecord::TryAutolabel(bool) () from /usr/lib/bareos/libbareossd.so
0000006 0x00007f3fa41c1639 in storagedaemon::DeviceControlRecord::MountNextWriteVolume() () from /usr/lib/bareos/libbareossd.so
0000007 0x00007f3fa41bab5c in storagedaemon::FixupDeviceBlockWriteError(storagedaemon::DeviceControlRecord*, int) () from /usr/lib/bareos/libbareossd.so
0000008 0x00007f3fa41b1ad0 in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000009 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000010 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000011 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000012 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000013 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000014 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000015 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000016 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000017 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
#18 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
#19 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f9affd700) at pthread_create.c:456
0000020 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 5 (Thread 0x7f3f9b7fe700 (LWP 19225)):
#0 0x00007f3fa41b6634 in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f9b7fe700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 4 (Thread 0x7f3f9bfff700 (LWP 19224)):
#0 0x00007f3fa41b6668 in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3f9bfff700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7f3fa1638700 (LWP 19223)):
#0 0x00007f3fa41b66b2 in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f3fa41aeece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f3fa41b095f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f3fa41b1a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f3fa41c740f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000557f97f71e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000557f97f6989d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000557f97f695c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000557f97f69380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000557f97f5bf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000557f97f6777f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000557f97f62a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000557f97f62293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f3fa3a9483d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3fa1638700) at pthread_create.c:456
0000015 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7f3fa0e37700 (LWP 14420)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
0000001 0x00007f3fa3a93e92 in watchdog_thread () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f3fa30dd4a4 in start_thread (arg=0x7f3fa0e37700) at pthread_create.c:456
0000003 0x00007f3fa1b2ad0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7f3fa4609040 (LWP 14418)):
#0 0x00007f3fa1b218bd in poll () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f3fa3a45f46 in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(ConfigurationParser*, void*), ConfigurationParser*, std::atomic<BnetServerState>*) () from /usr/lib/bareos/libbareos.so.18
0000002 0x0000557f97f623b9 in storagedaemon::StartSocketServer(dlist*) ()
0000003 0x0000557f97f57256 in main ()
#0 0x00007f3fa1b218bd in poll () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
No locals.
0000001 0x00007f3fa3a45f46 in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(ConfigurationParser*, void*), ConfigurationParser*, std::atomic<BnetServerState>*) () from /usr/lib/bareos/libbareos.so.18
No symbol table info available.
0000002 0x0000557f97f623b9 in storagedaemon::StartSocketServer(dlist*) ()
No symbol table info available.
0000003 0x0000557f97f57256 in main ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
dh-lin

dh-lin

2019-06-05 08:39

reporter   ~0003389

Blocked again with last messages (debug):

backup2-de1-sd (100): stored/dev.cc:554-197582 created new block of buf_len: 1048576 on device "pl2_tape2_" (/var/lib/bareos/storage/tapes_ng/kafkapldi)
BAREOS interrupted by signal 11: Segmentation violation

Do you need any further information or debugging? Please don't hesitate to contact me. I'll support you to get this issue resolved soon.

Dimitrij
arogge

arogge

2019-07-09 11:47

manager   ~0003424

Did you customize "Maximum Block Size" or "Minimum block size" in your pool?

This is not intended to be set for File-based volumes and is known to lead to crashes during autolabeling sometimes.
dh-lin

dh-lin

2019-07-09 12:01

reporter   ~0003425

Indeed we customized "Maximum Block Size" for our file based pools as I saw some advice's to increase performance doing that.
I'll remove this configuration option, but it seems to be a bug if this option is allowed and do crash a sd deamon.
arogge

arogge

2019-07-10 17:05

manager   ~0003431

If you can confirm that going back to the original block sizes fixes your issue, we'll see what we can do so nobody else runs into that issue.
dh-lin

dh-lin

2019-07-11 10:58

reporter   ~0003440

Is it safe an enough just deactivate this option or do we need recreate all pools or update pools?
arogge

arogge

2019-07-11 14:40

manager   ~0003442

The safest way uses a Scratch pool and is described here: https://docs.bareos.org/TasksAndConcepts/AutochangerSupport.html#setblocksizes
dh-lin

dh-lin

2019-07-16 09:39

reporter   ~0003485

We reverted configuration like requested to the default values. Unfortunately today we got next outage:


Created /var/lib/bareos/bareos-sd.core.2634 for doing postmortem debugging
[New LWP 2634]
[New LWP 2645]
[New LWP 10282]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/bareos-sd'.
#0 0x00007f4ef13998bd in poll () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
[Current thread is 1 (Thread 0x7f4ef3e81040 (LWP 2634))]
$1 = 1801675106
$2 = 1861239800
$3 = 1861239864
$4 = 0
$5 = -214791496
$6 = -214791414
$7 = -214791394
$8 = -214791384
$9 = 1801675106
$10 = -214791464
Environment variable "TestName" not defined.
#0 0x00007f4ef13998bd in poll () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4ef32bdf46 in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(ConfigurationParser*, void*), ConfigurationParser*, std::atomic<BnetServerState>*) () from /usr/lib/bareos/libbareos.so.18
0000002 0x0000556b6ea233b9 in storagedaemon::StartSocketServer(dlist*) ()
0000003 0x0000556b6ea18256 in main ()

Thread 3 (Thread 0x7f4eccff9700 (LWP 10282)):
#0 0x00007f4ef295eb5a in __waitpid (pid=10315, stat_loc=0x7f4eccff840c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
0000001 0x00007f4ef32f93a4 in SignalHandler () from /usr/lib/bareos/libbareos.so.18
0000002 <signal handler called>
0000003 0x00007f4ef3a3888e in storagedaemon::_blockDevice(char const*, int, storagedaemon::Device*, int) () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f4ef3a1df9e in storagedaemon::AcquireDeviceForAppend(storagedaemon::DeviceControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000005 0x0000556b6ea326eb in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x0000556b6ea2a89d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x0000556b6ea2a5c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x0000556b6ea2a380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x0000556b6ea1cf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x0000556b6ea2877f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000556b6ea23a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000556b6ea23293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f4ef330c83d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f4ef29554a4 in start_thread (arg=0x7f4eccff9700) at pthread_create.c:456
0000015 0x00007f4ef13a2d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7f4ef06af700 (LWP 2645)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
0000001 0x00007f4ef330be92 in watchdog_thread () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f4ef29554a4 in start_thread (arg=0x7f4ef06af700) at pthread_create.c:456
0000003 0x00007f4ef13a2d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7f4ef3e81040 (LWP 2634)):
#0 0x00007f4ef13998bd in poll () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f4ef32bdf46 in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(ConfigurationParser*, void*), ConfigurationParser*, std::atomic<BnetServerState>*) () from /usr/lib/bareos/libbareos.so.18
0000002 0x0000556b6ea233b9 in storagedaemon::StartSocketServer(dlist*) ()
0000003 0x0000556b6ea18256 in main ()
#0 0x00007f4ef13998bd in poll () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
No locals.
0000001 0x00007f4ef32bdf46 in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(ConfigurationParser*, void*), ConfigurationParser*, std::atomic<BnetServerState>*) () from /usr/lib/bareos/libbareos.so.18
No symbol table info available.
0000002 0x0000556b6ea233b9 in storagedaemon::StartSocketServer(dlist*) ()
No symbol table info available.
0000003 0x0000556b6ea18256 in main ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
arogge

arogge

2019-07-16 10:19

manager   ~0003486

This is another crash happening at a completely different location.
I'll try to track down what happened. Which Bareos version were you running on that SD when it crashed last time?
dh-lin

dh-lin

2019-07-16 10:25

reporter   ~0003487

we run Version: 18.2.6 (13 February 2019) self compiled for Debian stretch (9) from source on github.
arogge_adm

arogge_adm

2019-07-16 10:39

administrator   ~0003488

I had a look at the code for 18.2.6.
A SIGSEGV in _blockDevice() can only happen in two cases:
1. another thread changed *dev while _blockDevice() accessed it (unlikely)
2. the ASSERT(dev->blocked() == BST_NOT_BLOCKED)

Can you take a look at the logs? If the assertion failed you should see something like "Failed ASSERT: dev->blocked() == BST_NOT_BLOCKED" somewhere in the logs. This would help to track down the issue.
dh-lin

dh-lin

2019-07-16 10:44

reporter   ~0003489

Next SD died:


Created /var/lib/bareos/bareos-sd.core.1797 for doing postmortem debugging
[New LWP 1797]
[New LWP 1808]
[New LWP 3353]
[New LWP 3354]
[New LWP 3355]
[New LWP 3356]
[New LWP 3357]
[New LWP 3358]
[New LWP 3360]
[New LWP 3361]
[New LWP 3362]
[New LWP 3363]
[New LWP 3364]
[New LWP 3368]
[New LWP 14672]
[New LWP 14674]
[New LWP 28272]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/bareos-sd'.
#0 0x00007f115e9068bd in poll () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
[Current thread is 1 (Thread 0x7f11613ef040 (LWP 1797))]
$1 = 1801675106
$2 = -1833802760
$3 = -1833802696
$4 = 0
$5 = 1619614392
$6 = 1619614474
$7 = 1619614494
$8 = 1619614504
$9 = 1801675106
$10 = 1619614424
Environment variable "TestName" not defined.
#0 0x00007f115e9068bd in poll () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116082af46 in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(ConfigurationParser*, void*), ConfigurationParser*, std::atomic<BnetServerState>*) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00005627924133b9 in storagedaemon::StartSocketServer(dlist*) ()
0000003 0x0000562792408256 in main ()

Thread 17 (Thread 0x7f114ffff700 (LWP 28272)):
#0 0x00007f115fecbb5a in __waitpid (pid=28281, stat_loc=0x7f114fffe40c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
0000001 0x00007f11608663a4 in SignalHandler () from /usr/lib/bareos/libbareos.so.18
0000002 <signal handler called>
0000003 0x00007f1160fa588e in storagedaemon::_blockDevice(char const*, int, storagedaemon::Device*, int) () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f1160f8af9e in storagedaemon::AcquireDeviceForAppend(storagedaemon::DeviceControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000005 0x00005627924226eb in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x000056279241a89d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x000056279241a5c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x000056279241a380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x000056279240cf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x000056279241877f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f115fec24a4 in start_thread (arg=0x7f114ffff700) at pthread_create.c:456
0000015 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 16 (Thread 0x7f114dffb700 (LWP 14674)):
#0 0x00007f1160282e7b in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
0000001 0x9585f12337162b8f in ?? ()
0000002 0xa38685d1b04fb72d in ?? ()
0000003 0x3328f86ae8908a55 in ?? ()
0000004 0xa9a994d0ff8abe44 in ?? ()
0000005 0x9d2beb1aa1c28a84 in ?? ()
0000006 0x3e0219465acd2d0e in ?? ()
0000007 0xb377aef78b787b4d in ?? ()
0000008 0x2b3146fc45d257a9 in ?? ()
0000009 0x00007f113819ce28 in ?? ()
0000010 0x0000000000000033 in ?? ()
0000011 0x00007f1138013f24 in ?? ()
0000012 0x00007f1160284eaf in SHA1_Update () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
0000013 0x00007f11602306e5 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
0000014 0x00007f1160231077 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
0000015 0x00007f116059207b in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
0000016 0x00007f1160592a5b in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
0000017 0x00007f1160590113 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#18 0x00007f1160595ff6 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#19 0x00007f116059fdb9 in SSL_read () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
0000020 0x00007f1160869f41 in TlsOpenSslPrivate::OpensslBsockReadwrite(BareosSocket*, char*, int, bool) () from /usr/lib/bareos/libbareos.so.18
0000021 0x00007f1160868faf in TlsOpenSsl::TlsBsockReadn(BareosSocket*, char*, int) () from /usr/lib/bareos/libbareos.so.18
0000022 0x00007f116083da84 in BareosSocketTCP::read_nbytes(char*, int) () from /usr/lib/bareos/libbareos.so.18
#23 0x00007f116083c98e in BareosSocketTCP::recv() () from /usr/lib/bareos/libbareos.so.18
0000024 0x00007f1160829e57 in BgetMsg(BareosSocket*) () from /usr/lib/bareos/libbareos.so.18
0000025 0x0000562792422cb4 in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000026 0x000056279241a89d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000027 0x000056279241a5c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000028 0x000056279241a380 in storagedaemon::RunJob(JobControlRecord*) ()
#29 0x000056279240cf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000030 0x000056279241877f in storagedaemon::RunCmd(JobControlRecord*) ()
0000031 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000032 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000033 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000034 0x00007f115fec24a4 in start_thread (arg=0x7f114dffb700) at pthread_create.c:456
0000035 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 15 (Thread 0x7f114effd700 (LWP 14672)):
#0 0x00007f1160f9b645 in bcrc32(unsigned char*, int) () from /usr/lib/bareos/libbareossd.so
0000001 0x00007f1160f93ece in storagedaemon::SerBlockHeader(storagedaemon::DeviceBlock*, bool) () from /usr/lib/bareos/libbareossd.so
0000002 0x00007f1160f9595f in storagedaemon::DeviceControlRecord::WriteBlockToDev() () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160f96a4c in storagedaemon::DeviceControlRecord::WriteBlockToDevice() () from /usr/lib/bareos/libbareossd.so
0000004 0x00007f1160fac40f in storagedaemon::DeviceControlRecord::WriteRecord() () from /usr/lib/bareos/libbareossd.so
0000005 0x0000562792422e2d in storagedaemon::DoAppendData(JobControlRecord*, BareosSocket*, char const*) ()
0000006 0x000056279241a89d in storagedaemon::AppendDataCmd(JobControlRecord*) ()
0000007 0x000056279241a5c2 in storagedaemon::DoFdCommands(JobControlRecord*) ()
0000008 0x000056279241a380 in storagedaemon::RunJob(JobControlRecord*) ()
0000009 0x000056279240cf32 in storagedaemon::DoJobRun(JobControlRecord*) ()
0000010 0x000056279241877f in storagedaemon::RunCmd(JobControlRecord*) ()
0000011 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000012 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000013 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000014 0x00007f115fec24a4 in start_thread (arg=0x7f114effd700) at pthread_create.c:456
0000015 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 14 (Thread 0x7f10d77fe700 (LWP 3368)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f10d77fe700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 13 (Thread 0x7f10edffb700 (LWP 3364)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f10edffb700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 12 (Thread 0x7f10eeffd700 (LWP 3363)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f10eeffd700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 11 (Thread 0x7f10ef7fe700 (LWP 3362)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f10ef7fe700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 10 (Thread 0x7f10effff700 (LWP 3361)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f10effff700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 9 (Thread 0x7f110cff9700 (LWP 3360)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f110cff9700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 8 (Thread 0x7f110dffb700 (LWP 3358)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f110dffb700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 7 (Thread 0x7f110e7fc700 (LWP 3357)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f110e7fc700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 6 (Thread 0x7f110effd700 (LWP 3356)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f110effd700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 5 (Thread 0x7f110f7fe700 (LWP 3355)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f110f7fe700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 4 (Thread 0x7f110ffff700 (LWP 3354)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f110ffff700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7f112cff9700 (LWP 3353)):
#0 0x00007f115fecb7fd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116083ff57 in Bmicrosleep(int, int) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f1160fae338 in storagedaemon::UseDeviceCmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000003 0x00007f1160fad5f2 in storagedaemon::use_cmd(JobControlRecord*) () from /usr/lib/bareos/libbareossd.so
0000004 0x0000562792413a79 in storagedaemon::HandleDirectorConnection(BareosSocket*) ()
0000005 0x0000562792413293 in storagedaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
0000006 0x00007f116087983d in workq_server () from /usr/lib/bareos/libbareos.so.18
0000007 0x00007f115fec24a4 in start_thread (arg=0x7f112cff9700) at pthread_create.c:456
0000008 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7f115dc1c700 (LWP 1808)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
0000001 0x00007f1160878e92 in watchdog_thread () from /usr/lib/bareos/libbareos.so.18
0000002 0x00007f115fec24a4 in start_thread (arg=0x7f115dc1c700) at pthread_create.c:456
0000003 0x00007f115e90fd0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7f11613ef040 (LWP 1797)):
#0 0x00007f115e9068bd in poll () at ../sysdeps/unix/syscall-template.S:84
0000001 0x00007f116082af46 in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(ConfigurationParser*, void*), ConfigurationParser*, std::atomic<BnetServerState>*) () from /usr/lib/bareos/libbareos.so.18
0000002 0x00005627924133b9 in storagedaemon::StartSocketServer(dlist*) ()
0000003 0x0000562792408256 in main ()
#0 0x00007f115e9068bd in poll () at ../sysdeps/unix/syscall-template.S:84
84 in ../sysdeps/unix/syscall-template.S
No locals.
0000001 0x00007f116082af46 in BnetThreadServerTcp(dlist*, int, alist*, workq_tag*, bool, void* (*)(ConfigurationParser*, void*), ConfigurationParser*, std::atomic<BnetServerState>*) () from /usr/lib/bareos/libbareos.so.18
No symbol table info available.
0000002 0x00005627924133b9 in storagedaemon::StartSocketServer(dlist*) ()
No symbol table info available.
0000003 0x0000562792408256 in main ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
dh-lin

dh-lin

2019-07-16 10:48

reporter   ~0003490

In log (from first reported outage today) we only see:
BAREOS interrupted by signal 11: Segmentation violation

I think it will not help a lot.

Dimitrij
dh-lin

dh-lin

2019-07-16 10:53

reporter   ~0003491

I have an idea. On both cases backup was running on SD and also regular volume truncation (we have python based cronjob to truncate unused volumes and spare HDD space).
Probably it will help you to investigate.

Dimitrij
arogge_adm

arogge_adm

2019-07-16 10:55

administrator   ~0003492

That crashed at the exact same location as the last time. It would really help if I knew whether the ASSERT was triggered or an unintended nullptr dereference happened.
dh-lin

dh-lin

2019-07-16 10:59

reporter   ~0003493

Let me know what can I do to help you to investigate. Keep in mind it crashes randomly so I'll be not able to trigger.
also please review my last post. It was on bot SD today: Lot of backups and truncate:
prune volume=$volname yes
truncate volstatus=Purged pool=$pollname yes

it iterates over all pools and volumes in catalog.

Dimitrij
dh-lin

dh-lin

2019-07-22 09:08

reporter   ~0003510

Since we removed "Maximum Block Size" option from all pools we don't experience outages from SD. Also we migrate volume management (prune and truncate) to time with less backup jobs.
arogge

arogge

2019-07-22 09:27

manager   ~0003511

If your problem is solved, I'd like to close the issue. Is that OK with you?
dh-lin

dh-lin

2019-07-22 09:58

reporter   ~0003512

Sure, you can close.
thank you for pointing out what a problem is. I hope nobody else will hit it.

Dimitrij

Issue History

Date Modified Username Field Change
2019-05-16 09:15 dh-lin New Issue
2019-05-16 09:28 dh-lin Note Added: 0003374
2019-05-16 11:03 dh-lin Note Added: 0003375
2019-05-21 12:04 nwhisper Note Added: 0003378
2019-05-23 02:22 dh-lin Note Added: 0003379
2019-05-26 09:50 dh-lin Note Added: 0003383
2019-05-31 09:57 dh-lin Note Added: 0003386
2019-06-01 00:42 dh-lin Note Added: 0003387
2019-06-05 08:39 dh-lin Note Added: 0003389
2019-07-09 11:47 arogge Status new => feedback
2019-07-09 11:47 arogge Note Added: 0003424
2019-07-09 12:01 dh-lin Note Added: 0003425
2019-07-09 12:01 dh-lin Status feedback => new
2019-07-10 17:05 arogge Assigned To => arogge
2019-07-10 17:05 arogge Status new => feedback
2019-07-10 17:05 arogge Note Added: 0003431
2019-07-10 17:44 arogge Assigned To arogge =>
2019-07-11 10:58 dh-lin Note Added: 0003440
2019-07-11 10:58 dh-lin Status feedback => new
2019-07-11 14:40 arogge Note Added: 0003442
2019-07-11 14:40 arogge Assigned To => arogge
2019-07-11 14:40 arogge Status new => feedback
2019-07-16 09:39 dh-lin Note Added: 0003485
2019-07-16 09:39 dh-lin Status feedback => assigned
2019-07-16 10:19 arogge Note Added: 0003486
2019-07-16 10:25 dh-lin Note Added: 0003487
2019-07-16 10:39 arogge_adm Status assigned => feedback
2019-07-16 10:39 arogge_adm Note Added: 0003488
2019-07-16 10:44 dh-lin Note Added: 0003489
2019-07-16 10:44 dh-lin Status feedback => assigned
2019-07-16 10:48 dh-lin Note Added: 0003490
2019-07-16 10:53 dh-lin Note Added: 0003491
2019-07-16 10:55 arogge_adm Note Added: 0003492
2019-07-16 10:59 dh-lin Note Added: 0003493
2019-07-22 09:08 dh-lin Note Added: 0003510
2019-07-22 09:27 arogge Assigned To arogge =>
2019-07-22 09:27 arogge Status assigned => feedback
2019-07-22 09:27 arogge Note Added: 0003511
2019-07-22 09:58 dh-lin Note Added: 0003512
2019-07-22 09:58 dh-lin Status feedback => new
2019-07-22 10:10 arogge Assigned To => arogge
2019-07-22 10:10 arogge Status new => resolved
2019-07-22 10:10 arogge Resolution open => no change required
2019-07-22 10:10 arogge Status resolved => closed