View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001029bareos-coredirectorpublic2018-11-26 17:072019-12-18 15:40
Reporterfrank Assigned Toarogge  
PrioritynormalSeveritymajorReproducibilityhave not tried
Status closedResolutionunable to reproduce 
PlatformLinuxOSDebianOS Version8
Product Version17.2.7 
Summary0001029: Deadlock on DIRD
DescriptionEncountered a situation where there were 12 threads waiting for a wait-condition that could not be signaled due to a deadlock.
Steps To Reproducen/a
Additional Informationbackup02:/var/log/bareos# pidof bareos-dir
10412
backup02:/var/log/bareos# gdb -p 10412
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 10412
Reading symbols from /usr/sbin/bareos-dir...Reading symbols from /usr/lib/debug//usr/sbin/bareos-dir...done.
done.
Reading symbols from /usr/lib/bareos/libbareosndmp-17.2.7.so...Reading symbols from /usr/lib/debug//usr/lib/bareos/libbareosndmp-17.2.7.so...done.
done.
Loaded symbols for /usr/lib/bareos/libbareosndmp-17.2.7.so
Reading symbols from /usr/lib/bareos/libbareosfind-17.2.7.so...Reading symbols from /usr/lib/debug//usr/lib/bareos/libbareosfind-17.2.7.so...done.
done.
Loaded symbols for /usr/lib/bareos/libbareosfind-17.2.7.so
Reading symbols from /lib/x86_64-linux-gnu/libacl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libacl.so.1
Reading symbols from /usr/lib/bareos/libbareossql-17.2.7.so...Reading symbols from /usr/lib/debug//usr/lib/bareos/libbareossql-17.2.7.so...done.
done.
Loaded symbols for /usr/lib/bareos/libbareossql-17.2.7.so
Reading symbols from /usr/lib/bareos/libbareoscats-17.2.7.so...Reading symbols from /usr/lib/debug//usr/lib/bareos/libbareoscats-17.2.7.so...done.
done.
Loaded symbols for /usr/lib/bareos/libbareoscats-17.2.7.so
Reading symbols from /usr/lib/bareos/libbareoscfg-17.2.7.so...Reading symbols from /usr/lib/debug//usr/lib/bareos/libbareoscfg-17.2.7.so...done.
done.
Loaded symbols for /usr/lib/bareos/libbareoscfg-17.2.7.so
Reading symbols from /usr/lib/bareos/libbareos-17.2.7.so...Reading symbols from /usr/lib/debug//usr/lib/bareos/libbareos-17.2.7.so...done.
done.
Loaded symbols for /usr/lib/bareos/libbareos-17.2.7.so
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /lib/x86_64-linux-gnu/liblzo2.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/liblzo2.so.2
Reading symbols from /usr/lib/libfastlz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libfastlz.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libjansson.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libjansson.so.4
Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /lib/x86_64-linux-gnu/libwrap.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libwrap.so.0
Reading symbols from /lib/x86_64-linux-gnu/libcap.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcap.so.2
Reading symbols from /usr/lib/bareos/libbareoslmdb-17.2.7.so...Reading symbols from /usr/lib/debug//usr/lib/bareos/libbareoslmdb-17.2.7.so...done.
done.
Loaded symbols for /usr/lib/bareos/libbareoslmdb-17.2.7.so
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libpthread-2.19.so...done.
done.
[New LWP 19389]
[New LWP 9093]
[New LWP 6509]
[New LWP 1955]
[New LWP 2796]
[New LWP 30244]
[New LWP 29643]
[New LWP 26418]
[New LWP 22171]
[New LWP 28589]
[New LWP 29068]
[New LWP 29031]
[New LWP 9474]
[New LWP 9458]
[New LWP 10417]
[New LWP 10416]
[New LWP 10413]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib/x86_64-linux-gnu/libattr.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libattr.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnsl-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/bareos/backends/libbareoscats-postgresql.so...Reading symbols from /usr/lib/debug//usr/lib/bareos/backends/libbareoscats-postgresql-17.2.7.so...done.
done.
Loaded symbols for /usr/lib/bareos/backends/libbareoscats-postgresql.so
Reading symbols from /usr/lib/x86_64-linux-gnu/libpq.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libpq.so.5
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5.so.3
Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcom_err.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libkeyutils.so.1
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libresolv-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libresolv.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libsasl2.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28
Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libtasn1.so.6
Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libnettle.so.4
Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libhogweed.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgmp.so.10
Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libffi.so.6
0x00007f8dcf92014d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) info thread
  Id Target Id Frame
  18 Thread 0x7f8dce722700 (LWP 10413) "bareos-dir" 0x00007f8dcf92014d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
  17 Thread 0x7f8dc71df700 (LWP 10416) "bareos-dir" 0x00007f8dcec1faed in poll () at ../sysdeps/unix/syscall-template.S:81
  16 Thread 0x7f8dc69de700 (LWP 10417) "bareos-dir" 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
  15 Thread 0x7f8daffff700 (LWP 9458) "bareos-dir" 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
  14 Thread 0x7f8daf7fe700 (LWP 9474) "bareos-dir" 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
  13 Thread 0x7f8daeffd700 (LWP 29031) "bareos-dir" 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
  12 Thread 0x7f8dc51db700 (LWP 29068) "bareos-dir" 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
  11 Thread 0x7f8dc61dd700 (LWP 28589) "bareos-dir" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  10 Thread 0x7f8dc59dc700 (LWP 22171) "bareos-dir" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  9 Thread 0x7f8dc49da700 (LWP 26418) "bareos-dir" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  8 Thread 0x7f8dae7fc700 (LWP 29643) "bareos-dir" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  7 Thread 0x7f8dadffb700 (LWP 30244) "bareos-dir" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  6 Thread 0x7f8dad7fa700 (LWP 2796) "bareos-dir" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  5 Thread 0x7f8dacff9700 (LWP 1955) "bareos-dir" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  4 Thread 0x7f8d8bfff700 (LWP 6509) "bareos-dir" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  3 Thread 0x7f8d8b7fe700 (LWP 9093) "bareos-dir" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  2 Thread 0x7f8d8affd700 (LWP 19389) "bareos-dir" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 1 Thread 0x7f8dd2168740 (LWP 10412) "bareos-dir" 0x00007f8dcf92014d in nanosleep () at ../sysdeps/unix/syscall-template.S:81

(gdb) thread 10
[Switching to thread 10 (Thread 0x7f8dc59dc700 (LWP 22171))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8da804f5b8) at watchdog.c:194
0000004 0x00007f8dd10308b7 in start_thread_timer (jcr=0x1c75308, tid=140246882567936, wait=1800) at btimers.c:148
0000005 0x00007f8dd102be6c in BSOCK_TCP::connect (this=0x7f8da80e9dd8, jcr=0x1c75308, retry_interval=10, max_retry_time=140247049556047, max_retry_time@entry=1800, heart_beat=140247076130304, heart_beat@entry=0,
    name=0x47c6e1 "Storage daemon", host=0x7f8d940b9d68 "backup02.bni", service=0x0, port=9103, verbose=true) at bsock_tcp.c:112
0000006 0x00000000004452ce in connect_to_storage_daemon (jcr=0x1c75308, retry_interval=<optimized out>, max_retry_time=<optimized out>, verbose=verbose@entry=true) at sd_cmds.c:118
0000007 0x00000000004453a8 in connect_to_storage_daemon (jcr=jcr@entry=0x1c75308, retry_interval=retry_interval@entry=10, max_retry_time=<optimized out>, verbose=verbose@entry=true) at sd_cmds.c:136
0000008 0x000000000041442a in do_native_backup (jcr=jcr@entry=0x1c75308) at backup.c:434
0000009 0x0000000000429654 in job_thread (arg=0x1c75308) at job.c:514
0000010 0x000000000042eb71 in jobq_server (arg=0x6b7240 <job_queue>) at jobq.c:485
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x1bce9f8) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8dc59dc700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 11
[Switching to thread 11 (Thread 0x7f8dc61dd700 (LWP 28589))]
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
238 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: No such file or directory.
(gdb) bt
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
0000001 0x00007f8dd103ec9c in bthread_cond_timedwait_p (cond=0x6b7860 <wait_for_next_run_cond>, m=0x6b78a0 <_ZL5mutex>, abstime=0x7f8dc61dcd60, file=0x48ddca "stats.c", line=120) at lockmgr.c:813
0000002 0x0000000000447f2e in wait_for_next_run () at stats.c:120
0000003 statistics_thread (arg=0x6b7864 <wait_for_next_run_cond+4>, arg@entry=0x0) at stats.c:332
0000004 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8d94208158) at lockmgr.c:928
0000005 0x00007f8dcf919064 in start_thread (arg=0x7f8dc61dd700) at pthread_create.c:309
0000006 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f8dc49da700 (LWP 26418))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8db00a57a8) at watchdog.c:194
0000004 0x00007f8dd10308b7 in start_thread_timer (jcr=0x1bb3d68, tid=140246865782528, wait=1800) at btimers.c:148
0000005 0x00007f8dd102be6c in BSOCK_TCP::connect (this=0x7f8db0062368, jcr=0x1bb3d68, retry_interval=10, max_retry_time=140247049556047, max_retry_time@entry=1800, heart_beat=140247076130304, heart_beat@entry=0,
    name=0x47c6e1 "Storage daemon", host=0x7f8d940b9d68 "backup02.bni", service=0x0, port=9103, verbose=true) at bsock_tcp.c:112
0000006 0x00000000004452ce in connect_to_storage_daemon (jcr=0x1bb3d68, retry_interval=<optimized out>, max_retry_time=<optimized out>, verbose=verbose@entry=true) at sd_cmds.c:118
0000007 0x00000000004453a8 in connect_to_storage_daemon (jcr=jcr@entry=0x1bb3d68, retry_interval=retry_interval@entry=10, max_retry_time=<optimized out>, verbose=verbose@entry=true) at sd_cmds.c:136
0000008 0x000000000041442a in do_native_backup (jcr=jcr@entry=0x1bb3d68) at backup.c:434
0000009 0x0000000000429654 in job_thread (arg=0x1bb3d68) at job.c:514
0000010 0x000000000042eb71 in jobq_server (arg=0x6b7240 <job_queue>) at jobq.c:485
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x1b9f498) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8dc49da700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 8
[Switching to thread 8 (Thread 0x7f8dae7fc700 (LWP 29643))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8d94007308) at watchdog.c:194
0000004 0x00007f8dd10309c4 in start_bsock_timer (bsock=0x7f8dc0001168, wait=600) at btimers.c:184
0000005 0x00007f8dd102a4f0 in BSOCK::two_way_authenticate (this=0x7f8dc0001168, jcr=0x0, what=0x47c76e "Console", name=0x47c762 "*UserAgent*", password=..., tls=..., initiated_by_remote=true) at bsock.c:340
0000006 0x000000000041118a in authenticate_inbound_connection (tls=..., password=..., name=<optimized out>, what=<optimized out>, jcr=<optimized out>, this=<optimized out>) at ../lib/bsock.h:151
0000007 authenticate_user_agent (uac=0x7f8d9403dfa8) at authenticate.c:262
0000008 0x000000000046f3fe in handle_UA_client_request (user=user@entry=0x7f8dc0001168) at ua_server.c:84
0000009 0x00000000004421d3 in handle_connection_request (arg=0x7f8dc0001168) at socket_server.c:98
0000010 0x00007f8dd1057f15 in workq_server (arg=arg@entry=0x6b7540 <socket_workq>) at workq.c:336
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8dc0001378) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8dae7fc700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 7
[Switching to thread 7 (Thread 0x7f8dadffb700 (LWP 30244))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8da48f0828) at watchdog.c:194
0000004 0x00007f8dd10309c4 in start_bsock_timer (bsock=0x7f8dc0001378, wait=600) at btimers.c:184
0000005 0x00007f8dd102a4f0 in BSOCK::two_way_authenticate (this=0x7f8dc0001378, jcr=0x0, what=0x47c76e "Console", name=0x47c762 "*UserAgent*", password=..., tls=..., initiated_by_remote=true) at bsock.c:340
0000006 0x000000000041118a in authenticate_inbound_connection (tls=..., password=..., name=<optimized out>, what=<optimized out>, jcr=<optimized out>, this=<optimized out>) at ../lib/bsock.h:151
0000007 authenticate_user_agent (uac=0x7f8da4118768) at authenticate.c:262
0000008 0x000000000046f3fe in handle_UA_client_request (user=user@entry=0x7f8dc0001378) at ua_server.c:84
0000009 0x00000000004421d3 in handle_connection_request (arg=0x7f8dc0001378) at socket_server.c:98
0000010 0x00007f8dd1057f15 in workq_server (arg=arg@entry=0x6b7540 <socket_workq>) at workq.c:336
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8dc00017d8) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8dadffb700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 6
[Switching to thread 6 (Thread 0x7f8dad7fa700 (LWP 2796))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8d840262f8) at watchdog.c:194
0000004 0x00007f8dd10309c4 in start_bsock_timer (bsock=0x7f8dc00017d8, wait=600) at btimers.c:184
0000005 0x00007f8dd102a4f0 in BSOCK::two_way_authenticate (this=0x7f8dc00017d8, jcr=0x0, what=0x47c76e "Console", name=0x7f8dad7f9c30 "thomas.fachtan", password=..., tls=..., initiated_by_remote=true) at bsock.c:340
0000006 0x000000000041121a in authenticate_inbound_connection (tls=..., password=..., name=<optimized out>, what=<optimized out>, jcr=<optimized out>, this=<optimized out>) at ../lib/bsock.h:151
0000007 authenticate_user_agent (uac=0x7f8d840249f8) at authenticate.c:269
0000008 0x000000000046f3fe in handle_UA_client_request (user=user@entry=0x7f8dc00017d8) at ua_server.c:84
0000009 0x00000000004421d3 in handle_connection_request (arg=0x7f8dc00017d8) at socket_server.c:98
0000010 0x00007f8dd1057f15 in workq_server (arg=arg@entry=0x6b7540 <socket_workq>) at workq.c:336
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8dc0002a38) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8dad7fa700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 5
[Switching to thread 5 (Thread 0x7f8dacff9700 (LWP 1955))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8d7c003a78) at watchdog.c:194
0000004 0x00007f8dd10309c4 in start_bsock_timer (bsock=0x7f8dc0002bc8, wait=600) at btimers.c:184
0000005 0x00007f8dd102a4f0 in BSOCK::two_way_authenticate (this=0x7f8dc0002bc8, jcr=0x0, what=0x47c76e "Console", name=0x47c762 "*UserAgent*", password=..., tls=..., initiated_by_remote=true) at bsock.c:340
0000006 0x000000000041118a in authenticate_inbound_connection (tls=..., password=..., name=<optimized out>, what=<optimized out>, jcr=<optimized out>, this=<optimized out>) at ../lib/bsock.h:151
0000007 authenticate_user_agent (uac=0x7f8d7c002cd8) at authenticate.c:262
0000008 0x000000000046f3fe in handle_UA_client_request (user=user@entry=0x7f8dc0002bc8) at ua_server.c:84
0000009 0x00000000004421d3 in handle_connection_request (arg=0x7f8dc0002bc8) at socket_server.c:98
0000010 0x00007f8dd1057f15 in workq_server (arg=arg@entry=0x6b7540 <socket_workq>) at workq.c:336
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8dc0003de8) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8dacff9700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 4
[Switching to thread 4 (Thread 0x7f8d8bfff700 (LWP 6509))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8d8c00b4e8) at watchdog.c:194
0000004 0x00007f8dd10309c4 in start_bsock_timer (bsock=0x7f8dc0003f78, wait=600) at btimers.c:184
0000005 0x00007f8dd102a4f0 in BSOCK::two_way_authenticate (this=0x7f8dc0003f78, jcr=0x0, what=0x47c76e "Console", name=0x47c762 "*UserAgent*", password=..., tls=..., initiated_by_remote=true) at bsock.c:340
0000006 0x000000000041118a in authenticate_inbound_connection (tls=..., password=..., name=<optimized out>, what=<optimized out>, jcr=<optimized out>, this=<optimized out>) at ../lib/bsock.h:151
0000007 authenticate_user_agent (uac=0x7f8d8c002708) at authenticate.c:262
0000008 0x000000000046f3fe in handle_UA_client_request (user=user@entry=0x7f8dc0003f78) at ua_server.c:84
0000009 0x00000000004421d3 in handle_connection_request (arg=0x7f8dc0003f78) at socket_server.c:98
0000010 0x00007f8dd1057f15 in workq_server (arg=arg@entry=0x6b7540 <socket_workq>) at workq.c:336
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8dc0005198) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8d8bfff700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 3
[Switching to thread 3 (Thread 0x7f8d8b7fe700 (LWP 9093))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8da00f6108) at watchdog.c:194
0000004 0x00007f8dd10309c4 in start_bsock_timer (bsock=0x7f8dc0005328, wait=600) at btimers.c:184
0000005 0x00007f8dd102a4f0 in BSOCK::two_way_authenticate (this=0x7f8dc0005328, jcr=0x0, what=0x47c76e "Console", name=0x47c762 "*UserAgent*", password=..., tls=..., initiated_by_remote=true) at bsock.c:340
0000006 0x000000000041118a in authenticate_inbound_connection (tls=..., password=..., name=<optimized out>, what=<optimized out>, jcr=<optimized out>, this=<optimized out>) at ../lib/bsock.h:151
0000007 authenticate_user_agent (uac=0x7f8da001d958) at authenticate.c:262
0000008 0x000000000046f3fe in handle_UA_client_request (user=user@entry=0x7f8dc0005328) at ua_server.c:84
0000009 0x00000000004421d3 in handle_connection_request (arg=0x7f8dc0005328) at socket_server.c:98
0000010 0x00007f8dd1057f15 in workq_server (arg=arg@entry=0x6b7540 <socket_workq>) at workq.c:336
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8dc0006548) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8d8b7fe700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

(gdb) thread 2
[Switching to thread 2 (Thread 0x7f8d8affd700 (LWP 19389))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8d80003b88) at watchdog.c:194
0000004 0x00007f8dd10309c4 in start_bsock_timer (bsock=0x7f8dc00066d8, wait=600) at btimers.c:184
0000005 0x00007f8dd102a4f0 in BSOCK::two_way_authenticate (this=0x7f8dc00066d8, jcr=0x0, what=0x47c76e "Console", name=0x47c762 "*UserAgent*", password=..., tls=..., initiated_by_remote=true) at bsock.c:340
0000006 0x000000000041118a in authenticate_inbound_connection (tls=..., password=..., name=<optimized out>, what=<optimized out>, jcr=<optimized out>, this=<optimized out>) at ../lib/bsock.h:151
0000007 authenticate_user_agent (uac=0x7f8d80002de8) at authenticate.c:262
0000008 0x000000000046f3fe in handle_UA_client_request (user=user@entry=0x7f8dc00066d8) at ua_server.c:84
0000009 0x00000000004421d3 in handle_connection_request (arg=0x7f8dc00066d8) at socket_server.c:98
0000010 0x00007f8dd1057f15 in workq_server (arg=arg@entry=0x6b7540 <socket_workq>) at workq.c:336
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8dc0007b48) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8d8affd700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

(gdb) thread 1
[Switching to thread 1 (Thread 0x7f8dd2168740 (LWP 10412))]
#0 0x00007f8dcf92014d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007f8dcf92014d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
0000001 0x00007f8dd102d8f4 in bmicrosleep (sec=60, usec=0) at bsys.c:171
0000002 0x000000000044718f in wait_for_next_job (one_shot_job_to_run=0x5bf290e4 <error: Cannot access memory at address 0x5bf290e4>) at scheduler.c:173
0000003 0x000000000040f976 in main (argc=<optimized out>, argv=<optimized out>) at dird.c:437
(gdb)


(gdb) thread 12
[Switching to thread 12 (Thread 0x7f8dc51db700 (LWP 29068))]
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
0000001 0x00007f8dd102bb00 in BSOCK_TCP::read_nbytes (this=0x7f8d9c002188, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f8dd102b37f in BSOCK_TCP::recv (this=0x7f8d9c002188) at bsock_tcp.c:550
0000003 0x0000000000426b9f in bget_dirmsg (bs=0x7f8d9c002188, allow_any_message=false) at getmsg.c:154
0000004 0x0000000000433f6c in msg_thread (arg=0xb, arg@entry=0x1bb30d8) at msgchan.c:434
0000005 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8d9c001138) at lockmgr.c:928
0000006 0x00007f8dcf919064 in start_thread (arg=0x7f8dc51db700) at pthread_create.c:309
0000007 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 13
[Switching to thread 13 (Thread 0x7f8daeffd700 (LWP 29031))]
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
81 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
0000001 0x00007f8dd102bb00 in BSOCK_TCP::read_nbytes (this=0x7f8d9c058e08, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f8dd102b37f in BSOCK_TCP::recv (this=0x7f8d9c058e08) at bsock_tcp.c:550
0000003 0x0000000000426b9f in bget_dirmsg (bs=0x7f8d9c058e08, allow_any_message=false) at getmsg.c:154
0000004 0x0000000000412632 in wait_for_job_termination (jcr=0x1bb30d8, timeout=0) at backup.c:715
0000005 0x0000000000414799 in do_native_backup (jcr=jcr@entry=0x1bb30d8) at backup.c:650
0000006 0x0000000000429654 in job_thread (arg=0x1bb30d8) at job.c:514
0000007 0x000000000042eb71 in jobq_server (arg=0x6b7240 <job_queue>) at jobq.c:485
0000008 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x1d45a28) at lockmgr.c:928
0000009 0x00007f8dcf919064 in start_thread (arg=0x7f8daeffd700) at pthread_create.c:309
0000010 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

(gdb) thread 14
[Switching to thread 14 (Thread 0x7f8daf7fe700 (LWP 9474))]
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
81 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
0000001 0x00007f8dd102bb00 in BSOCK_TCP::read_nbytes (this=0x7f8d901754c8, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f8dd102b37f in BSOCK_TCP::recv (this=0x7f8d901754c8) at bsock_tcp.c:550
0000003 0x0000000000426b9f in bget_dirmsg (bs=0x7f8d901754c8, allow_any_message=false) at getmsg.c:154
0000004 0x0000000000433f6c in msg_thread (arg=0x8, arg@entry=0x1c73708) at msgchan.c:434
0000005 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8d901824d8) at lockmgr.c:928
0000006 0x00007f8dcf919064 in start_thread (arg=0x7f8daf7fe700) at pthread_create.c:309
0000007 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) thread 15
[Switching to thread 15 (Thread 0x7f8daffff700 (LWP 9458))]
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
81 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
0000001 0x00007f8dd102bb00 in BSOCK_TCP::read_nbytes (this=0x7f8d900f2188, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f8dd102b37f in BSOCK_TCP::recv (this=0x7f8d900f2188) at bsock_tcp.c:550
0000003 0x0000000000426b9f in bget_dirmsg (bs=0x7f8d900f2188, allow_any_message=false) at getmsg.c:154
0000004 0x0000000000412632 in wait_for_job_termination (jcr=0x1c73708, timeout=0) at backup.c:715
0000005 0x0000000000414799 in do_native_backup (jcr=jcr@entry=0x1c73708) at backup.c:650
0000006 0x0000000000429654 in job_thread (arg=0x1c73708) at job.c:514
0000007 0x000000000042eb71 in jobq_server (arg=0x6b7240 <job_queue>) at jobq.c:485
0000008 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x1bcabd8) at lockmgr.c:928
0000009 0x00007f8dcf919064 in start_thread (arg=0x7f8daffff700) at pthread_create.c:309
0000010 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

(gdb) thread 16
[Switching to thread 16 (Thread 0x7f8dc69de700 (LWP 10417))]
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
81 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x00007f8dcf91fa9d in read () at ../sysdeps/unix/syscall-template.S:81
0000001 0x00007f8dd102bb00 in BSOCK_TCP::read_nbytes (this=0x7f8db8001838, ptr=<optimized out>, nbytes=4) at bsock_tcp.c:978
0000002 0x00007f8dd102b37f in BSOCK_TCP::recv (this=0x7f8db8001838) at bsock_tcp.c:550
0000003 0x00007f8dd1033294 in cram_md5_respond (bs=0x7f8db8001838, password=0x7f8da8144e08 "469bf773b1ee39b955b8636641a55856", tls_remote_need=0x7f8dc69ddb0c, compatible=0x7f8dc69ddb0b) at cram-md5.c:118
0000004 0x00007f8dd102a695 in BSOCK::two_way_authenticate (this=0x7f8db8001838, jcr=0x1d46f18, what=0x47c705 "File Daemon", name=0x7f8da8146598 "domino03.bni", password=..., tls=..., initiated_by_remote=false) at bsock.c:365
0000005 0x0000000000410c7a in authenticate_outbound_connection (tls=..., password=..., name=<optimized out>, what=<optimized out>, jcr=<optimized out>, this=<optimized out>) at ../lib/bsock.h:146
0000006 authenticate_with_file_daemon (jcr=0xe, jcr@entry=0x1d46f18) at authenticate.c:148
0000007 0x00000000004247e0 in connect_to_file_daemon (jcr=0x1d46f18, retry_interval=retry_interval@entry=10, max_retry_time=<optimized out>, verbose=verbose@entry=true) at fd_cmds.c:183
0000008 0x00000000004265a3 in cancel_file_daemon_job (ua=ua@entry=0x7f8db8001148, jcr=jcr@entry=0x1c73708) at fd_cmds.c:1065
0000009 0x0000000000429eff in cancel_job (ua=0x7f8db8001148, jcr=0x1c73708) at job.c:697
0000010 0x000000000042a225 in job_monitor_watchdog (self=0xe) at job.c:773
0000011 0x00007f8dd1057497 in watchdog_thread (arg=arg@entry=0x0) at watchdog.c:282
0000012 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x1d50498) at lockmgr.c:928
0000013 0x00007f8dcf919064 in start_thread (arg=0x7f8dc69de700) at pthread_create.c:309
0000014 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

(gdb) thread 17
[Switching to thread 17 (Thread 0x7f8dc71df700 (LWP 10416))]
#0 0x00007f8dcec1faed in poll () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007f8dcec1faed in poll () at ../sysdeps/unix/syscall-template.S:81
0000001 0x00007f8dd10227c3 in bnet_thread_server_tcp (addr_list=addr_list@entry=0x1b9eaa8, max_clients=<optimized out>, sockfds=<optimized out>, client_wq=client_wq@entry=0x6b7540 <socket_workq>, nokeepalive=<optimized out>,
    handle_client_request=handle_client_request@entry=0x442150 <handle_connection_request(void*)>) at bnet_server_tcp.c:306
0000002 0x00000000004423df in connect_thread (arg=arg@entry=0x1b9eaa8) at socket_server.c:115
0000003 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x1d46ec8) at lockmgr.c:928
0000004 0x00007f8dcf919064 in start_thread (arg=0x7f8dc71df700) at pthread_create.c:309
0000005 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) thread 18
[Switching to thread 18 (Thread 0x7f8dce722700 (LWP 10413))]
#0 0x00007f8dcf92014d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007f8dcf92014d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
0000001 0x00007f8dd102d8f4 in bmicrosleep (sec=sec@entry=30, usec=usec@entry=0) at bsys.c:171
0000002 0x00007f8dd103e76c in check_deadlock () at lockmgr.c:568
0000003 0x00007f8dcf919064 in start_thread (arg=0x7f8dce722700) at pthread_create.c:309
0000004 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

(gdb) thread 20
[Switching to thread 20 (Thread 0x7f8d89ffb700 (LWP 1954))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8d78003818) at watchdog.c:194
0000004 0x00007f8dd10309c4 in start_bsock_timer (bsock=0x7f8dc0009088, wait=600) at btimers.c:184
0000005 0x00007f8dd102a4f0 in BSOCK::two_way_authenticate (this=0x7f8dc0009088, jcr=0x0, what=0x47c76e "Console", name=0x47c762 "*UserAgent*", password=..., tls=..., initiated_by_remote=true) at bsock.c:340
0000006 0x000000000041118a in authenticate_inbound_connection (tls=..., password=..., name=<optimized out>, what=<optimized out>, jcr=<optimized out>, this=<optimized out>) at ../lib/bsock.h:151
0000007 authenticate_user_agent (uac=0x7f8d78002a78) at authenticate.c:262
0000008 0x000000000046f3fe in handle_UA_client_request (user=user@entry=0x7f8dc0009088) at ua_server.c:84
0000009 0x00000000004421d3 in handle_connection_request (arg=0x7f8dc0009088) at socket_server.c:98
0000010 0x00007f8dd1057f15 in workq_server (arg=arg@entry=0x6b7540 <socket_workq>) at workq.c:336
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x7f8dc000a2a8) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8d89ffb700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


(gdb) thread 10
[Switching to thread 10 (Thread 0x7f8dc59dc700 (LWP 22171))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
0000001 0x00007f8dd104ba9e in rwl_writelock_p (rwl=0x7f8dd1274e80 <lock>, file=<optimized out>, line=329) at rwlock.c:236
0000002 0x00007f8dd105733e in wd_lock () at watchdog.c:329
0000003 0x00007f8dd1057a4c in register_watchdog (wd=0x7f8da804f5b8) at watchdog.c:194
0000004 0x00007f8dd10308b7 in start_thread_timer (jcr=0x1c75308, tid=140246882567936, wait=1800) at btimers.c:148
0000005 0x00007f8dd102be6c in BSOCK_TCP::connect (this=0x7f8da80e9dd8, jcr=0x1c75308, retry_interval=10, max_retry_time=140247049556047, max_retry_time@entry=1800, heart_beat=140247076130304, heart_beat@entry=0,
    name=0x47c6e1 "Storage daemon", host=0x7f8d940b9d68 "backup02.bni", service=0x0, port=9103, verbose=true) at bsock_tcp.c:112
0000006 0x00000000004452ce in connect_to_storage_daemon (jcr=0x1c75308, retry_interval=<optimized out>, max_retry_time=<optimized out>, verbose=verbose@entry=true) at sd_cmds.c:118
0000007 0x00000000004453a8 in connect_to_storage_daemon (jcr=jcr@entry=0x1c75308, retry_interval=retry_interval@entry=10, max_retry_time=<optimized out>, verbose=verbose@entry=true) at sd_cmds.c:136
0000008 0x000000000041442a in do_native_backup (jcr=jcr@entry=0x1c75308) at backup.c:434
0000009 0x0000000000429654 in job_thread (arg=0x1c75308) at job.c:514
0000010 0x000000000042eb71 in jobq_server (arg=0x6b7240 <job_queue>) at jobq.c:485
0000011 0x00007f8dd103e7ff in lmgr_thread_launcher (x=0x1bce9f8) at lockmgr.c:928
0000012 0x00007f8dcf919064 in start_thread (arg=0x7f8dc59dc700) at pthread_create.c:309
0000013 0x00007f8dcec2862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

(gdb) p lock
$1 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 12, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 12 times>, "\f", '\000' <repeats 26 times>,
    __align = 0}, read = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, write = {
    __data = {__lock = 0, __futex = 12, __total_seq = 12, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x7f8dd1274e80 <lock>, __nwaiters = 24, __broadcast_seq = 0},
    __size = "\000\000\000\000\f\000\000\000\f", '\000' <repeats 23 times>, "\200N'э\177\000\000\030\000\000\000\000\000\000", __align = 51539607552}, writer_id = 140246899353344, priority = 0, valid = 16435934, r_active = 0,
  w_active = 1, r_wait = 0, w_wait = 12}
(gdb)

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f8dc49da700 (LWP 26418))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) p lock
$2 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 12, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 12 times>, "\f", '\000' <repeats 26 times>,
    __align = 0}, read = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, write = {
    __data = {__lock = 0, __futex = 12, __total_seq = 12, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x7f8dd1274e80 <lock>, __nwaiters = 24, __broadcast_seq = 0},
    __size = "\000\000\000\000\f\000\000\000\f", '\000' <repeats 23 times>, "\200N'э\177\000\000\030\000\000\000\000\000\000", __align = 51539607552}, writer_id = 140246899353344, priority = 0, valid = 16435934, r_active = 0,
  w_active = 1, r_wait = 0, w_wait = 12}
(gdb)
TagsNo tags attached.

Activities

arogge

arogge

2019-07-19 15:15

manager   ~0003504

Looks like the watchdog got stuck.

How long did this situation persist?
Is this reproducible somehow?
arogge

arogge

2019-12-18 15:40

manager   ~0003691

closing due to no response on feedback request

Issue History

Date Modified Username Field Change
2018-11-26 17:07 frank New Issue
2018-11-26 17:07 frank Status new => assigned
2018-11-26 17:07 frank Assigned To => franku
2018-11-26 17:15 frank Additional Information Updated
2019-07-15 15:53 franku Assigned To franku =>
2019-07-15 15:53 franku Status assigned => new
2019-07-19 15:15 arogge Status new => feedback
2019-07-19 15:15 arogge Note Added: 0003504
2019-12-18 15:39 arogge Assigned To => arogge
2019-12-18 15:39 arogge Status feedback => resolved
2019-12-18 15:39 arogge Resolution open => unable to reproduce
2019-12-18 15:40 arogge Status resolved => closed
2019-12-18 15:40 arogge Note Added: 0003691