 |
|
|
|
A quick look at the stack traces reveals that its crashing in a pthread_kill so it would be interesting to see a core file from the crash to see what the thread id looks like as I get the feeling that might be either bogus or NULL. Maybe it also makes sense to install the debug symbols but a core file is probably more interesting first. |
|
|
|
|
|
bareos-dir segfaultet again (so two consecutive days), see uploaded traceback. core is available |
|
|
Seems to be exactly the same problem: #0 0x00007f308af162ad in __libc_waitpid (pid=<value optimized out>, stat_loc=<value optimized out>, options=<value optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:41 0000001 0x00007f308bdfa08f in signal_handler (sig=11) at signal.c:228 0000002 <signal handler called> 0000003 __pthread_kill (threadid=139844753794816, signo=12) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:43 0000004 0x00007f308bde8e4e in JCR::my_thread_send_signal (this=0x260aec8, sig=12) at jcr.c:698 0000005 0x00000000004416c8 in cancel_storage_daemon_job (ua=0x7f3044167748, jcr=0x260aec8, silent=true) at sd_cmds.c:699 0000006 0x000000000044182d in cancel_storage_daemon_job (jcr=0x260aec8) at sd_cmds.c:723 So don't think a extra core makes sense at this time, just set it aside when we want to have confirmation. |
|
|
Hmm cannot really start much with the core files as it seems either some stack frames got hit or its because I cannot fully get exactly the same versions of each and every rpm on a test CentOS 6 install. Maybe its easier to do the postmortem on the system that generated the core. No idea if you have installed any debuginfo rpms on that machine but we might try seeing what the following gives. # gdb /usr/sbin/bareos-dir bareos-dir.core.21416 (gdb) thread 15 (gdb) bt (gdb) up 4 (gdb) print *this |
|
|
|
|
|
bareos-dir segfaultet again (so three consecutive days), see uploaded traceback. core is available. |
|
|
Think we can use the latest core as the traceback shows the same items. Only its in thread 10 instead of 15. Seems you have every time a FD Job that fails and seems to trigger some bug that must come from upstream. So you can try the above but then with this latest core and use thread 10 |
|
|
$ gdb /usr/sbin/bareos-dir bareos-dir.core.45040 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/bareos-dir...Reading symbols from /usr/lib/debug/usr/sbin/bareos-dir.debug...done. done. [New Thread 45042] [New Thread 45114] [New Thread 45115] [New Thread 45116] [New Thread 45213] [New Thread 45214] [New Thread 45216] [New Thread 45218] [New Thread 45219] [New Thread 45220] [New Thread 45221] [New Thread 45222] [New Thread 45225] [New Thread 45233] [New Thread 45234] [New Thread 45237] [New Thread 64740] [New Thread 65006] [New Thread 12851] [New Thread 22510] [New Thread 29387] [New Thread 31020] [New Thread 31273] [New Thread 31274] [New Thread 31298] [New Thread 31743] [New Thread 45040] Missing separate debuginfo for Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/71/a96b79419fe40ff1706d59ed06d6dbe37c8e97 Reading symbols from /usr/lib64/bareos/libbareosndmp-14.2.2.so...Reading symbols from /usr/lib/debug/usr/lib64/bareos/libbareosndmp-14.2.2.so.debug...done. done. Loaded symbols for /usr/lib64/bareos/libbareosndmp-14.2.2.so Reading symbols from /usr/lib64/bareos/libbareosfind-14.2.2.so...Reading symbols from /usr/lib/debug/usr/lib64/bareos/libbareosfind-14.2.2.so.debug...done. done. Loaded symbols for /usr/lib64/bareos/libbareosfind-14.2.2.so Reading symbols from /lib64/libacl.so.1.1.0...Reading symbols from /usr/lib/debug/lib64/libacl.so.1.1.0.debug...done. done. Loaded symbols for /lib64/libacl.so.1.1.0 Reading symbols from /usr/lib64/bareos/libbareossql-14.2.2.so...Reading symbols from /usr/lib/debug/usr/lib64/bareos/libbareossql-14.2.2.so.debug...done. done. Loaded symbols for /usr/lib64/bareos/libbareossql-14.2.2.so Reading symbols from /usr/lib64/bareos/libbareoscats-14.2.2.so...Reading symbols from /usr/lib/debug/usr/lib64/bareos/libbareoscats-14.2.2.so.debug...done. done. Loaded symbols for /usr/lib64/bareos/libbareoscats-14.2.2.so Reading symbols from /usr/lib64/bareos/libbareoscfg-14.2.2.so...Reading symbols from /usr/lib/debug/usr/lib64/bareos/libbareoscfg-14.2.2.so.debug...done. done. Loaded symbols for /usr/lib64/bareos/libbareoscfg-14.2.2.so Reading symbols from /usr/lib64/bareos/libbareos-14.2.2.so...Reading symbols from /usr/lib/debug/usr/lib64/bareos/libbareos-14.2.2.so.debug...done. done. Loaded symbols for /usr/lib64/bareos/libbareos-14.2.2.so Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from /usr/lib/debug/lib64/libz.so.1.2.3.debug...done. done. Loaded symbols for /lib64/libz.so.1.2.3 Reading symbols from /usr/lib64/liblzo2.so.2.0.0...Reading symbols from /usr/lib/debug/usr/lib64/liblzo2.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib64/liblzo2.so.2.0.0 Reading symbols from /usr/lib64/libfastlz.so.1.0.0...Reading symbols from /usr/lib/debug/usr/lib64/libfastlz.so.1.0.0.debug...done. done. Loaded symbols for /usr/lib64/libfastlz.so.1.0.0 Reading symbols from /usr/lib64/libssl.so.1.0.1e...Reading symbols from /usr/lib/debug/usr/lib64/libssl.so.1.0.1e.debug...done. done. Loaded symbols for /usr/lib64/libssl.so.1.0.1e Reading symbols from /usr/lib64/libcrypto.so.1.0.1e...Reading symbols from /usr/lib/debug/usr/lib64/libcrypto.so.1.0.1e.debug...done. done. Loaded symbols for /usr/lib64/libcrypto.so.1.0.1e Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from /usr/lib/debug/lib64/libpthread-2.12.so.debug...done. [Thread debugging using libthread_db enabled] done. Loaded symbols for /lib64/libpthread-2.12.so Reading symbols from /lib64/libdl-2.12.so...Reading symbols from /usr/lib/debug/lib64/libdl-2.12.so.debug...done. done. Loaded symbols for /lib64/libdl-2.12.so Reading symbols from /lib64/libwrap.so.0.7.6...Reading symbols from /usr/lib/debug/lib64/libwrap.so.0.7.6.debug...done. done. Loaded symbols for /lib64/libwrap.so.0.7.6 Reading symbols from /lib64/libcap.so.2.16...Reading symbols from /usr/lib/debug/lib64/libcap.so.2.16.debug...done. done. Loaded symbols for /lib64/libcap.so.2.16 Reading symbols from /usr/lib64/libstdc++.so.6.0.13...Reading symbols from /usr/lib/debug/usr/lib64/libstdc++.so.6.0.13.debug...done. done. Loaded symbols for /usr/lib64/libstdc++.so.6.0.13 Reading symbols from /lib64/libm-2.12.so...Reading symbols from /usr/lib/debug/lib64/libm-2.12.so.debug...done. done. Loaded symbols for /lib64/libm-2.12.so Reading symbols from /lib64/libgcc_s-4.4.7-20120601.so.1...Reading symbols from /usr/lib/debug/lib64/libgcc_s-4.4.7-20120601.so.1.debug...done. done. Loaded symbols for /lib64/libgcc_s-4.4.7-20120601.so.1 Reading symbols from /lib64/libc-2.12.so...Reading symbols from /usr/lib/debug/lib64/libc-2.12.so.debug...done. done. Loaded symbols for /lib64/libc-2.12.so Reading symbols from /lib64/libattr.so.1.1.0...Reading symbols from /usr/lib/debug/lib64/libattr.so.1.1.0.debug...done. done. Loaded symbols for /lib64/libattr.so.1.1.0 Reading symbols from /lib64/libgssapi_krb5.so.2.2...Reading symbols from /usr/lib/debug/lib64/libgssapi_krb5.so.2.2.debug...done. done. Loaded symbols for /lib64/libgssapi_krb5.so.2.2 Reading symbols from /lib64/libkrb5.so.3.3...Reading symbols from /usr/lib/debug/lib64/libkrb5.so.3.3.debug...done. done. Loaded symbols for /lib64/libkrb5.so.3.3 Reading symbols from /lib64/libcom_err.so.2.1...Reading symbols from /usr/lib/debug/lib64/libcom_err.so.2.1.debug...done. done. Loaded symbols for /lib64/libcom_err.so.2.1 Reading symbols from /lib64/libk5crypto.so.3.1...Reading symbols from /usr/lib/debug/lib64/libk5crypto.so.3.1.debug...done. done. Loaded symbols for /lib64/libk5crypto.so.3.1 Reading symbols from /lib64/ld-2.12.so...Reading symbols from /usr/lib/debug/lib64/ld-2.12.so.debug...done. done. Loaded symbols for /lib64/ld-2.12.so Reading symbols from /lib64/libnsl-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnsl-2.12.so.debug...done. done. Loaded symbols for /lib64/libnsl-2.12.so Reading symbols from /lib64/libkrb5support.so.0.1...Reading symbols from /usr/lib/debug/lib64/libkrb5support.so.0.1.debug...done. done. Loaded symbols for /lib64/libkrb5support.so.0.1 Reading symbols from /lib64/libkeyutils.so.1.3...Reading symbols from /usr/lib/debug/lib64/libkeyutils.so.1.3.debug...done. done. Loaded symbols for /lib64/libkeyutils.so.1.3 Reading symbols from /lib64/libresolv-2.12.so...Reading symbols from /usr/lib/debug/lib64/libresolv-2.12.so.debug...done. done. Loaded symbols for /lib64/libresolv-2.12.so Reading symbols from /lib64/libselinux.so.1...Reading symbols from /usr/lib/debug/lib64/libselinux.so.1.debug...done. done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from /lib64/libnss_ldap.so.2...Reading symbols from /usr/lib/debug/lib64/libnss_ldap.so.2.debug...done. done. Loaded symbols for /lib64/libnss_ldap.so.2 Reading symbols from /usr/lib64/gconv/ISO8859-15.so...Reading symbols from /usr/lib/debug/usr/lib64/gconv/ISO8859-15.so.debug...done. done. Loaded symbols for /usr/lib64/gconv/ISO8859-15.so Reading symbols from /usr/lib64/bareos/backends/libbareoscats-mysql-14.2.2.so...Reading symbols from /usr/lib/debug/usr/lib64/bareos/backends/libbareoscats-mysql-14.2.2.so.debug...done. done. Loaded symbols for /usr/lib64/bareos/backends/libbareoscats-mysql-14.2.2.so Reading symbols from /usr/lib64/mysql/libmysqlclient_r.so.16.0.0...Reading symbols from /usr/lib/debug/usr/lib64/mysql/libmysqlclient_r.so.16.0.0.debug...done. done. Loaded symbols for /usr/lib64/mysql/libmysqlclient_r.so.16.0.0 Reading symbols from /lib64/libcrypt-2.12.so...Reading symbols from /usr/lib/debug/lib64/libcrypt-2.12.so.debug...done. done. Loaded symbols for /lib64/libcrypt-2.12.so Reading symbols from /lib64/libfreebl3.so...Reading symbols from /usr/lib/debug/lib64/libfreebl3.so.debug...done. done. Loaded symbols for /lib64/libfreebl3.so Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/sbin/bareos-dir -g bareos -c /etc/bareos/bareos-dir.conf'. #0 0x00007f62a34aaf3d in nanosleep () at ../sysdeps/unix/syscall-template.S:82 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) thread 10 [Switching to thread 10 (Thread 0x7f6273fff700 (LWP 45220))]#0 0x00007f62a34ab2ad in __libc_waitpid (pid=<value optimized out>, stat_loc=<value optimized out>, options=<value optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:41 41 int result = INLINE_SYSCALL (wait4, 4, pid, stat_loc, options, NULL); (gdb) bt #0 0x00007f62a34ab2ad in __libc_waitpid (pid=<value optimized out>, stat_loc=<value optimized out>, options=<value optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:41 0000001 0x00007f62a438f08f in signal_handler (sig=11) at signal.c:228 0000002 <signal handler called> 0000003 __pthread_kill (threadid=140059869173504, signo=12) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:43 0000004 0x00007f62a437de4e in JCR::my_thread_send_signal (this=0x1373498, sig=12) at jcr.c:698 0000005 0x00000000004416c8 in cancel_storage_daemon_job (ua=0x7f6268124888, jcr=0x1373498, silent=true) at sd_cmds.c:699 0000006 0x000000000044182d in cancel_storage_daemon_job (jcr=0x1373498) at sd_cmds.c:723 0000007 0x000000000040f8bf in wait_for_job_termination (jcr=0x1373498, timeout=<value optimized out>) at backup.c:689 0000008 0x0000000000412587 in do_native_backup (jcr=0x1373498) at backup.c:615 0000009 0x000000000042a909 in job_thread (arg=0x1373498) at job.c:419 0000010 0x000000000042c83b in jobq_server (arg=0x69e180) at jobq.c:450 0000011 0x00007f62a43812fd in lmgr_thread_launcher (x=0x1195918) at lockmgr.c:932 0000012 0x00007f62a34a39d1 in start_thread (arg=0x7f6273fff700) at pthread_create.c:301 0000013 0x00007f62a243d8fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) up 4 0000004 0x00007f62a437de4e in JCR::my_thread_send_signal (this=0x1373498, sig=12) at jcr.c:698 698 pthread_kill(this->my_thread_id, sig); (gdb) print *this $1 = {mutex = {__data = {__lock = 2, __count = 0, __owner = 45220, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000??\000\000\001", '\000' <repeats 26 times>, __align = 2}, _use_count = 1, m_JobType = 66, m_JobLevel = 73, m_Protocol = 0, my_thread_killable = true, link = {next = 0x137fc98, prev = 0x1322838}, my_thread_id = 140059869173504, dir_bsock = 0x0, store_bsock = 0x7f626800fa78, file_bsock = 0x0, daemon_free_jcr = 0x4277b0 <dird_free_jcr(JCR*)>, msg_queue = 0x118b568, msg_queue_mutex = {__data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, dequeuing_msgs = false, job_end_push = {<SMARTALLOC> = {<No data fields>}, items = 0x0, num_items = 0, max_items = 0, num_grow = 1, cur_item = 0, own_items = false}, VolumeName = 0x7f62800067d0 "EHYP5160", errmsg = 0x1373210 "", Job = "magma9-standard-job.2015-03-25_19.03.47_48", '\000' <repeats 85 times>, JobId = 69714, VolSessionId = 317, VolSessionTime = 1427226407, JobFiles = 0, JobErrors = 0, JobWarnings = 0, LastRate = 0, DumpLevel = 0, JobBytes = 0, LastJobBytes = 0, ReadBytes = 0, FileId = 0, JobStatus = 69, JobPriority = 10, sched_time = 1427306627, initial_sched_time = 1427306627, start_time = 1427312059, run_time = 0, last_time = 0, end_time = 0, wait_time_sum = 5618, wait_time = 0, job_started_time = 0, client_name = 0x1372ef0 "magma9-fd", JobIds = 0x0, RestoreBootstrap = 0x0, stime = 0x1374f60 "2015-03-22 21:07:48", sd_auth_key = 0x7f6268027dc8 "AEOB-MDFN-EJFA-FKLA-EPFJ-FHKM-PEDD-DBFC", jcr_msgs = 0x1374d48, ClientId = 0, where = 0x0, RegexWhere = 0x0, where_bregexp = 0x0, cached_pnl = 0, cached_path = 0x0, passive_client = false, prefix_links = false, gui = false, authenticated = false, cached_attribute = false, batch_started = false, cmd_plugin = false, opt_plugin = false, keep_path_list = false, accurate = true, HasBase = false, rerunning = false, job_started = false, suppress_output = false, cjcr = 0x0, buf_size = 0, compress = {deflate_buffer = 0x0, inflate_buffer = 0x0, deflate_buffer_size = 0, inflate_buffer_size = 0, workset = {pZLIB = 0x0, pLZO = 0x0, pZFAST = 0x0}}, attr = 0x0, db = 0x7f62840029b8, db_batch = 0x0, nb_base_files = 0, nb_base_files_used = 0, ar = 0x0, id_list = 0x0, plugin_ctx_list = 0x0, plugin_ctx = 0x0, plugin_sp = 0x0, comment = 0x7f6280006680 "", max_bandwidth = 0, path_list = 0x0} (gdb) quit |
|
|
Fix committed to bareos bareos-15.2 branch with changesetid 5617. |
|
|
Fix committed to bareos bareos-14.2 branch with changesetid 5787. |
|
|
Fix committed to bareos bareos-13.2 branch with changesetid 5841. |
- Anonymous
