View Issue Details

IDProjectCategoryView StatusLast Update
0001438bareos-corefile daemonpublic2022-05-03 10:34
Reporterjcottin Assigned Tobruno-at-bareos  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionunable to reproduce 
PlatformLinuxOSCentOSOS Version7
Product Version21.1.1 
Summary0001438: director cannot connect to the client
DescriptionHi,

We use ansible to add new client to bareos.

Today we encounter a problem on a server.
The director cannot connect to the client.

telnet is ok though

Server => Client:
telnet 10.0.9.102 9102
Trying 10.0.9.102...
Connected to 10.0.9.102.
Escape character is '^]'.
^]

The log only says:
17-Mar 20:30 bareos-dir JobId 0: Fatal error: Network error during CRAM MD5 with 10.0.9.102
17-Mar 20:30 bareos-dir JobId 0: Fatal error: Unable to authenticate with File daemon at "10.0.9.102:9102"

Steps To ReproduceServer - Debian 10.11:
cat /etc/bareos/bareos-dir.d/client/vm-rd-fd.conf
Client {
  Name = "vm-rd-fd"
  Address = "10.0.9.102"
  Password = "BIG_PASSWORD"
  #Passive = yes
}

systemctl restart bareos-dir

Client - Centos 7.3:
cat /etc/bareos/bareos-fd.d/director/bareos-dir.conf
Director {
  Name = bareos-dir
  Password = "SAME_BIG_PASSWORD"
  Description = "Allow the configured Director to access this file daemon."
}

systemctl restart bareos-fd

*status client=vm-rd-fd
Connecting to Client vm-d-fd at 10.0.9.102:9102
Probing client protocol... (result will be saved until config reload)
Failed to connect to Client vm-rd-fd.
====
TagsNo tags attached.

Activities

jcottin

jcottin

2022-03-18 16:15

reporter   ~0004542

I tried with version 20.
same behavior.

https://docs.bareos.org/Appendix/Troubleshooting.html?highlight=error#client-access-problems
Everything looks ok.

daemon is running
IP and Port are correct
no firewall
no tcpwrapper
password and name are correct
bruno-at-bareos

bruno-at-bareos

2022-03-21 17:03

manager   ~0004544

Hello, could you list here the installed packages and their sources (own build, or bareos.org), also detailing the platform, you talk about debian and centos is selected.

Maybe you can also tried to check if it works with TLS disabled, and raise also the debug level to level 100 or 200 and report back.
Thanks
jcottin

jcottin

2022-03-22 08:40

reporter   ~0004545

Hi Bruno,

The backup server is a debian running Bareos 21
cat /etc/debian_version
10.11

bareos-bconsole/unknown,now 21.0.0-4 amd64 [installed,automatic]
bareos-client/unknown,now 21.0.0-4 amd64 [installed,automatic]
bareos-common/unknown,now 21.0.0-4 amd64 [installed,automatic]
bareos-database-common/unknown,now 21.0.0-4 amd64 [installed,automatic]
bareos-database-postgresql/unknown,now 21.0.0-4 amd64 [installed]
bareos-database-tools/unknown,now 21.0.0-4 amd64 [installed,automatic]
bareos-director/unknown,now 21.0.0-4 amd64 [installed,automatic]
bareos-filedaemon/unknown,now 21.0.0-4 amd64 [installed]
bareos-storage/unknown,now 21.0.0-4 amd64 [installed,automatic]
bareos-tools/unknown,now 21.0.0-4 amd64 [installed,automatic]
bareos-webui/unknown,now 21.0.0-4 all [installed]
bareos/unknown,now 21.0.0-4 amd64 [installed]

The server that I try to backup is a Centos 7
The source are from the bareos repository, I tried 21 and 20
cat /etc/yum.repos.d/bareos.repo
[bareos]
baseurl = http://download.bareos.org/bareos/release/20/CentOS_7/

yum list installed | grep bareos
bareos-common.x86_64 20.0.1-3.el7 @bareos
bareos-filedaemon.x86_64 20.0.1-3.el7 @bareos


On the server:
egrep -v '#|^$' /etc/bareos/bareos-dir.d/client/vm-rd-fd.conf
Client {
  Name = "vm-rd-fd"
  Address = "10.0.9.102"
  Password = "BIG_PASSWORD"
  TLS Enable = no
}

On the client:
egrep -v '#|^$' /etc/bareos/bareos-fd.d/client/myself.conf
Client {
  Name = vm-rd-fd
  Maximum Concurrent Jobs = 20
  TLS Enable = no
}

bconsole
*status client=vm-rd-fd
Connecting to Client vm-rd-fd at 10.0.9.102:9102
Probing client protocol... (result will be saved until config reload)
Failed to connect to Client vm-rd-fd.

I didn't find how to raise the debug level.
bruno-at-bareos

bruno-at-bareos

2022-03-22 10:02

manager   ~0004546

A few points to check like stated in documentation here https://docs.bareos.org/bareos-21/Appendix/Troubleshooting.html#authorization-errors
Be sure you can ping/resolve hosts with the name entered in each configuration file. Also check if reverse dns is working and setup properly as stated in documentation.
On your CentOS fd did you set SELinux correctly to allow bareos-fd actions ?

Maybe you're unlucky and have setup a password that doesn't work, check if changing it can resolve the problem.

setdebug is documented here
https://docs.bareos.org/bareos-21/TasksAndConcepts/BareosConsole.html?#id42
jcottin

jcottin

2022-03-22 11:55

reporter   ~0004547

Hi,

Thanks for the pointers.

The password is the same on both side.
The director's name is the default bareos-dir.

SELinux is disabled.
SELinux status: disabled

I specified the IP address the client.
ping and telnet ok using the IP

The password is only composed of letter and number.
It has already been changed a few times.

I set the debug like this:
bconsole
*setdebug level=200 trace=1 dir
*status client=vm-rd-fd

Here's the output:
=================================
bareos-dir (10): dird/ua_audit.cc:137-0 : Console [default] from [::1] cmdline setdebug level=200 trace=1 dir
bareos-dir (120): dird/ua_cmds.cc:1407-0 setdebug:setdebug level=200 trace=1 dir:

bareos-dir (10): dird/ua_audit.cc:137-0 : Console [default] from [::1] cmdline status client=vm-rd-fd
bareos-dir (20): dird/ua_status.cc:171-0 status:status client=vm-rd-fd:
bareos-dir (120): dird/job.cc:406-0 Client Initiated Connection from "vm-rd-fd" is not allowed.
bareos-dir (100): lib/bsock.cc:84-0 Construct BareosSocket
bareos-dir (100): lib/bsock.cc:161-0 All source addresses
bareos-dir (100): lib/bsock_tcp.cc:220-0 Current host[ipv4;10.0.9.102;9102] All host[ipv4;10.0.9.102;9102]
bareos-dir (100): lib/bsock_tcp.cc:148-0 who=Client: vm-rd-fd host=10.0.9.102 port=9102
bareos-dir (10): dird/fd_cmds.cc:152-0 Opened connection with File daemon
bareos-dir (50): dird/authenticate.cc:159-0 Sent: Hello Director bareos-dir calling

bareos-dir (50): lib/cram_md5.cc:261-0 cram-auth failed with Client: vm-rd-fd
bareos-dir (50): dird/authenticate.cc:171-0 Unable to authenticate with File daemon at "10.0.9.102:9102"

bareos-dir (100): lib/bsock.cc:136-0 Destruct BareosSocket
bareos-dir (100): lib/tls_openssl_private.cc:96-0 Destruct TlsOpenSslPrivate
bareos-dir (200): dird/job.cc:1526-0 Start dird FreeJcr
bareos-dir (200): dird/job.cc:1592-0 End dird FreeJcr
bareos-dir (100): lib/jcr.cc:273-0 FreeCommonJcr: 7fafc80a1e70
bareos-dir (200): lib/util.cc:918-0 edit_job_codes: /usr/bin/bsmtp -h localhost -f "(Bareos) <%r>" -s "Bareos daemon message" %r
bareos-dir (100): lib/bsys.cc:78-0 safe_unlink unlinking: /var/lib/bareos/bareos-dir.-Console-.2022-03-22_18.44.27_11.-938910368.mail
bareos-dir (100): lib/bsock.cc:136-0 Destruct BareosSocket
=================================

Did I correctly executed the debug steps ?
bruno-at-bareos

bruno-at-bareos

2022-03-30 16:29

manager   ~0004557

Hi thanks for your try, yes the debug is set correctly.
Never forget to remove it afterwards, otherwise the trace file can quickly eat all free space.

No having the configuration file (complete version) doesn't help to make conclusion, but I would recommend to use if possible for you real fqdn hostname in the configuration (director and client side)
and be really pitnick about the resource name you use on director -> client configuration but also client-director configuration, because fi the Address can be different, but the Name entry must match on FD and DIR, as it's used as TLS-PSK identity.

Which is probably leading to the error you're seeing.
jcottin

jcottin

2022-03-30 16:52

reporter   ~0004559

I only use IP addresses.

This is the full configuration.
You can ask me if something is missing.
-------------

# ClientToBackup : vm-rd-fd
===================================================
# cat /etc/bareos/bareos-fd.d/client/myself.conf
Client {
  Name = vm-rd-fd
  Maximum Concurrent Jobs = 20
  TLS Enable = no
}

# cat /etc/bareos/bareos-fd.d/director/bareos-dir.conf
Director {
  Name = bareos-dir
  Password = "Client_password"
  Description = "Allow the configured Director to access this file daemon."
}
===================================================
# BackupServer
# cat /etc/bareos/bareos-dir.d/client/vm-rd-fd.conf
Client {
  Name = "vm-rd-fd"
  Address = "10.0.9.102"
  Password = "Client_password"
  #Passive = yes
  TLS Enable = no
}

# cat /etc/bareos/bareos-dir.d/director/bareos-dir.conf
Director { # define myself
  Name = bareos-dir
  QueryFile = "/usr/lib/bareos/scripts/query.sql"
  Maximum Concurrent Jobs = 10
  Password = "Director_password" # Console password
  Messages = Daemon
  Auditing = yes
}
jcottin

jcottin

2022-04-30 03:08

reporter   ~0004580

Do you need more information?
bruno-at-bareos

bruno-at-bareos

2022-05-02 09:36

manager   ~0004581

Hello,

Yes we would like to have a trace log with a higher level on both side.

For the client, it will be easier to run it on the foreground directly with the debug level activated.
as we can't connect and send it the trace signal.

So with a ssh connection as root on your failing client

#Stop the running daemon if any
systemctl stop bareos-fd
#start the daemon in debug mode
/usr/sbin/bareos-fd -dt -d1000 -vvv -u root -g bareos | tee /var/lib/bareos/vm-rd-fd.trace

For the director, after clean the existing trace file, you can use the exact same procedure as before.
in bconsole
setdebug level=1000 trace=1 timestamp=1 dir
status client= vm-rd-fd

Attach (maybe compress as gz before) both trace file here.

Thanks.
jcottin

jcottin

2022-05-02 19:10

reporter   ~0004582

Problem solved after a system upgrade from centos 7.3 to 7.9

=============
Connecting to Client vm-rd-fd at 10.0.9.102:9102
 Handshake: Immediate TLS, Encryption: PSK-AES256-CBC-SHA SSLv3

vm-rd-fd Version: 21.0.0 (21 December 2021) CentOS Linux release 7.9.2009 (Core)
Daemon started 02-May-22 12:55. Jobs: run=0 running=0, bareos.org build binary
 Sizeof: boffset_t=8 size_t=8 debug=1000 trace=0 bwlimit=0kB/s
=============
bruno-at-bareos

bruno-at-bareos

2022-05-03 10:34

manager   ~0004584

Ok thanks for the report, closing.
bruno-at-bareos

bruno-at-bareos

2022-05-03 10:34

manager   ~0004585

Can't be reproduced after system update to current path level
CentOS 7.3 to 7.9

Issue History

Date Modified Username Field Change
2022-03-17 13:34 jcottin New Issue
2022-03-18 16:15 jcottin Note Added: 0004542
2022-03-21 17:03 bruno-at-bareos Note Added: 0004544
2022-03-22 08:40 jcottin Note Added: 0004545
2022-03-22 10:02 bruno-at-bareos Note Added: 0004546
2022-03-22 11:55 jcottin Note Added: 0004547
2022-03-30 16:29 bruno-at-bareos Note Added: 0004557
2022-03-30 16:52 jcottin Note Added: 0004559
2022-04-30 03:08 jcottin Note Added: 0004580
2022-05-02 09:36 bruno-at-bareos Note Added: 0004581
2022-05-02 19:10 jcottin Note Added: 0004582
2022-05-03 10:34 bruno-at-bareos Note Added: 0004584
2022-05-03 10:34 bruno-at-bareos Assigned To => bruno-at-bareos
2022-05-03 10:34 bruno-at-bareos Status new => closed
2022-05-03 10:34 bruno-at-bareos Resolution open => unable to reproduce
2022-05-03 10:34 bruno-at-bareos Note Added: 0004585