View Issue Details

IDProjectCategoryView StatusLast Update
0001551bareos-coreGeneralpublic2023-09-12 11:37
Reportermschiff Assigned Tobruno-at-bareos  
PrioritynormalSeverityblockReproducibilityalways
Status closedResolutionfixed 
PlatformLinuxOSanyOS Version3
Product Version22.1.0 
Summary0001551: 22.1.0 breaks compatibility with older file-daemons
DescriptionWe are slowly migrating from TLS-PKI to TLS-PSK. The SD and DIR are current and allow both to be used while new clients use TLS-PSK and older clients use TLS-PKI.

After upgrading the DIR and SD to 22.1 all old clients are now unable to connect:

04-Sep 23:47 dir JobId 70778: Start Backup JobId 70778, Job=Backup.2023-09-04_23.47.55_46
04-Sep 23:47 dir JobId 70778: Connected Storage daemon at dir:9103, encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
04-Sep 23:47 dir JobId 70778: Encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
04-Sep 23:47 dir JobId 70778: Probing client protocol... (result will be saved until config reload)
04-Sep 23:48 dir JobId 70778: TLS negotiation failed (while probing client protocol

As I could not find anything in the release notes about breaking changes, I guess this was not intentional.

dir config:
  TLS Enable = yes
  TLS Require = yes
  TLS Verify Peer = yes
  TLS CA Certificate File = /etc/bareos/ssl/ca.crt

new client:
  TLS Enable = yes
  TLS Require = yes

old client:
  TLS Enable = yes
  TLS Require = yes
  TLS Certificate = /etc/bareos/ssl/dir.crt.pem
  TLS Key = /etc/bareos/ssl/dir.key.pem
  TLS CA Certificate File = /etc/bareos/ssl/ca.crt
Tagsbroken, configuration, director, network

Activities

mschiff

mschiff

2023-09-05 00:58

reporter   ~0005372

Here is a log when it is working (now after downgrade to 22.0.3):

05-Sep 00:55 dir JobId 70780: Start Backup JobId 70780, Job=Backup.2023-09-05_00.55.19_03
05-Sep 00:55 dir JobId 70780: Connected Storage daemon at dir:9103, encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
05-Sep 00:55 dir JobId 70780: Encryption: TLS_AES_256_GCM_SHA384 TLSv1.3
05-Sep 00:55 dir JobId 70780: Probing client protocol... (result will be saved until config reload)
05-Sep 00:55 dir JobId 70780: TLS negotiation failed (while probing client protocol)
05-Sep 00:55 dir JobId 70780: Connected Client: client-fd at client:9102, encryption: DHE-RSA-AES256-GCM-SHA384 TLSv1.2
05-Sep 00:55 dir JobId 70780: Handshake: Cleartext
05-Sep 00:55 dir JobId 70780: Encryption: DHE-RSA-AES256-GCM-SHA384 TLSv1.2
bruno-at-bareos

bruno-at-bareos

2023-09-05 13:57

manager   ~0005375

Maybe because one is using TLS 1.3 and the other is using (limited?) to TLS 1.2 ?
mschiff

mschiff

2023-09-05 22:10

reporter   ~0005387

Then it should automatically downgrade to TLSv1.2, I did not have any tls version or cipher lists configued. But I tried now, but it makes no difference.

Is it possible that upgrading from 22.0.3 to 22.1 will break using TLS-PKI and TLS-PSK at the same time? (to support old and new clients)
bruno-at-bareos

bruno-at-bareos

2023-09-06 10:22

manager   ~0005388

Hello, Sorry I got confused yesterday, and mixed the version your were using.
So reading it again with fresh eyes lead to two things.

Either we have a bug since version 18.2.5 (introduction of TLS-PSK) which should have work with TLS-PKI, but you log seems to prove the inverse:
see the following line
JobId 70780: TLS negotiation failed (while probing client protocol)
JobId 70780: Handshake: Cleartext

The TLS-PKI failed, and then the connection was "downgraded" to clear text :scream:

In 22.1.0 we backported the new default which is : don't allow tls connection to be downgraded to cleartext by default (PR1398)
this time as in configuration you have require tls = yes the process failed, and as no cleartext downgrade escape path to build the connection.

So 22.1.0 is better that 22.0.3 in terms of "real" security.

I would like to know if you can first test that your certificates are still valid, so really a connection can be made with.
That will help us to make sure the trouble is somewhere in the code when connection are made and TLS mode is choose.
mschiff

mschiff

2023-09-06 13:48

reporter   ~0005389

Thank you for your message!

We are currently using a mix of old and new clients, so are using TLS-PKI (old clients) as well as TLS-PSK (new clients).
And the new version is changing behavior somewhere and therefore maybe breaks compatibility.

This leads to these questions for me:
- Is there a new config option so that we can restore the previous behavior? (read: allow downgrade again) I could not find anythign in the release notes about that
- What is the issue with the Cleartext Handshake? In the message after that is says which cipher suite is now being used for TLS.



Were
bruno-at-bareos

bruno-at-bareos

2023-09-06 14:53

manager   ~0005390

From dev team, for normal PKI we do connect -> handshake/auth -> then do TLS
which is not more allowed by default on 22.1 due to the insecure authentication scram/md5

So to recover the old insecure function you will have to setup "TLS Require = No" but keep the TLS Enable = Yes (which the default)

A question is raise is what you consider what a old client version (as below 20 is not more supported)
mschiff

mschiff

2023-09-06 15:02

reporter   ~0005391

Thanks! I have now sorted it out.

We have old clients down to bareos-13, but will get rid of then in the future.

So to solve the issue (using oepnssl 1.1.1) it was enought now to set "TLS Require = No" for old clients to restore the old behavior. I know its not perfect, but better than nothing and in this environment also its not going through public networks...

The second issue I was having was really not bareos related, but I will leave the info here for future reference:

When upgrading from openssl 1.1.1 to a much newer openssl-3.0.x (3.0.9 here currently) you have to configure the SECLEVEL as well because otherwise openssl itself will not allow TLSv1 connections anymore (Yes above I was writing TLSv1.2, but this must have been an older, but not the oldest client)
like this (at least):

TLS Cipher List = "DEFAULT@SECLEVEL=0"

So no issue in bareos itself, only that some kind of a BIGGER warning and how to restore old behavior in the Release Notes would have been very nice as it would have saved me a lot of trouble now ;-)
arogge

arogge

2023-09-06 16:36

manager   ~0005392

Bareos 17.2 is out of support for quite some time. While breaking this was not our intention it is the right thing to do:
if you look at https://docs.bareos.org/DeveloperGuide/tls-techdoc.html you can see that the old TLS implementation (i.e. pre-18.2) does CRAM-MD5 before establishing a secure channel using TLS. When requiring TLS, I would expect everything (including the authentication) to happen on an encrypted link - which is what TLS Require enforces since 22.1.
If you need backwards compatibility to the inherently broken pre-18.2 protocol, you'll have to disable "TLS Require" (side note: TLS Require will be the default in Bareos 23 and newer).

Concerning the release notes we would probably have mentioned it, if we had realized that we broke backwards compatibility.
bruno-at-bareos

bruno-at-bareos

2023-09-12 11:37

manager   ~0005411

pki with new protocol do encryption then auth.
The old one fail

Issue History

Date Modified Username Field Change
2023-09-05 00:14 mschiff New Issue
2023-09-05 00:14 mschiff Tag Attached: broken
2023-09-05 00:14 mschiff Tag Attached: configuration
2023-09-05 00:14 mschiff Tag Attached: director
2023-09-05 00:14 mschiff Tag Attached: network
2023-09-05 00:58 mschiff Note Added: 0005372
2023-09-05 13:57 bruno-at-bareos Note Added: 0005375
2023-09-05 22:10 mschiff Note Added: 0005387
2023-09-06 10:22 bruno-at-bareos Note Added: 0005388
2023-09-06 13:48 mschiff Note Added: 0005389
2023-09-06 14:53 bruno-at-bareos Note Added: 0005390
2023-09-06 15:02 mschiff Note Added: 0005391
2023-09-06 16:36 arogge Note Added: 0005392
2023-09-12 11:37 bruno-at-bareos Assigned To => bruno-at-bareos
2023-09-12 11:37 bruno-at-bareos Status new => closed
2023-09-12 11:37 bruno-at-bareos Resolution open => fixed
2023-09-12 11:37 bruno-at-bareos Note Added: 0005411