View Issue Details

IDProjectCategoryView StatusLast Update
0000328bareos-coredirectorpublic2014-08-18 18:38
Reporternorbert.tobolski Assigned To 
PrioritynormalSeveritycrashReproducibilityalways
Status closedResolutionno change required 
PlatformCentos 6.5OSCentOSOS Version6
Summary0000328: Director crashes on NDMP backup
DescriptionUpdated to version 14.3.0 from bareos:master repo.

When starting NDMP Backup director crashes with this output:

Waiting for data... (interrupt to abort)BAREOS interrupted by signal 11: Segmentation violation
Kaboom! bareos-dir, srv-bhv-backup-dir got signal 11 - Segmentation violation. Attempting traceback.
Kaboom! exepath=/root
Calling: /root/btraceback /root/bareos-dir 19958 /var/lib/bareos
The btraceback call returned 255
Dumping: /var/lib/bareos/srv-bhv-backup-dir.19958.bactrace
srv-bhv-backup-dir: authenticate.c:104-375 Failed ASSERT: store->password.encoding == p_encoding_md5
execv: /root/btraceback failed: ERR=Permission denied
Steps To ReproduceThis happens everytime the ndmp backup is starting. When excluding this backup task, normal client backups over bareos-fd will work fine.
Additional InformationUpdated to version 14.3.0 as in the 13.2 version the storage daemon crashes when using ndmp.

Conenction to netapp system via bconsole -> status -> client works fine and shows me the available volumes.
TagsNo tags attached.

Relationships

related to 0000329 closed Add some safeguards against running a Native Backup Job on NDMP client or storage. 

Activities

norbert.tobolski

norbert.tobolski

2014-08-18 09:08

reporter  

ndmp-client.conf (2,074 bytes)
mvwieringen

mvwieringen

2014-08-18 11:09

developer   ~0000950

The problem you run into is a so called ASSERT which checks for a certain
condition being true. It seems something went wrong parsing your config
as the paired storage of a NDMP backup should point to a normal storage
type which will normally always have a md5 type password but in your config
that doesn't seem to be true.

What does a "show storage" say in bconsole ? That will print the data internally
stored back to human readable format.

I do see something fishy in your config however:

Storage {
  Name = Storage-Netapp01
  Address = 10.120.1.88
}

Storage {
  Name = NDMPFile-Netapp01
  Address = 10.120.1.50 # N.B. Use a fully qualified name here
  Port = 10001
}

You are not trying to use the NDMP storage on the NDMP device right ? As they
only thing that works is using the Bareos SD as NDMP storage node.

Maybe its a bit easier to understand your setup when you describe a bit what
you are trying to accomplish. e.g. what NDMP device, storage amount etc.,
norbert.tobolski

norbert.tobolski

2014-08-18 14:58

reporter   ~0000951

Hi,

we have the following setup.

1x NAS System called netapp01

1x Bareos-Backup (Director + Storage Daemon)

I changed the configuration for better understanding.

We try to backup the data on NAS netapp01 via ndmp to the storage NDMPFile-Netapp01 (ndmp) which is paired with normal bareos-storage called "Storage-Backup"

Connection between Bareos and netapp ndmp works with clear or md5 password.

Storage Daemon on Backup server is configured to listen on port 10001.

bareos-sd.conf:

Storage {
  Name = srv-bhv-backup-sd
  Maximum Concurrent Jobs = 20
  NDMP Enable = Yes
  NDMP Port = 10001

  # remove comment in next line to load plugins from specified directory
  Plugin Directory = /usr/lib64/bareos/plugins
}

Ndmp {
  Name = NDMPFile-Netapp01
  Username = ndmp
  Password = Ra8y7h5yZD72OcD
  AuthType = Clear
  Loglevel = 3
}

Device {
  Name = Storage-Backup
  Device Type = File
  Archive Device = /Bareos-Backup/BHV/Storage-Backup
  LabelMedia = yes
  Random Access = yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = yes
  Maximum Concurrent Jobs = 20
}


bares-dir.conf:

Client {
  Name = netapp01
  Address = netapp01.vanguard.de
  Port = 10000
  Catalog = MyCatalog
  File Retention = 90 days
  Job Retention = 6 months
  Protocol = NDMPv4
  Auth Type = clear # Clear == Clear Text, MD5 == Challenge protocol
  Username = "ndmp-backup" # username of the NDMP user on the DATA AGENT e.g. storage box being backuped.
  Password = "VDJMRhHH5R0rslOc" # password of the NDMP user on the DATA AGENT e.g. storage box being backuped.
}

Job {
  Name = Backup-netapp01
  Type = Backup
  Level = Full
  Client = netapp01
  FileSet = netapp01-Fileset
  Schedule = netapp01-Schedule
  Storage = NDMPFile-Netapp01
  Pool = netapp01-Pool
  Messages = Standard
}

### NetApp01 Paired Storage
Storage {
  Name = Storage-Backup
  Password = Tv3JM3pbi9Wv5wWJDDlObEGBTXMP4+imn9SCbVDjenj5
  Address = 10.110.0.30
  SDPort = 9103
  Device = Storage-Backup
  Media Type = File
  Maximum Concurrent Jobs = 20
}

Storage {
  Name = NDMPFile-Netapp01
  Address = 10.110.0.30 # N.B. Use a fully qualified name here
  Port = 10001
  Protocol = NDMPv4
  Auth Type = Clear
  Username = ndmp
  Password = Ra8y7h5yZD72OcD
  Device = NDMPFile-Netapp01
  Media Type = File
  PairedStorage = Storage-Backup
  Maximum Concurrent Jobs = 20
}


Thank you, for your help.
mvwieringen

mvwieringen

2014-08-18 15:08

developer   ~0000952

What does show storage say in bconsole ?
norbert.tobolski

norbert.tobolski

2014-08-18 15:15

reporter   ~0000953

Hi,

here is the output of console -> show storage:
...
...
...
storage {
  name = "Storage-Netapp01"
  address = "10.110.0.30"
  password = "[md5]3071f49a99129c0629ea574a4e6b8016"
  device = "Storage-Netapp01"
  mediatype = "File"
  maximumconcurrentjobs = 20
}

storage {
  name = "NDMPFile-Netapp01"
  protocol = NDMPV4
  authtype = Clear
  address = "10.110.0.30"
  port = 10001
  username = "ndmp"
  password = "Ra8y7h5yZD72OcD"
  device = "NDMPFile-Netapp01"
  mediatype = "File"
  maximumconcurrentjobs = 20
  pairedstorage = "Storage-Netapp01"
mvwieringen

mvwieringen

2014-08-18 15:36

developer   ~0000954

Ok that output shows that it parsed things right as the native password
for the storage daemon is stored as md5. So I get the feeling something
goes wrong trying to determine your paired storage for whatever reason.

Can you run the bareos-dir as bareos-dir -f -d 200 > log 2>&1 and attach
that log to the bug report.
norbert.tobolski

norbert.tobolski

2014-08-18 16:04

reporter  

bareos-debug.log (6,085 bytes)
norbert.tobolski

norbert.tobolski

2014-08-18 16:04

reporter   ~0000957

enclosed you can find the log file, when running in debug mode bareos isn't able to connect to the database.
mvwieringen

mvwieringen

2014-08-18 16:12

developer   ~0000958

Without a debug log I cannot do any more remote debugging. Are you normally
running the director under the bareos user ? If so you should su to the bareos
user (probably have to enable shell login by changing the shell in /etc/passwd
or use 'su -s /bin/bash') It seems postgresql normally authenticates using
a user trust relationship probably.
norbert.tobolski

norbert.tobolski

2014-08-18 16:22

reporter  

bareos-debug-new.log (18,391 bytes)
norbert.tobolski

norbert.tobolski

2014-08-18 16:23

reporter   ~0000959

ok thanks, changing shell for bareos works.

I start the backup job and you can see the crash in log file.
mvwieringen

mvwieringen

2014-08-18 16:42

developer   ~0000960

Ok that it gives the ASSERT is good as it shows a config error you are trying
to run a native backup job on an NDMP client.

Try adding 'Protocol = NDMP' to the Job definition and it will probably run
better. I will add some safeguards in the code to reject running a Native Job
against a NDMP client and/or NDMP storage.

But don't know when we have the time to work on this extra safeguards.
norbert.tobolski

norbert.tobolski

2014-08-18 18:08

reporter   ~0000961

Hi,

that was the issue. Sorry for the circumstances.

We have a new problem, but i opened a new ticket for this.

You can close this ticket.

Thanks for the fast help.

Issue History

Date Modified Username Field Change
2014-08-18 09:08 norbert.tobolski New Issue
2014-08-18 09:08 norbert.tobolski File Added: ndmp-client.conf
2014-08-18 11:09 mvwieringen Note Added: 0000950
2014-08-18 11:09 mvwieringen Status new => feedback
2014-08-18 14:58 norbert.tobolski Note Added: 0000951
2014-08-18 14:58 norbert.tobolski Status feedback => new
2014-08-18 15:08 mvwieringen Note Added: 0000952
2014-08-18 15:08 mvwieringen Priority high => normal
2014-08-18 15:08 mvwieringen Status new => feedback
2014-08-18 15:15 norbert.tobolski Note Added: 0000953
2014-08-18 15:15 norbert.tobolski Status feedback => new
2014-08-18 15:36 mvwieringen Note Added: 0000954
2014-08-18 15:36 mvwieringen Status new => feedback
2014-08-18 16:04 norbert.tobolski File Added: bareos-debug.log
2014-08-18 16:04 norbert.tobolski Note Added: 0000957
2014-08-18 16:04 norbert.tobolski Status feedback => new
2014-08-18 16:12 mvwieringen Note Added: 0000958
2014-08-18 16:12 mvwieringen Status new => feedback
2014-08-18 16:22 norbert.tobolski File Added: bareos-debug-new.log
2014-08-18 16:23 norbert.tobolski Note Added: 0000959
2014-08-18 16:23 norbert.tobolski Status feedback => new
2014-08-18 16:42 mvwieringen Note Added: 0000960
2014-08-18 16:42 mvwieringen Status new => feedback
2014-08-18 16:49 mvwieringen Relationship added related to 0000329
2014-08-18 18:08 norbert.tobolski Note Added: 0000961
2014-08-18 18:08 norbert.tobolski Status feedback => new
2014-08-18 18:38 mvwieringen Status new => closed
2014-08-18 18:38 mvwieringen Resolution open => no change required