 |
I forgot to say that when the system is stable it takes no time to display a job list. Memory footprint is probably very low as well. real 0m0.062s user 0m0.010s sys 0m0.010s |
|
|
You have the following in your director config: Maximum Concurrent Jobs = 20 That means you cannot run more the 20 jobs concurrently keep in mind that every bconsole is a new Job too. There is a work-queue in the director that is sized using the above setting and it means that after it has reached 20 jobs it will only accept any new connection after one other has finished. First of all I think its kind of useless to query your system like this and especially by using so many bconsole sessions. Keep in mind that bconsole is a hollow program it does nothing more the display the user agent that the director runs. So you are probably just blowing up the director and as that stop accepting connections it seems to trigger some stuff in the bconsole that starts leaking memory in a frantic way. I think it makes much more sense to use NSCA probes in Nagios for these kind of monitoring which turn this whole monitoring into something passive and when the NSCA probe doesn't get the clearance the backup ran it will trigger an error in your monitoring. As to why bconsole blows up no idea but I think the code is not very robust when it comes to handling this resource starvation errors. You could do a kill -SEGV of one of the processes so it will invoke the traceback handler but for that to have any info you need to install the debug rpms and I don't know how that exactly works on centos. |
|
|
1. I didn't know bconsole was a Job. 2. Yes, I agree that the current monitoring system is brain dead. It has to be redone. However because you seem to know what kind of passive check would you run ? 3. For the kill -SEGV are you talking about debug symbols for bareos ? I can install : bareos-debuginfo.x86_64. Just to be sure I assume I will have to kill -SEGV `pidof bconsole` and not the director right. |
|
|
Yes you need to install the debug symbols otherwise there is not much to debug with the created core. It should also create a proper core file so we can do some postmortem debugging. And yes it has to be the bconsole as you showed that to use the large amount of memory. I think the dir is fine as it won't run more then 20 jobs concurrently anyway. As to the passive check how about create one in Nagios and then in the post backup script post via NSCA to Nagios that the Job succeeded ? When you put a freshness on such a passive check it will automatically trigger if your backups don't work for a longer time. e.g. if you expect a backup every day a freshness of 25 hours or so should work. Its been quite some time I used Nagios myself but ages ago that was a nice and elegant solution. |
|
|
If you can capture a core-file we can see how we can analyze that to get an idea what is eating all the memory. |
|
|
Hi, I can be wrong but I believe bconsole is trying to execute something it cannot find because everything is deployed in a /usr/lib/debug prefix. Is it something you wanted to do ? I am wondering because I believe you wanted to deploy it from / hence the .debug suffix to all binaries. If I have more time I will try to see what is it trying to start. Obviously for the moment I cannot get a coredump :) # rpm -q bareos-debuginfo bareos-debuginfo-13.2.1-81.1.el6.x86_64 # rpm -ql bareos-debuginfo |grep bconsole /usr/lib/debug/usr/sbin/bconsole.debug # ls -l /usr/lib/debug/usr/sbin/bconsole.debug -r-xr-xr-x 1 root root 86072 Sep 9 22:56 /usr/lib/debug/usr/sbin/bconsole.debug [/usr/lib/debug/usr/sbin]# /usr/lib/debug/usr/sbin/bconsole.debug -bash: /usr/lib/debug/usr/sbin/bconsole.debug: bad ELF interpreter: No such file or directory # strace /usr/lib/debug/usr/sbin/bconsole.debug execve("/usr/lib/debug/usr/sbin/bconsole.debug", ["/usr/lib/debug/usr/sbin/bconsole"...], [/* 21 vars */]) = -1 ENOENT (No such file or directory) dup(2) = 3 fcntl(3, F_GETFL) = 0x8402 (flags O_RDWR|O_APPEND|O_LARGEFILE) fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0408a01000 lseek(3, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) write(3, "strace: exec: No such file or di"..., 40strace: exec: No such file or directory ) = 40 close(3) = 0 munmap(0x7f0408a01000, 4096) = 0 exit_group(1) = ? # ls -l /usr/lib/debug/usr/sbin/bconsole.debug -r-xr-xr-x 1 root root 86072 Sep 9 22:56 bconsole.debug # ls -l /usr/sbin/bconsole -rwxr-xr-x 1 root root 37216 Sep 9 22:56 /usr/sbin/bconsole # uname -r 2.6.32-358.23.2.el6.x86_64 |
|
|
Re, I don't know why bconsole.debug is not working :/ # ls -l /usr/sbin/bconsole.debug -r-xr-xr-x 1 root root 86072 Dec 11 16:57 /usr/sbin/bconsole.debug # file /usr/sbin/bconsole* /usr/sbin/bconsole: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped /usr/sbin/bconsole.debug: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped # bconsole.debug -bash: /usr/sbin/bconsole.debug: bad ELF interpreter: No such file or directory # strace -s 1024 bconsole.debug execve("/usr/sbin/bconsole.debug", ["bconsole.debug"], [/* 21 vars */]) = -1 ENOENT (No such file or directory) <<<<<<<<<<<<<<<<< WHY ?? dup(2) = 3 fcntl(3, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE) fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f95647bb000 lseek(3, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) write(3, "strace: exec: No such file or directory\n", 40strace: exec: No such file or directory ) = 40 close(3) = 0 munmap(0x7f95647bb000, 4096) = 0 exit_group(1) = ? # strace -s 1024 bconsole 2>&1|head execve("/usr/sbin/bconsole", ["bconsole"], [/* 21 vars */]) = 0 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< no problem here. brk(0) = 0x1c89000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa934130000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/x86_64/libreadline.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls/x86_64", 0x7ffff42565c0) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/libreadline.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 open("/usr/lib64/x86_64/libreadline.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/usr/lib64/x86_64", 0x7ffff42565c0) = -1 ENOENT (No such file or directory) |
|
|
Maybe you can look at this: http://le-huy.blogspot.de/2011/01/using-debuginfo-packages-in-redhat.html but it seems the debug file is nothing more then a so called symbol table. e.g. normally the binaries are stripped so when you then start gdb it cannot really show any symbols in your binary only some decoded assembly. So you should just start the normal binary and get it to crash when its so big. Then it should create a corefile and put gdb on that showing the stacktrace using the symbol table in the .debug file. |
|
|
Hi, I finally have the time to look into it again. I have restarted nrpe so a lot of bconsole should be launched again. However it has not crashed or increased the load yet. That could be related to the fact I deleted a lot of volumes in the Catalog and on disk. I have 9228 media left and I removed 6659 media today. We will see how it goes but could this be related to a certain amount of rows in the media table maybe ? Does it sound possible ? I will keep you posted if I can reproduce the issue again. |
|
|
It finally happened again. I have a core file for : 5557 root 20 0 1716m 1.1g 140 D 0.3 7.1 0:08.91 bconsole You can download a 1.5GB core file via http://bogan.dmw.fr/~tom/bconsole.core.5557 Also note that my bareos-dir is currently using 4GB and I cannot connect with bconsole but it could be because it's running our backup jobs at the moment. 14765 bareos 20 0 4368m 8468 776 S 0.0 0.1 13:20.46 bareos-dir |
|
|
Some first impressions: - compress the core file next time with bzip2 ==> 1.5 Gb => 83 Kb. - Its seems from the above there is really a lot of data that compresses like hell. Doing a fast strings -a on the core already reveals one possible problem ==> /root/.bconsole_history e.g. the readline history file is going out of control that might account for quite some of the 1.5 Gb of ram. Just remove it as I guess its not usable anyway. - You seem to use the world famous one volume per job which is known to scale poorly with both Bacula and Bareos. - The last thing you say is that your dir does not respond that is more or less the same problem as why all the NRPE spawned bconsoles hang and that is that the director doesn't allow enough concurrent jobs (and as every console is a job in the director). So first analyze drop the history file and start a complete redesign of your backup environment this is never going to work, you are just way outside the design envelop of both Bacula and Bareos. Maybe there is also a bug somewhere but with these vast amount of data its unlikely we are ever going to find most of it. I will see if I can get any info from the core but I would be interested if you remove the history file if a single bconsole is already using a somewhat reasonable amount of memory, |
|
|
Hi Marco, My apologies for the big corefile, it didn't cross my mind that this thing would compress that much or compress at all. Sorry. About the .bconsole_history I emptied it. I will check if the problem is as worse. Thank you for looking into the corefile. I quickly tried to sort | uniq my bconsole.history and indeed there is a lot of crap in there probably due to check_bacula but a lot didn't make much sense. Anyway now it's empty I will keep an eye on it. About the redesign that's rather unfortunate. Could you explain why this is not going to work ? We do not use tapes at all. I find the idea of one volume per job very simple to understand and very simple to manage. I can understand Bareos/bacula is not designed for that but I didn't know that maybe you should state that somewhere. It is even explained in the Bacula documentation how to do it IIRC. How could we guess as a user that Bacula/Bareos is not designed for that ? I am sure a lot of people are using a Volume per Job because it's simple and because now it's published everywhere on the Internet (Blog, articles, howtos, etc). I think if you believe this is a wrong design you should write about it - if time allows - to explain why and what are the other solutions. Otherwise it will be worse and worse. Tapes are not dead but a lot of people backup D2D(2D) therefore the concept of volume recycling or fixed sizes and etc are maybe obsolete in that context, IMO. At the moment, I am disappointed that Bareos won't answer my requirements. We moved from our old Bacula to Bareos with this design in mind and now we have migrated more 200 servers to it. I am not going to roll back now but I will plan another migration based on your advises or with another software (I do NOT mean to offend here). At the moment it works OK if I remove our silly monitoring which I agree is badly design. RunJobs + NCSA are perfect for that, thanks for the hint. Please keep me posted about the design, what's wrong and how you would do it. Thank you. |
|
|
The console history had a million entries about deleting volumes which triggered me to think you were using one volume per job. Could it be that there is a permission problem on the bconsole history ? Normally the history should have a max of 200 entries and its truncated by the code but that doesn't seem to work. The problem with that is that is doesn't scale. The queries for volumes are sized on a tape based system e.g. that means a limited number of volumes. With the one volume per job the number of volumes explode (it works nicely for small setups but I wonder how well it will work larger setups.) Also you create new volumes all the time and that means the mediaid will increase all the time, that will end eventually (maybe not soon but that depends a bit on how many volumes you create a day.) About the documentation, that is work in progress as you know we got that by forking bacula and it has quite some work to be done there. Also bacula and bareos both have the same problem with a large number of volumes so in that sense bacula would run into the same problems eventually. My prefered setup is using fixed size virtual tapes and have them in a pool with automatic recycling and then use copy jobs to tape. We also changed the default config to use some saner defaults. Problem with fixed sized volumes is however the chance of things becoming corrupt and that is what I have seen a serious problem on Linux. I have been spoiled in that sense on Solaris with ZFS where that never happens anymore. Maybe in the future when BTRFS stabilizes or ZFS gets a real option that is also solved. My first change would be to use one volume per day, that limits the risks of corruption to one day. And it makes you not create a million volumes a day. About the real problem in the core, I guess it just crashes in the memory allocator as it ran out of memory. The smartalloc stuff from bacula will do a null pointer dereference as way to force a core dump (bit like an assert). Other then that if the design you now have makes you happy you may want to keep it but I think it will break eventually. |
|
|
What's the biggest number for a media/volume in the code ? With a PostgreSQL, I can see it's an integer so 2147483647 [1]. Ideally if we can, it would be good to change it to bigserial or a bigint. And in the code as an unsigned long ;) However at the moment I am doing a bit less than 300 volumes a day. If the max is only pgsql int type aka 2147483647 and not in bareos code then I still have 19611 years in front of me - at a rate of 300 volume a day minus the mediaID I already used. About the doco, I fully understand. I was not whinging about it. About limitations. I know that Bareos limitations come from Bacula. My point about changing software would be for something else that would allow me to have one volume per job. What everybody wants is backup concurrency (many SD otherwise you cannot write to the same volume IIRC) and the code design not tied to the tape design. I also understand that there is a lot work if you guys would like to move towards that direction. IMO, keeping a such design based on tape would be a mistake. Tapes will still exist but are less important these days. Backups have to be fast, reliable, and should autoclean on disk once the retention make them obsolete to free space for the new backups coming. I will think about your 1 volume per day. I am not sure it will suit our needs but it's a compromise. About the issue, I dunno why .bconsole_history was clobbered with a lot of deletes. What I can tell you is that the .bconsole_history file was probably longer than 200 lines. I stopped my sort | uniq -c after 5min. I believe sorting 200 lines should have taken less than 1 sec on my server. However, interestingly enough, at the moment, .bconsole_history is exactly 200 long. # wc -l .bconsole_history 200 .bconsole_history Weird... Anyway, if you don't hear from me in 3 weeks you can resolve this ticket. Either the problem stopped or I finally moved to nagios passive checks. I will do my best to ask you to resolve the ticket or to give new info about the issue. Thanks for your support. [1] : http://www.postgresql.org/docs/9.2/static/datatype-numeric.html |
- Anonymous
