 |
I can confirm I am also seeing the same issue with MariaDB 5.5 and the following bareos versions on Centos 7 bareos.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-bat.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-bconsole.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-client.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-common.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-database-common.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-database-mysql.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-database-tools.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-director.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-filedaemon.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-storage.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-storage-tape.x86_64 15.2.2-37.1.el7 @bareos_bareos-15.2 bareos-webui.noarch 15.2.2-41.1.el7 @bareos_bareos-15.2 |
|
|
Crontab this every morning or script it after backups etc echo '.bvfs_update' | bconsole If you never manually run update on path hierarchy cache, it will take forever when you first run a large restore job, since it runs that exact same command first. Google more for bvfs_update.. |
|
|
Hi I ran, echo '.bvfs_update' | bconsole IT made no difference to the time out in the restore pain of the webui. Additionally this is the error message i get emailed, once the restore does time out. 19-Jan 10:43 sydplbak001-dir: ERROR in output_formatter.c:633 Failed to send result as json. Maybe result message to long? Message length = 323 |
|
|
@reddrop: hege is not suggesting that it would be faster. Rather, he/she is suggesting automating running that command after backups are completed so that it does the time consuming work before you need to attempt a restore. It is an interesting workaround but does not resolve the actual issues. |
|
|
You can easily verify it's doing bvfs_update from the database, if you look at active sql queries. It's doing inserts into PathVisibility table. I don't know why this small speedup patch is not yet applied from master to 15.2 branch, it will help with bvfs_update speed. I've had it patched into my 15.2 for long time: https://github.com/bareos/bareos/commit/df52b6735cd39acf01b13a22f52e0287a76bb926 My suggestion resolves the problems you have if PathVisibility is the culprit, which most certainly is the case. I don't consider it as a "workaround", it does exactly the same things as webui. Updates the cache beforehand, so restores are possible, but running it beforehand you don't need to wait long for first time directory listing. Bareos should make this better documented and update the cache even automatically or optimize the whole thing another way. |
|
|
I don't think i was clear enough in my previous post: I ran echo '.bvfs_update' | bconsole several times and my webui is still timing out when trying list the files for selection. The query being run while the webui spins forever and says loading is: SELECT 'D', tmp.PathId, 0, tmp.Path, JobId, LStat, FileId FROM (SELECT PPathId AS PathId, '..' AS Pa... |
|
|
So, let's break it down step by step to track down the culprit. What the webui basically does first when calling the restore module is following: 1. Switch into api 2 mode (json) .api 2 2. Get all needed job ids for a normal complete restore. With the "all" option, all defined Filesets will be used. .bvfs_get_jobids jobid=10 all 3. Compute the directory cache for the jobs you got from previous command .bvfs_update jobid=1,2,3,10 4. Get the root or windows drives for representing our first level of the tree. .bvfs_lsdirs jobid=1,2,3,10 path= Every subtree/node underneath toplevel is lazy loaded by the use of .bvfs_lsdirs and .bvfs_lsfiles. Please excute those commands above in bconsole and verify if you encounter any problems during the initial construction of the tree in your environment and let us know. Also you can test those commands in api 0 to look for any differences. In addition you can try to list directories and files with lots of content in api mode 2 to look for any anomalies. By this we can better differentiate where to look further, director/db or webui. More details of the bvfs api are documented here: http://doc.bareos.org/master/html/bareos-developer-guide.html#sec:bvfs |
|
|
@frank: The patch hege linked to would fix the primary cause of this issue. The person who wrote that patch says it took his from 17-18 hours down to only 17 minutes, for a dataset of greater than 1 million files. For convenience, the relate post is here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=737821 The remaining issue would then be related to the webui hanging during processing. As for your instructions above, I'll give it a try, if I can figure out how to use the api 2 mode you mentioned. Thanks |
|
|
The linked patch was for MySQL so I'm not so sure that such drastic performance improvements are seen with PostgreSQL as the original initiator of the bugs says he is using. The reason its only in master is because we haven't had a chance to fully test what side effects its has on other databases and workloads. Most people also need to tune their database both PostgreSQL and any MySQL derivate as some problems just occur due to the query analyzer taking the non-optimal query plan. |
|
|
OK. I've run the commands frank indicated; however, I have no idea what I'm looking at and how to tell if the output is correct or not. Do you want me to post the output of a run against the linux client with a large number of files (assuming it will contain no sensitive information)? |
|
|
We are not so interested in the actual output more in response to certain commands and how fast they respond. Because I get the feeling its more a tuning problem then anything else. So its probably more a stopwatch test to see how long the different steps take and see where the bottleneck is if there is one. If we don't find any we can always zoom in on particular data but for a first test that is not strictly needed. It would be interesting however if some error is posted to have that data. b.t.w. for eveything only attach data to the bug so we can also remove it after processing etc. |
|
|
Here are my results (based on the steps frank posted): Step 1: immediate Step 2: approx 1/4 seconds Step 3: 29:29 minutes Step 4: immediate Note: My postgresql install is already tuned, as I noted in the root post. |
|
|
Ok so it spends its time in the update_path_hierarchy_cache() function so this has little to do with the webui. You could try the experimental master director with the changed query or run the current director with a debug level 500 which prints the the queries it fires against the database so we can exactly see what query it is. Other option would be to look in postgresql and see what query its executing. |
|
|
Minor typo. Step 3 should have read: 23:29 minutes |
|
|
Is there a way to delete that path hierarchy cache so I perform the additional tests you've requested, without having to perform a new full backup and compound the processing time? Other questions: Where do I get the experimental master director? I installed Bareos from the Ubuntu PPA distro. |
|
|
The cache is a cache so you don't have to do a new backup. It only builds a cache to speed up things which in your situation only seem to slow down the process. .bvfs_clear_cache clears the cache e.g. truncates the meta table. regarding experimental packages we publish those on http://download.bareos.org/bareos/experimental/nightly/ there are for most if not all Linux distributions packages there. Its build from the current master branch and after a internal function test published during the night around 04:30 CET. As the database schema is still the same it should just work. However if you have storage and filed on the same machine they also need to be upgraded as the shared libs are numbered by version. |
|
|
Thanks. I'll test and post my results sometime between now and tomorrow morning. |
|
|
Some good news: It was my fault. I ran pgtune and edited my postgresql.conf but forgot the "MB" after "wal_buffers = 16" and also forget to apply "effective_cache_size" setting. I discovered my error when building a VM to run the experimental build and conduct the tests. With those fixed, the .bvfs_update step completes in 30 seconds on both the production server and the VM I had setup for testing. Clearing the cache and choosing the client from the Restore page of the webui, it completes in 35 seconds. So, only marginally slower from the webui. The net result is that on a properly tuned Postresql performance is reasonable, and there is no perceptible performance difference between the current release and experimental builds. A RECOMMENDATION: With all that said, on restart, bareos-dir outputs a warning if the "max_connections" setting for Postgresql is below the "MaxConcurrentJobs" for bareos-dir. It would be great, given how critical the tuning it to bareos-dir performance, if bareos-dir would also warn if the database has not been tuned. |
|
|
Honestly I do think that is something for some external tool. There are quite some tuning tools for different databases which take into account the filling of the different tables etc. That is not something for a backup tool to try doing better. b.t.w. if you set it to 16 do you then get 16 x 4 Kb or 8 Kb ? As that would mean 64 or 128 Kb max. The 30 vs 35 seconds is not to bad as it also has to go through php into the javascript applet and then being rendered by your browser. |
|
|
"b.t.w. if you set it to 16 do you then get 16 x 4 Kb or 8 Kb ? As that would mean 64 or 128 Kb max." I'm not sure what you're referring to with that statement. Are you referring to the wal_buffers? If so, I have that set at 16MB. |
|
|
If my post wasn't supposed to changed the bug status, that was not intentional, as all I did was refresh the page and post my response. |
|
|
I was more interested in what tiny size you end up with when setting it to 16 without any multiplier like Mb/Gb etc. But I think we can close this bug as it was obviously mostly a tuning problem on the backend. |
|
|
According to the Postgresql documentation, "any positive value less than 32kB will be treated as 32kB." http://www.postgresql.org/docs/9.2/static/runtime-config-wal.html Yes the bug can be closed. I have not other issues related to this. Any other issues seem particular to MySQL queries, as noted earlier, and are likely best dealt with via a different bug report. Thank you for all your help. |
|
|
Closed as stated above. |
- Anonymous
