View Issue Details

IDProjectCategoryView StatusLast Update
0001331bareos-core[All Projects] storage daemonpublic2021-07-27 17:33
ReporterkaushalshriyanAssigned To 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
PlatformLinuxOSCentOS OS Version7.9.2009 (Core)
Product Version20.0.1 
Fixed in Version 
Summary0001331: Backup data to AWS S3 bucket using the BareOS utility
DescriptionI am running bareos 20.0 on CentOS Linux release 7.9.2009 (Core). I have installed bareos 20.0 using https://download.bareos.org/bareos/release/20/CentOS_7/x86_64/ I am trying to push the backup to AWS S3 by following the document https://docs.bareos.org/TasksAndConcepts/StorageBackends.html. When I execute the run process I am encountering

23-Mar 10:28 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)

23-Mar 10:28 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok

#cat /etc/bareos/bareos-sd.d/device/droplet/aws.profile
# Generic host, but can't access buckets younger than 24h:
# Region specific host name. Can access also new buckets.
host="s3.amazonaws.com"
use_https = "true"
backend = "s3"
aws_region = "ap-south-1"
aws_auth_sign_version = "4"
access_key = "XXXXXXXXXXXXXXXXXXX"
secret_key = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
pricing_dir = ""

cat /etc/bareos/bareos-sd.d/device/AWS_S3_1-00.conf

Device {
  Name = "AWS_S3_1-00"
  Media Type = "S3_Object1"
  Archive Device = "AWS S3 Storage"
  Device Type = droplet
  Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/aws.profile,bucket=droplet-bareos,chunksize=100M"
  Label Media = yes # Lets Bareos label unlabeled media
  Random Access = yes
  Automatic Mount = yes # When device opened, read it
  Removable Media = no
  Always Open = no
  Maximum Concurrent Jobs = 1
}

cat /etc/bareos/bareos-dir.d/storage/S3_Object.conf

Storage {
  Name = "S3_Object"
  Address = "krithilinux" # N.B. Use a fully qualified name here (do not use "localhost" here).
  Password = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  Maximum Concurrent Jobs = 1
  Collect Statistics = yes
  Allow Compression = Yes
  Device = "AWS_S3_1-00"
  Media Type = "S3_Object1"
}

#rpm -qa |grep bareos
bareos-database-postgresql-20.0.1-3.el7.x86_64
bareos-client-20.0.1-3.el7.x86_64
bareos-storage-droplet-20.0.1-3.el7.x86_64
bareos-common-20.0.1-3.el7.x86_64
bareos-database-common-20.0.1-3.el7.x86_64
bareos-database-tools-20.0.1-3.el7.x86_64
bareos-filedaemon-20.0.1-3.el7.x86_64
bareos-tools-20.0.1-3.el7.x86_64
bareos-20.0.1-3.el7.x86_64
bareos-webui-20.0.1-3.el7.x86_64
bareos-bconsole-20.0.1-3.el7.x86_64
bareos-director-20.0.1-3.el7.x86_64
bareos-storage-20.0.1-3.el7.x86_64
Steps To ReproduceI have attached the screenshot for reproducing the issue executing in http://192.168.0.177/bareos-webui/job/details/
Additional Information==> bareos.log <==
23-Mar 16:36 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:36 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:36 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:36 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:37 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:37 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:37 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:37 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:38 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:38 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:38 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:38 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:39 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:40 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:40 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:40 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:40 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:41 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:41 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:41 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:41 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:392: init_ssl_conn: SSL connect error: 0 (error:00000005:lib(0):func(0):DH lib)
23-Mar 16:42 bareos-sd: ERROR in backends/droplet_device.cc:113 error: ../../../../../core/src/droplet/libdroplet/src/conn.c:395: init_ssl_conn: SSL certificate verification status: 0: ok
23-Mar 16:42 bareos-sd JobId 44: Warning: stored/label.cc:389 Open device "AWS_S3_1-00" (AWS S3 Storage) Volume "Full-0003" failed: ERR=stored/dev.cc:734 Could not open: AWS S3 Storage/Full-0003, ERR=Success

23-Mar 16:42 bareos-sd JobId 44: Warning: stored/mount.cc:276 Open device "AWS_S3_1-00" (AWS S3 Storage) Volume "Full-0003" failed: ERR=stored/dev.cc:734 Could not open: AWS S3 Storage/Full-0003, ERR=Success

23-Mar 16:42 apigeeapicrafterprodbackup-fd JobId 44: Fatal error: filed/dir_cmd.cc:2697 Bad response to Append Data command. Wanted 3000 OK data
, got 3903 Error append data

23-Mar 16:42 bareos-dir JobId 44: Fatal error: Director's comm line to SD dropped.
23-Mar 16:42 bareos-dir JobId 44: Error: Bareos bareos-dir 20.0.1 (02Mar21):
  Build OS: CentOS Linux release 7.6.1810 (Core)
  JobId: 44
  Job: apigeebackup.2021-03-23_16.17.19_08
  Backup Level: Full
  Client: "apigeeapicrafterprodbackup" 20.0.0 (16Dec20) CentOS Linux release 7.6.1810 (Core),redhat
  FileSet: "LinuxConfig" 2021-03-22 19:26:06
  Pool: "Full" (From command line)
  Catalog: "MyCatalog" (From Client resource)
  Storage: "S3_Object" (From Job resource)
  Scheduled time: 23-Mar-2021 16:17:19
  Start time: 23-Mar-2021 16:17:21
  End time: 23-Mar-2021 16:42:58
  Elapsed time: 25 mins 37 secs
  Priority: 10
  FD Files Written: 0
  SD Files Written: 0
  FD Bytes Written: 0 (0 B)
  SD Bytes Written: 0 (0 B)
  Rate: 0.0 KB/s
  Software Compression: None
  VSS: no
  Encryption: no
  Accurate: no
  Volume name(s):
  Volume Session Id: 1
  Volume Session Time: 1616496336
  Last Volume Bytes: 0 (0 B)
  Non-fatal FD errors: 1
  SD Errors: 0
  FD termination status: Fatal Error
  SD termination status: Error
  Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com
  Job triggered by: User
  Termination: *** Backup Error ***
Tagsdroplet
bareos-master: impact
bareos-master: action
bareos-19.2: impact
bareos-19.2: action
bareos-18.2: impact
bareos-18.2: action
bareos-17.2: impact
bareos-17.2: action
bareos-16.2: impact
bareos-16.2: action
bareos-15.2: impact
bareos-15.2: action
bareos-14.2: impact
bareos-14.2: action
bareos-13.2: impact
bareos-13.2: action
bareos-12.4: impact
bareos-12.4: action

Activities

kaushalshriyan

kaushalshriyan

2021-03-23 14:59

reporter  

bareossslerror.png (649,167 bytes)
steptoreproduce.png (611,432 bytes)
ideacloud

ideacloud

2021-03-30 06:25

reporter   ~0004107

I have the same problem. No AWS connections using the plugin work correctly. They all show the same error in
perhallenborg

perhallenborg

2021-06-02 11:46

reporter   ~0004134

I also have the same problem.

02-Jun 10:49 bareos-sd JobId 16: Warning: stored/label.cc:389 Open device "AWS_S3_1-00" (AWS S3 Storage) Volume "S3_Full-0003" failed: ERR=stored/dev.cc:734 Could not open: AWS S3 Storage/S3_Full-0003, ERR=Success

02-Jun 10:50 bareos-sd JobId 16: Warning: stored/label.cc:389 Open device "AWS_S3_1-00" (AWS S3 Storage) Volume "S3_Full-0003" failed: ERR=stored/dev.cc:734 Could not open: AWS S3 Storage/S3_Full-0003, ERR=Success

02-Jun 10:50 bareos-sd JobId 16: Warning: stored/mount.cc:276 Open device "AWS_S3_1-00" (AWS S3 Storage) Volume "S3_Full-0003" failed: ERR=stored/dev.cc:734 Could not open: AWS S3 Storage/S3_Full-0003, ERR=Success

02-Jun 10:50 bareos-sd JobId 16: Please mount append Volume "S3_Full-0003" or label a new one for:
    Job: S3_backup-bareos-fd.2021-06-02_10.38.04_45
    Storage: "AWS_S3_1-00" (AWS S3 Storage)
    Pool: S3_Full
    Media type: S3_Object1
02-Jun 10:56 bareos-sd JobId 16: Warning: stored/mount.cc:276 Open device "AWS_S3_1-00" (AWS S3 Storage) Volume "S3_Full-0003" failed: ERR=stored/dev.cc:734 Could not open: AWS S3 Storage/S3_Full-0003, ERR=Success


I can mount via console but not label:

*mount
The defined Storage resources are:
     1: File
     2: S3_Object
Select Storage resource (1-2): 2
Connecting to Storage daemon S3_Object at bareos-test:9103 ...
3001 OK mount requested. Device="AWS_S3_1-00" (AWS S3 Storage)
*label
The defined Storage resources are:
     1: File
     2: S3_Object
Select Storage resource (1-2): 2
Enter new Volume name: S3_Full-0004
Defined Pools:
     1: Scratch
     2: S3_Incremental
     3: S3_Full
     4: S3_Differential
     5: Incremental
     6: Full
     7: Differential
Select the Pool (1-7): 3
Connecting to Storage daemon S3_Object at bareos-test:9103 ...
Sending label command for Volume "S3_Full-0004" Slot 0 ...
3910 Unable to open device ""AWS_S3_1-00" (AWS S3 Storage)": ERR=stored/dev.cc:734 Could not open: AWS S3 Storage/S3_Full-0004, ERR=Success

Label command failed for Volume S3_Full-0004.
Do not forget to mount the drive!!!


Centos 7.4, latest bareos
perhallenborg

perhallenborg

2021-06-02 17:00

reporter   ~0004135

Last edited: 2021-06-03 09:15

View 2 revisions

Problem solved. Added AWS policy "AmazonS3FullAccess" to the bucket and it connected and tried to label and write to volumes. But no volumes were written:

105 2021-06-02 17:04:56 bareos-sd JobId 19: End of medium on Volume "S3_Full-0015" Bytes=104,832,224 Blocks=1,625 at 02-Jun-2021 17:04.
104 2021-06-02 17:04:56 bareos-sd JobId 19: Error: stored/block.cc:822 Write error on fd=0 at file:blk 0:104832223 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=No such file or directory.
103 2021-06-02 17:04:56 bareos-sd JobId 19: Error: stored/block.cc:803 Write error at 0:104832223 on device "AWS_S3_1-00" (AWS S3 Storage). ERR=No such file or directory.
102 2021-06-02 17:04:30 bareos-sd JobId 19: New volume "S3_Full-0015" mounted on device "AWS_S3_1-00" (AWS S3 Storage) at 02-Jun-2021 17:04.
101 2021-06-02 17:04:30 bareos-sd JobId 19: Wrote label to prelabeled Volume "S3_Full-0015" on device "AWS_S3_1-00" (AWS S3 Storage)
100 2021-06-02 17:04:30 bareos-sd JobId 19: Labeled new Volume "S3_Full-0015" on device "AWS_S3_1-00" (AWS S3 Storage).
99 2021-06-02 17:04:30 bareos-dir JobId 19: Created new Volume "S3_Full-0015" in catalog.

No complaints on permissions or likewise. Just nothing written.

Mc.Sim

Mc.Sim

2021-06-03 09:53

reporter   ~0004136

@perhallenborg
Have you created the bucket `droplet-bareos` before running the task?
One suggestion: try to test with these options (iothreads and retries):
`Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/aws.profile,bucket=droplet-bareos,chunksize=100M,iothreads=0,retries=1"`
perhallenborg

perhallenborg

2021-06-03 10:03

reporter   ~0004137

Bucket created, but with another name. I can connect and write to the bucket from my Mac, as a mounted device. So the keys and secrets work. And Bareos doesn't complain about not being connected, as far as I can see. Tried also with iothreads=2
Tried nightly build Bareos 21 but same issue.
Mc.Sim

Mc.Sim

2021-06-03 10:27

reporter   ~0004138

So, the name of the backet at AWS-side are matched to the Bareos settings name. Correct?
When I debug my S3 storage, it is useful to enable logging on the AWS side.
what does bconsole show you, if you ask something like:
*show storage="AWS_S3_1-00"
*status storage="AWS_S3_1-00"
?
perhallenborg

perhallenborg

2021-06-03 10:33

reporter   ~0004139

Buckets are a match. Tried loggin on AWS side but it is empty.
Config was taken directly from Bareos wiki.

Enter a period (.) to cancel a command.
*show storage="AWS_S3_1_00"
storage resource AWS_S3_1_00 not found.
You have messages.
*status storage="AWS_S3_1_00"
Storage resource "AWS_S3_1_00": not found
The defined Storage resources are:
1: File
2: S3_Object
Select Storage resource (1-2): 2
Connecting to Storage daemon S3_Object at bareos-test.adnet.adlibris.se:9103

bareos-sd Version: 21.0.0~pre540.f3597a14d (01 June 2021) CentOS Linux release 7.9.2009 (Core)
Daemon started 03-Jun-21 10:13. Jobs: run=1, running=0, pre-release version binary
 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 bwlimit=0kB/s

Running Jobs:
No Jobs running.
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
 JobId Level Files Bytes Status Finished Name
===================================================================
    23 Full 570 58.88 M Error 03-Jun-21 08:41 S3_backup-bareos-fd
    24 Full 570 58.88 M Cancel 03-Jun-21 08:52 S3_backup-bareos-fd
    25 Incr 0 0 Cancel 03-Jun-21 09:06 S3_backup-bareos-fd
    26 Full 0 0 Cancel 03-Jun-21 09:10 S3_backup-bareos-fd
    27 Incr 0 0 Cancel 03-Jun-21 09:20 S3_backup-bareos-fd
    28 Incr 0 0 Error 03-Jun-21 09:23 S3_backup-bareos-fd
    29 Full 54,796 2.424 G Cancel 03-Jun-21 09:26 S3_backup-bareos-fd
    30 Incr 16 2.967 M Error 03-Jun-21 10:00 S3_backup-bareos-fd
    31 Full 61,355 2.524 G Error 03-Jun-21 10:03 S3_backup-bareos-fd
    32 Full 61,355 2.524 G Error 03-Jun-21 10:26 S3_backup-bareos-fd
====

Device status:

Device "AWS_S3_1-00" (AWS S3 Storage) is not open.
Backend connection is working.
No pending IO flush requests.
==
====

Used Volume status:
====
Mc.Sim

Mc.Sim

2021-06-03 10:53

reporter   ~0004140

Ahhh, *show storage="AWS_S3_1-00" should be *show storage="S3_Object"

> Backend connection is working.

Tell u that bareos SD can connect.

That is mine Device configuration

```
#cat storage/bareos-sd.d/device/WASABI_ObjectStorage.conf
Device {
  Name = WASABI_ObjectStorage
  Media Type = S3_Object1
  Archive Device = WASABI Object Storage
  Device Options = "profile=/etc/bareos/bareos-sd.d/device/droplet/wasabi_us-east-1.profile,bucket=i-clf1-bareos,chunksize=100M,iothreads=7,retries=0"
  Device Type = droplet
  Label Media = yes # lets Bareos label unlabeled media
  Random Access = yes
  Automatic Mount = yes # when device opened, read it
  Removable Media = no
  Always Open = no
  Maximum Concurrent Jobs = 1
}
# cat storage/bareos-sd.d/device/droplet/wasabi_us-east-1.profile
host = s3.us-east-1.wasabisys.com:443
use_https = true
backend = s3
aws_region = us-east-1
aws_auth_sign_version = 4
access_key = ""
secret_key = ""
pricing_dir = ""
#

```

I can see only one difference: the host contain the 443 port.
> host = s3.us-east-1.wasabisys.com:443

Try to use it.
perhallenborg

perhallenborg

2021-06-03 13:04

reporter   ~0004141

I already do, doesn't work without it.

host = <bucketname>.s3-eu-north-1.amazonaws.com:443 # This parameter is only used as baseurl and will be prepended with bucket and location set in device resource to form correct url
use_https = true
access_key = xx
secret_key = xx
pricing_dir = "/etc/bareos/pricing" # If not empty, an droplet.csv file will be created which will record all S3 operations.
backend = s3
aws_auth_sign_version = 4 # Currently, AWS S3 uses version 4. The Ceph S3 gateway uses version 2.
aws_region = eu-north-1

Issue History

Date Modified Username Field Change
2021-03-23 14:59 kaushalshriyan New Issue
2021-03-23 14:59 kaushalshriyan Tag Attached: droplet
2021-03-23 14:59 kaushalshriyan File Added: bareossslerror.png
2021-03-23 14:59 kaushalshriyan File Added: steptoreproduce.png
2021-03-30 06:25 ideacloud Note Added: 0004107
2021-06-02 11:46 perhallenborg Note Added: 0004134
2021-06-02 17:00 perhallenborg Note Added: 0004135
2021-06-03 09:15 perhallenborg Note Edited: 0004135 View Revisions
2021-06-03 09:53 Mc.Sim Note Added: 0004136
2021-06-03 10:03 perhallenborg Note Added: 0004137
2021-06-03 10:27 Mc.Sim Note Added: 0004138
2021-06-03 10:33 perhallenborg Note Added: 0004139
2021-06-03 10:53 Mc.Sim Note Added: 0004140
2021-06-03 13:04 perhallenborg Note Added: 0004141