Storage architecture optical backups

by on under sysadmin
19 minute read

Backing up data using optical media

  • When looking to purchase a Blu-ray burner watch out for Blu-ray drives which only read BD but write DVD and CD.

Burning software

Brasero

  • Will create a MD5 checksum for you.
  • Works on CD-R and DVD-R but not BD-R discs.
  • Enabling the burn the image directly without saving it to disc option will skip making a local ISO cache before burning to disc in /tmp
  • CD-R write speeds are one of the fastest I have tested so far as high as 34x. Using Pioneer, VM passthrough Windows host -> Linux guest and Sony 700 MB.

K3B

  • You need to be careful with adding files up to the limit as you may get “mkisofs crashed” error.
  • Long File-Name Support for Burning Data Discs in Ubuntu with K3b
  • Works on CD-R, DVD-R and BD-R discs.
  • You will need to create your own checksum.
  • Enabling Create image option will make a local ISO cache before burning to disc in /tmp.
  • You will need to manually enable verify disc. This should really be on by default.
  • DB-R write speeds only seen as high as 3x but have seen it dip to 1.7x. Using Pioneer, VM passthrough Windows host -> Linux guest and Verbatim 25 GB 6x.

Windows Explorer

ImgBurn

  • Works on CD-R?, DVD-R and BD-R discs.
  • Gives some automatic file system recommendation that better suits MP3 BD-R data discs.
  • You will need to create your own checksum. TODO: Check settings further.
  • Produces a graph output which can be read in DVDINFOPro.
  • Possible to span large image across CD’s?
  • How to split a large file in 2 small ones
  • DB-R 8.1x write speed. Uses 318 Mbps pulling from a NAS at those write speeds. Using Pioneer, VM passthrough Windows host -> Windows guest and Verbatim 25 GB 6x.
  • DVD-R 13.4x write speed. Using Pioneer, VM passthrough Windows host -> Windows guest and Philips 4.7 GB 12x.

Nero

Write speed tests

Id Date Disc brand Disc type Write software File source IO type Project type File type Write device Write speed max Write speed average Result
16 2020-12-07 Philips DVD-R K3B USB3 Linux direct Data MP4 DU-8A5LH     OK
15 2020-12-06 Philips DVD-R Brasero USB3 Linux direct Data MP3 SU-208GB     OK
14 2020-12-07 Sony CD-R Brasero USB3 Linux direct Data MP3 SU-208GB     OK
13 2020-12-14 Sony CD-R Windows Local cache Windows direct Data MP3 BDR-212M N/A N/A OK
12 2020-12-14 Sony CD-R Brasero Local cache Windows -> VM passthrough -> Linux Data MP3 BDR-212M 34x 24.9x OK
11 2020-12-14 Philips DVD-R Brasero Local cache Windows -> VM passthrough -> Linux Data MP3 BDR-212M 16x 10.9x OK
10 2020-12-14 Philips DVD-R K3B Local cache Windows -> VM passthrough -> Linux Data MP3 BDR-212M 15x   OK
9 2020-12-14 Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data MP3 BDR-212M 13.4x   OK
8 2020-12-14 Sony CD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data MP3 BDR-212M N/A N/A ERROR
7 2020-12-14 Sony CD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data MP3 BDR-212M N/A N/A ERROR
6 2020-12-14 Verbatim BD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data MP3 BDR-212M 8.1x   OK
5 2020-12-12 Verbatim BD-R K3B Remote SMB Windows -> VM passthrough -> Linux Data MP3 BDR-212M 2.4x   OK
4 2020-12-12 Verbatim BD-R K3B Local cache Live Linux USB direct Data (FS Linux only) MP4/MOV BDR-212M 3.3x   OK
3 2020-12-12 Verbatim BD-R Windows Local cache Windows direct Data MP4/MOV BDR-212M N/A N/A OK
2 2020-12-12 Verbatim BD-R K3B Remote SMB Windows -> VM passthrough -> Linux Data MP4, MOV, JPG, TNL, MTS BDR-212M     ERROR Ran out of disc space
1 2020-12-12 Verbatim BD-R K3B Local cache Live Linux USB direct Data MP4, MOV BDR-212M 3.3x   OK
  • BDR-212M is a Pioneer drive.
  • SU-208GB is a TSSTcorp drive.
  • DU-8A5LH is a drive commonly found in DELL laptops.
  • Write software Windows refers to Windows Explorer but is kept short for brevity.
  • Some items maybe missing write speed if the software doesn’t show it like in the case of Windows Explorer, otherwise I may have not recorded the speed at the time.

Compatibility/playback tests

WriteId Read device Result
1 PS3 OK
1 Xbox One S Error 0x80820002
3 PS3 OK
3 Xbox One S Error 0x80820002
4 PS3 OK
4 Xbox One S Error 0x80820002
5 PS3 OK
5 Xbox One S Error 0x80820002
6 PS3 OK
6 Xbox One S Error 0x80820002
9 PS3 OK
9 Xbox One S Error 0x80820002
9 LG DVD/CD Player DVX392H OK
10 PS3 OK
10 Xbox One S Error 0x80820002
10 LG DVD/CD Player DVX392H OK
11 PS3 OK
11 Xbox One S Error 0x80820002
11 LG DVD/CD Player DVX392H OK
12 PS3 OK
12 Xbox One S Error 0x80820002
12 LG DVD/CD Player DVX392H OK
13 PS3 OK
13 Xbox One S Error 0x80820002
13 LG DVD/CD Player DVX392H OK
14 PS3 OK
14 Xbox One S Error 0x80820002
14 LG DVD/CD Player DVX392H OK
15 PS3 OK
15 Xbox One S Error 0x80820002
15 LG DVD/CD Player DVX392H OK
16 PS3 OK
16 Xbox One S Error 0x80820002
16 LG DVD/CD Player DVX392H Err
  • 1, 3 and 4 the PS3 wouldn’t find the MOV files but this is a separate issue relating to PS3 codecs not a medium issue. It did however play MP4 just fine.

Optical medium models

Verbatim

The Verbatim product line can be a bit confusing and their main website could do with updating. The EU website is a bit more modern but lacking clear comparisons. Luckily I finally found the information I was after here: Verbatim Optical Media July 2020 Brochure

  • Annoyingly you cannot get the same “SURFACE” across all medium, in addition it’s specific to the “CONFIGURATION” too.
  • “Wide Printable No-ID” seems to be the only “SURFACE” which is common across nearly all medium. Second to that would be “Matt Silver” of which CD-R and BD-R doesn’t come in.
  • Their professional line is all “No-ID” annoyingly. They intend you to print them however.
  • The “CRYSTAL” surface is specific to CD-R AZO only.
  • Be warned that “PART NO” 43791 under the “Datalife” branding is “Non-AZO”. “PART NO” 97693 under the “Life Series” is also “Non-AZO”.
  • > Archival Life: Extra Protection - up to 40 years; AZO - up to 100 years.

Below are some “consistent” “SURFACE” configurations you could get. Otherwise it’s a “mix” of surfaces for decent prices:

Type Configuration Surface Product name Product Id
Consistent 50pk Spindle Wide Printable No-ID CD-R AZO 43438
Consistent 50pk Spindle Wide Printable No-ID DVD-R AZO 43533
Consistent 50pk Spindle Wide Printable No-ID BD-R Datalife 43812
Consistent 25pk Spindle Wide Printable ID CD-R AZO 43439
Consistent 25pk Spindle Wide Printable ID DVD-R AZO 43538
Mixed 50pk Spindle Crystal CD-R AZO 43343
Mixed 50pk Spindle Matt Silver DVD-R AZO 43548
Mixed 50pk Spindle Wide Printable No-ID BD-R Datalife 43812
Supplementary 25pk Spindle Matt Silver DVD+R DL AZO 43757
Supplementary 25pk Spindle Wide Printable No-ID DVD+R DL AZO 43667
Supplementary 10pk Spindle White/Blue BD-R DL 43746

Technologies

  • “MediDisk”
  • “Datalife”
  • “HARD COAT”
  • “SUPER HARD COAT”
  • “AZO”
  • “SERL”
  • “EXTRA PROTECTION”
  • “M.A.B.L”
  • “CRYSTAL”
  • “M-DISC”

Surfaces

  • “Mat Silver”
  • “Inkjet Printable”
  • “White/Blue”
  • “Wide Printable No-ID”
  • “Wide Print No-ID” (inconsistency)
  • “Wide Printable ID”
  • “5 Colours”
  • “Colour” (possible inconsistency)
  • “Crystal”
  • “Extra Protection”
  • “Silver”
  • “Thermal No-ID”
  • “DataLifePlus Wide Thermal No-ID”
  • “DataLifePlus Wide Printable No-ID”
  • “DataLifePlus Waterproof Printable No-ID”

Philips

Can’t seem to find a data sheet or product brochure unfortunately. So no way to quantify what a “very long archive lifespan” is.

File systems

  • K3B Linux/Unix only seems to read back just fine on Windows 10. It looks to be ISO 9660 Level 3 + Rock Ridge extensions.
  • K3B Linux/Unix + Windows is ISO 9660 Level 3 + Rock Ridge extensions + Joliet extensions.
  • ISO 1999?
  • ISO 9660 file system
  • Joliet extension to ISO 9660.
  • Rock Ridge extension to ISO 9660.
  • UDF file system
  • UDF with K3B
  • ImgBurn ISO 9660 + Joliet for MP3 data disks.
  • ImgBurn UDF 1.02
  • ImgBurn UDF + ISO 9660 - Bridge disc TODO: Same as UDF structures on K3B?
  • Brasero uses ISO 9660 Level 2? But will switch to Level 3 when you add files over 2 GB and you accept the warning message.

Scripts

ffmpeg

  • How can I use ffmpeg to split MPEG video into 10 minute chunks?
  • Below is a adaptation I used for DVD splits of long 6 hour mp4 video files. Some work is needed still since this causes issues on the time stamps.
  • One downside with this is using this for data disc project wont make use of the disks full capacity as the disc can fit more mp4 runtime than 120 mins in DVD VIDEO_TS mode.
  • 1: ffmpeg -i "$1" -c copy -map 0 -segment_time 01:18:00 -f segment %03d"$1"
  • That creates a file around: 3.6 GB (3604783722 bytes) in size although it will depend how compressed your source file is.
  • 2: ffmpeg -i "$1" -c:v copy -c:a copy -segment_time 01:18:00 -f segment %03d"$1"
    • ffmpeg -i 001out.mp4 -c:v copy -c:a copy 001out-reprocessed.mp4 this will fix the timestamps must be some flag im missing on original line.

Tar compress, split and GPG

rsync

  • rsync -a --progress
  • rsync -a --delete --progress
  • rsync -a --delete --progress --dry-run

tar tape length

  • tar -M -L 500M -cvf backup.tar.1 *
  • Uses base 2 size suffixes (G, M): SIZE x 1024 use the bytes suffix (c) for anything else.
  • gnu.org/software/tar/manual/html_section/tar_78.html
    • > Sure enough, to extract a split member you would need all volumes its parts reside on.
  • tar has interactive mode so you can swap tapes or other medium and specify it’s path when it reaches the end of tape 1.
  • Useful for large files which would span more than one medium on their own.

7-Zip

  1. Add to archive
  2. Compression level use Store for a tar like substitute otherwise use default for compressible files.
  3. Split into blocks of, you can choose DVD or select one from the list then manually change the size.

dirsplit

  • linux.die.net/man/1/dirsplit
  • Brasero seems to work with soft or hard links.
  • ImgBurn only works with hard links from what I tested over SMB.
  • To see all the flags you want this command: $ dirsplit --longhelp
  • Uses base 2 (G, M) or base 10 (g, m) size suffixes.

Example of a file too large to fit onto one medium

$ dirsplit -s 200m -L -S camera/
Building file list, please wait...
Too large object(s) (15574072797) for the given max size: /media/anon/camera/large-file-1.mp4 (maybe coalesced in arrays, check manually)

mkisofs catalog example

$ dirsplit -s 24g -S camera/

$ ls
camera  vol_1.list  vol_2.list
$ dirsplit -s 24g -l -S camera/`

$ ls
camera  vol_1  vol_2
$ dirsplit -s 24g -L -S camera/`

$ ls
camera  vol_1  vol_2

File size suffix and file managers

Base 2 file

tar -M -L 6G -cvf 1-tar *

My file manager reports it as: 6.4 GB (6442455040 bytes)

$ du  *
6291464	1-tar
$ du -hs *
6.1G	1-tar
$ du --si *
6.5G	1-tar
$ bc -l
6442455040/1024
6291460.00000000000000000000
6291460/1024
6144.00390625000000000000
6144/1024
6.00000000000000000000

Base 10 file

split -b 500MB backup.tar.gz backup.tar.gz.

My file manager reports it as: 500.0 MB (500000000 bytes)

$ du *
488284	backup.tar.gz.aa
$ du -hs *
477M	backup.tar.gz.aa
$ du --si *
501M	backup.tar.gz.aa
$ bc -l
500000000/100
5000000.00000000000000000000
5000000/100
50000.00000000000000000000
50000/100
500.00000000000000000000

Other

TODO: Some kinda what tool for what job consolidation

  • backup-dirsplit-public.sh for music and videos public domain non sensitive? Data disc. No compression needed here and no encryption as containers are already compressed.
  • backup-tar-split-gpg-mv-bd-private.sh for pretty much anything personal.
  • backup-tar-split-mv-bd-public.sh for pretty much anything public domain large data set like public code backups.
  • Tar length for HDD backups or something. No compression needed here and no encryption as it can be done at entire disk level.

What to include on a disc backup label

  • Volume name
  • Date of burning
  • Disc number of how many in the backup set
  • Any notes of special file exclusions or inclusions.
  • Encrypted or unencrypted?
  • Owner
  • Method of backup e.g. tar, split or is this obvious from file extensions?
  • File system type
  • Write software
  • Alphabetic range if storing music dumps e.g. A-E
  • Unique int id number which can then be stored in a central index e.g. spreadsheet or CSV along with other metadata.

Disc vs Disk

  • Disc, discus, round, optical disc, compact disc
  • Disk, diskette, floppy disk, hard disk drive

Colour rings

I couldn’t find much on Blu-ray/Blueray colour/color rings/bands burn line colour.

forum.imgburn.com/index.php?/profile/33189-e5frog/&do=content&all_activity=1&page=2

e5frog replied to e5frog’s topic in ImgBurn Support

But the log says OPC: No, and isn’t it supposed to do that when writing the lead-in, once… Tried 8x, fail after 78%, same kind of bands in the written area. The successful write before it also has some of these bands of different color (it’s very hard to get a good picture of it). Trying another 8x write with OPC on…

e5frog replied to e5frog’s topic in ImgBurn Support

Sorry, I didn’t save the log and I went through five discs that burned up to 90% before write error occured, all had increasingly lighter “bands” where they were written to. Sixth disc worked right away, no strange patterns on the disc, same looking surface on all the burnt area. So, I guess it’s the discs, but thanks for the feedback, I’ll see to it that I upgrade again then.

CAV vs CLV

DVD Burning and Media Quality Concepts

With the advent of 8x media, Z-CLV (zonal constant linear velocity) and P-CAV (partial constant angular velocity) were introduced. Z-CLV starts at a speed like 4x, then at a certain point in the media, jumps to 6x, then 8x, etc., until it reaches the maximum speed. Sometimes a 16x Z-CLV burn will only burn a few hundred MB at 16x, which is why “x” speeds mean almost nothing anymore. P-CAV is similar, but does not jump speed. It increases velocity from 4x to 4.5x to 5x, etc., until it reaches it’s top speed. Much like Z-CLV, it may not hit max speed until the last 30 seconds worth of burning. This is why a 12x burn is almost an identical wait time to the speed of a 16x burn.

This CLV burn is the same color from beginning to end.

Z-CLV burning methods leave a mark on the dye, as the burn speed alters color slightly. These zones are perfect circles with a hard edge lines. Not to be confused with imperfect-shaped dye rings caused from inferior media. P-CAV may gradually change colors.

Moisés Cardona - VERBATIM 4X BDXL 100GB BLANK MEDIA

Here, we can see the written disc with its Z-CLV zones:

My BDR-212M burner seems to be CAV as per: pioneer.jp/device_e/product-e/ibs/device_e/pdf/BDR-212M_Medialist.pdf

This Polish website goes above and beyond with testing it:

Interestingly my SU-208GB drive supports DVD DL and this website shows it using a different tracking for DVD DL mydvddrives.com/tsstcorp-cddvdw-su-208gb-su208gb

  • DVD+R 8x ( CAV )
  • DVD+R DL 6x ( ZCLV )

Below are some tests I did with Verbatim BD-R Datalife 43812 to try reproduce colour some rings by toggling local image cache before write to hopefully change buffer times.

Create image test

Before write

2021-01-08-create-image-test-before-write.png

Cover

2021-01-08-create-image-test-cover.png

After write with local image cache

2021-01-08-create-image-test-k3b-write.png

Identical test

Before write

2021-01-08-identical-test-before-write.png

Cover

2021-01-08-identical-test-cover.png

After write with no local image cache image 1

2021-01-08-identical-test-k3b-write-1.png

After write with no local image cache image 2

2021-01-08-identical-test-k3b-write-2.png

Side by side

2021-01-08-side-by-side.png

2021-01-08-ring-vs-no-ring.png

Imperfections

I noticed a few imperfections so I now check the blank medium before using it.

imperfection-1.png

imperfection-2.png

imperfection-3.png

imperfection-4.png

imperfection-5.png

comments powered by Disqus