Storage architecture optical backups

by on under sysadmin
26 minute read

Backing up data using optical media

  • When looking to purchase a Blu-ray burner watch out for Blu-ray drives which only read BD but write DVD and CD.

Burning software

  • Disc spanning
  • Had some issues getting FLAC onto a CD-R audio project without codecs.
  • Proxmox supports cd-rom passthrough but I had issues where it seemed the guest didn’t have full control.
  • VirtualBox works flawlessly so far.

virtualbox-passthrough

Brasero

  • Will create a MD5 checksum for you.
  • Works on CD-R and DVD-R but not BD-R discs.
  • Enabling the burn the image directly without saving it to disc option will skip making a local ISO cache before burning to disc in /tmp
  • CD-R write speeds are one of the fastest I have tested so far as high as 34x. Using Pioneer, VM passthrough Windows host -> Linux guest and Sony 700 MB.
  • Had it ask if I want to rename files for full Windows compatibility, I press okay it creates a checksum and then says do I want to continue without full Windows compatibility OK or Cancel. Normally this works fine so there must be some edge case.
  • Crashes on audio projects often.

K3B

  • You need to be careful with adding files up to the limit as you may get “mkisofs crashed” error.
  • Long File-Name Support for Burning Data Discs in Ubuntu with K3b
  • Works on CD-R, DVD-R and BD-R discs.
  • You will need to create your own checksum.
  • Enabling Create image option will make a local ISO cache before burning to disc in /tmp.
  • You will need to manually enable verify disc. This should really be on by default.
  • DB-R write speeds only seen as high as 3x but have seen it dip to 1.7x. Using Pioneer, VM passthrough Windows host -> Linux guest and Verbatim 25 GB 6x.
  • “cdrecord has no permission to open the device”
  • Calls mkisofs

Windows Explorer

ImgBurn

  • Works on CD-R?, DVD-R and BD-R discs.
  • Gives some automatic file system recommendation that better suits MP3 BD-R data discs.
  • You will need to create your own checksum. TODO: Check settings further.
  • Produces a graph output which can be read in DVDINFOPro.
  • Possible to span large image across CD’s?
  • How to split a large file in 2 small ones
  • DB-R 8.1x write speed. Uses 318 Mbps pulling from a NAS at those write speeds. Using Pioneer, VM passthrough Windows host -> Windows guest and Verbatim 25 GB 6x.
  • DVD-R 13.4x write speed. Using Pioneer, VM passthrough Windows host -> Windows guest and Philips 4.7 GB 12x.

Nero

Write speed tests

Id Date Disc brand Disc type Write software File source IO type Project type File type Write device Write speed max Write speed average Result
1 2020-12-12 Verbatim BD-R K3B Local cache Live Linux USB direct Data MP4, MOV BDR-212M 3.3x   OK
2 2020-12-12 Verbatim BD-R K3B Remote SMB Windows -> VM passthrough -> Linux Data MP4, MOV, JPG, TNL, MTS BDR-212M     ERROR Ran out of disc space
3 2020-12-12 Verbatim BD-R Windows Local cache Windows direct Data MP4/MOV BDR-212M N/A N/A OK
4 2020-12-12 Verbatim BD-R K3B Local cache Live Linux USB direct Data (FS Linux only) MP4/MOV BDR-212M 3.3x   OK
5 2020-12-12 Verbatim BD-R K3B Remote SMB Windows -> VM passthrough -> Linux Data MP3 BDR-212M 2.4x   OK
6 2020-12-14 Verbatim BD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data MP3 BDR-212M 8.1x   OK
7 2020-12-14 Sony CD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data MP3 BDR-212M N/A N/A ERROR
8 2020-12-14 Sony CD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data MP3 BDR-212M N/A N/A ERROR
9 2020-12-14 Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data MP3 BDR-212M 13.4x   OK
10 2020-12-14 Philips DVD-R K3B Local cache Windows -> VM passthrough -> Linux Data MP3 BDR-212M 15x   OK
11 2020-12-14 Philips DVD-R Brasero Local cache Windows -> VM passthrough -> Linux Data MP3 BDR-212M 16x 10.9x OK
12 2020-12-14 Sony CD-R Brasero Local cache Windows -> VM passthrough -> Linux Data MP3 BDR-212M 34x 24.9x OK
13 2020-12-14 Sony CD-R Windows Local cache Windows direct Data MP3 BDR-212M N/A N/A OK
14 2020-12-07 Sony CD-R Brasero USB3 Linux direct Data MP3 SU-208GB     OK
15 2020-12-06 Philips DVD-R Brasero USB3 Linux direct Data MP3 SU-208GB     OK
16 2020-12-07 Philips DVD-R K3B USB3 Linux direct Data MP4 DU-8A5LH     OK
  • BDR-212M is a Pioneer drive.
  • SU-208GB is a TSSTcorp drive.
  • DU-8A5LH is a drive commonly found in DELL laptops.
  • Write software Windows refers to Windows Explorer but is kept short for brevity.
  • Some items maybe missing write speed if the software doesn’t show it like in the case of Windows Explorer, otherwise I may have not recorded the speed at the time.

Compatibility/playback tests

WriteId Read device Result
1 PS3 OK
1 Xbox One S Error 0x80820002
3 PS3 OK
3 Xbox One S Error 0x80820002
4 PS3 OK
4 Xbox One S Error 0x80820002
5 PS3 OK
5 Xbox One S Error 0x80820002
6 PS3 OK
6 Xbox One S Error 0x80820002
9 PS3 OK
9 Xbox One S Error 0x80820002
9 LG DVX392H OK
10 PS3 OK
10 Xbox One S Error 0x80820002
10 LG DVX392H OK
11 PS3 OK
11 Xbox One S Error 0x80820002
11 LG DVX392H OK
12 PS3 OK
12 Xbox One S Error 0x80820002
12 LG DVX392H OK
13 PS3 OK
13 Xbox One S Error 0x80820002
13 LG DVX392H OK
14 PS3 OK
14 Xbox One S Error 0x80820002
14 LG DVX392H OK
15 PS3 OK
15 Xbox One S Error 0x80820002
15 LG DVX392H OK
16 PS3 OK
16 Xbox One S Error 0x80820002
16 LG DVX392H Err
  • 1, 3 and 4 the PS3 wouldn’t find the MOV files but this is a separate issue relating to PS3 codecs not a medium issue. It did however play MP4 just fine.

Optical medium models

Verbatim

The Verbatim product line can be a bit confusing and their main website could do with updating. The EU website is a bit more modern but lacking clear comparisons. Luckily I finally found the information I was after here: Verbatim Optical Media July 2020 Brochure

  • Annoyingly you cannot get the same “SURFACE” across all medium, in addition it’s specific to the “CONFIGURATION” too.
  • “Wide Printable No-ID” seems to be the only “SURFACE” which is common across nearly all medium. Second to that would be “Matt Silver” of which CD-R and BD-R doesn’t come in.
  • Their professional line is all “No-ID” annoyingly. They intend you to print them however.
  • The “CRYSTAL” surface is specific to CD-R AZO only.
  • Be warned that “PART NO” 43791 under the “Datalife” branding is “Non-AZO”. “PART NO” 97693 under the “Life Series” is also “Non-AZO”.
  • > Archival Life: Extra Protection - up to 40 years; AZO - up to 100 years.

Below are some “consistent” “SURFACE” configurations you could get. Otherwise it’s a “mix” of surfaces for decent prices:

Type Configuration Surface Product name Product Id
Consistent 50pk Spindle Wide Printable No-ID CD-R AZO 43438
Consistent 50pk Spindle Wide Printable No-ID DVD-R AZO 43533
Consistent 50pk Spindle Wide Printable No-ID BD-R Datalife 43812
Consistent 25pk Spindle Wide Printable ID CD-R AZO 43439
Consistent 25pk Spindle Wide Printable ID DVD-R AZO 43538
Mixed 50pk Spindle Crystal CD-R AZO 43343
Mixed 50pk Spindle Matt Silver DVD-R AZO 43548
Mixed 50pk Spindle Wide Printable No-ID BD-R Datalife 43812
Supplementary 25pk Spindle Matt Silver DVD+R DL AZO 43757
Supplementary 25pk Spindle Wide Printable No-ID DVD+R DL AZO 43667
Supplementary 10pk Spindle White/Blue BD-R DL 43746

Technologies

  • “MediDisk”
  • “Datalife”
  • “HARD COAT”
  • “SUPER HARD COAT”
  • “AZO”
  • “SERL”
  • “EXTRA PROTECTION”
  • “M.A.B.L”
  • “CRYSTAL”
  • “M-DISC”

Surfaces

  • “Mat Silver”
  • “Inkjet Printable”
  • “White/Blue”
  • “Wide Printable No-ID”
  • “Wide Print No-ID” (inconsistency)
  • “Wide Printable ID”
  • “5 Colours”
  • “Colour” (possible inconsistency)
  • “Crystal”
  • “Extra Protection”
  • “Silver”
  • “Thermal No-ID”
  • “DataLifePlus Wide Thermal No-ID”
  • “DataLifePlus Wide Printable No-ID”
  • “DataLifePlus Waterproof Printable No-ID”

Philips

Can’t seem to find a data sheet or product brochure unfortunately. So no way to quantify what a “very long archive lifespan” is.

File systems

  • K3B Linux/Unix only seems to read back just fine on Windows 10. It looks to be ISO 9660 Level 3 + Rock Ridge extensions.
  • K3B Linux/Unix + Windows is ISO 9660 Level 3 + Rock Ridge extensions + Joliet extensions.
  • ISO 1999?
  • ISO 9660 file system
  • Joliet extension to ISO 9660.
  • Rock Ridge extension to ISO 9660.
  • UDF file system
  • UDF with K3B
  • ImgBurn UDF 1.02
  • ImgBurn ISO 9660 + Joliet: MP3 data disks
  • ImbBurn ISO 9660 + Joliet: DivX video disc
  • ImgBurn ISO 9660 + UDF - Bridge disc TODO: Same as UDF structures on K3B?
  • ISO 9660 + Joliet + UDF - vs doing each separate
  • Brasero uses ISO 9660 Level 2? But will switch to Level 3 when you add files over 2 GB and you accept the warning message.

Scripts

ffmpeg

  • How can I use ffmpeg to split MPEG video into 10 minute chunks?
  • Below is a adaptation I used for DVD splits of long 6 hour mp4 video files. Some work is needed still since this causes issues on the time stamps.
  • One downside with this is using this for data disc project wont make use of the disks full capacity as the disc can fit more mp4 runtime than 120 mins in DVD VIDEO_TS mode.
  • 1: ffmpeg -i "$1" -c copy -map 0 -segment_time 01:18:00 -f segment %03d"$1"
  • That creates a file around: 3.6 GB (3604783722 bytes) in size although it will depend how compressed your source file is.
  • 2: ffmpeg -i "$1" -c:v copy -c:a copy -segment_time 01:18:00 -f segment %03d"$1"
    • ffmpeg -i 001out.mp4 -c:v copy -c:a copy 001out-reprocessed.mp4 this will fix the timestamps must be some flag im missing on original line.

Tar compress, split and GPG

rsync

  • rsync -a --progress
  • rsync -a --delete --progress
  • rsync -a --delete --progress --dry-run

tar tape length

  • tar -M -L 500M -cvf backup.tar.1 *
  • Uses base 2 size suffixes (G, M): SIZE x 1024 use the bytes suffix (c) for anything else.
  • gnu.org/software/tar/manual/html_section/tar_78.html
    • > Sure enough, to extract a split member you would need all volumes its parts reside on.
  • tar has interactive mode so you can swap tapes or other medium and specify it’s path when it reaches the end of tape 1.
  • Useful for large files which would span more than one medium on their own.

7-Zip

  1. Add to archive
  2. Compression level use Store for a tar like substitute otherwise use default for compressible files.
  3. Split into blocks of, you can choose DVD or select one from the list then manually change the size.

dirsplit

  • linux.die.net/man/1/dirsplit
  • Brasero seems to work with soft or hard links.
  • ImgBurn only works with hard links from what I tested over SMB.
  • To see all the flags you want this command: $ dirsplit --longhelp
  • Uses base 2 (G, M) or base 10 (g, m) size suffixes.

Example of a file too large to fit onto one medium

$ dirsplit -s 200m -L -S camera/
Building file list, please wait...
Too large object(s) (15574072797) for the given max size: /media/anon/camera/large-file-1.mp4 (maybe coalesced in arrays, check manually)

mkisofs catalog example

$ dirsplit -s 24g -S camera/

$ ls
camera  vol_1.list  vol_2.list
$ dirsplit -s 24g -l -S camera/`

$ ls
camera  vol_1  vol_2
$ dirsplit -s 24g -L -S camera/`

$ ls
camera  vol_1  vol_2

File size suffix and file managers

Base 2 file

tar -M -L 6G -cvf 1-tar *

My file manager reports it as: 6.4 GB (6442455040 bytes)

$ du  *
6291464	1-tar
$ du -hs *
6.1G	1-tar
$ du --si *
6.5G	1-tar
$ bc -l
6442455040/1024
6291460.00000000000000000000
6291460/1024
6144.00390625000000000000
6144/1024
6.00000000000000000000

Base 10 file

split -b 500MB backup.tar.gz backup.tar.gz.

My file manager reports it as: 500.0 MB (500000000 bytes)

$ du *
488284	backup.tar.gz.aa
$ du -hs *
477M	backup.tar.gz.aa
$ du --si *
501M	backup.tar.gz.aa
$ bc -l
500000000/100
5000000.00000000000000000000
5000000/100
50000.00000000000000000000
50000/100
500.00000000000000000000

Other

TODO: Some kinda what tool for what job consolidation

  • backup-dirsplit-public.sh for music and videos public domain non sensitive? Data disc. No compression needed here and no encryption as containers are already compressed.
  • backup-tar-split-gpg-mv-bd-private.sh for pretty much anything personal.
  • backup-tar-split-mv-bd-public.sh for pretty much anything public domain large data set like public code backups.
  • Tar length for HDD backups or something. No compression needed here and no encryption as it can be done at entire disk level.

What to include on a disc backup label

  • Volume name
  • Date of burning
  • Disc number of how many in the backup set
  • Any notes of special file exclusions or inclusions.
  • Encrypted or unencrypted?
  • Owner
  • Method of backup e.g. tar, split or is this obvious from file extensions?
  • File system type e.g. ISO 9660
  • Write software
  • Alphabetic range if storing music dumps e.g. A-E
  • Unique int id number which can then be stored in a central index e.g. spreadsheet or CSV along with other metadata.

Disc vs Disk

  • Disc, discus, round, optical disc, compact disc
  • Disk, diskette, floppy disk, hard disk drive

Colour rings

I couldn’t find much on Blu-ray/Blueray colour/color rings/bands burn line colour.

forum.imgburn.com/index.php?/profile/33189-e5frog/&do=content&all_activity=1&page=2

e5frog replied to e5frog’s topic in ImgBurn Support

But the log says OPC: No, and isn’t it supposed to do that when writing the lead-in, once… Tried 8x, fail after 78%, same kind of bands in the written area. The successful write before it also has some of these bands of different color (it’s very hard to get a good picture of it). Trying another 8x write with OPC on…

e5frog replied to e5frog’s topic in ImgBurn Support

Sorry, I didn’t save the log and I went through five discs that burned up to 90% before write error occured, all had increasingly lighter “bands” where they were written to. Sixth disc worked right away, no strange patterns on the disc, same looking surface on all the burnt area. So, I guess it’s the discs, but thanks for the feedback, I’ll see to it that I upgrade again then.

CAV vs CLV

DVD Burning and Media Quality Concepts

With the advent of 8x media, Z-CLV (zonal constant linear velocity) and P-CAV (partial constant angular velocity) were introduced. Z-CLV starts at a speed like 4x, then at a certain point in the media, jumps to 6x, then 8x, etc., until it reaches the maximum speed. Sometimes a 16x Z-CLV burn will only burn a few hundred MB at 16x, which is why “x” speeds mean almost nothing anymore. P-CAV is similar, but does not jump speed. It increases velocity from 4x to 4.5x to 5x, etc., until it reaches it’s top speed. Much like Z-CLV, it may not hit max speed until the last 30 seconds worth of burning. This is why a 12x burn is almost an identical wait time to the speed of a 16x burn.

This CLV burn is the same color from beginning to end.

Z-CLV burning methods leave a mark on the dye, as the burn speed alters color slightly. These zones are perfect circles with a hard edge lines. Not to be confused with imperfect-shaped dye rings caused from inferior media. P-CAV may gradually change colors.

Moisés Cardona - VERBATIM 4X BDXL 100GB BLANK MEDIA

Here, we can see the written disc with its Z-CLV zones:

My BDR-212M burner seems to be CAV as per: pioneer.jp/device_e/product-e/ibs/device_e/pdf/BDR-212M_Medialist.pdf

This Polish website goes above and beyond with testing it:

Interestingly my SU-208GB drive supports DVD DL and this website shows it using a different tracking for DVD DL mydvddrives.com/tsstcorp-cddvdw-su-208gb-su208gb

  • DVD+R 8x ( CAV )
  • DVD+R DL 6x ( ZCLV )

Below are some tests I did with Verbatim BD-R Datalife 43812 to try reproduce colour some rings by toggling local image cache before write to hopefully change buffer times.

Create image test

Before write

2021-01-08-create-image-test-before-write.png

Cover

2021-01-08-create-image-test-cover.png

After write with local image cache

2021-01-08-create-image-test-k3b-write.png

Identical test

Before write

2021-01-08-identical-test-before-write.png

Cover

2021-01-08-identical-test-cover.png

After write with no local image cache image 1

2021-01-08-identical-test-k3b-write-1.png

After write with no local image cache image 2

2021-01-08-identical-test-k3b-write-2.png

Side by side

2021-01-08-side-by-side.png

2021-01-08-ring-vs-no-ring.png

Imperfections

I noticed a few imperfections so I now check the blank medium before using it.

imperfection-1.png

imperfection-2.png

imperfection-3.png

imperfection-4.png

imperfection-5.png

Further compatibility testing

Data set 1 write

Id Date Disc brand Disc type Write software File source IO type Project type File system File type Write device Write speed max Write speed average Result
18 2021-02-27 Kodak DVD+R DL ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data UDF 1.02 MP4 BDR-212M 10.5x 8x OK
19   Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data UDF 1.02 MP4 BDR-212M     OK
20   Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+UDF MP4 BDR-212M     OK
21   Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet+UDF MP4 BDR-212M     OK
22 2021-02-27 Kodak DVD+R DL ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet MP3 BDR-212M     OK
23   Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet+UDF FLAC, TXT, WAV, M4A, AIFF, JPG, PNG, GIF, MKV, MP4, AVI, MP3, WEBM BDR-212M     OK
24   Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet FLAC, TXT, WAV, M4A, AIFF, JPG, PNG, GIF, MKV, MP4, AVI, MP3, WEBM BDR-212M     OK
25   Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+UDF 1.02 FLAC, TXT, WAV, M4A, AIFF, JPG, PNG, GIF, MKV, MP4, AVI, MP3, WEBM BDR-212M     OK
26   Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data UDF 1.02 FLAC, TXT, WAV, M4A, AIFF, JPG, PNG, GIF, MKV, MP4, AVI, MP3, WEBM BDR-212M     OK
27   Kodak DVD+R DL Brasero Remote SMB Linux direct Data ISO 9660 JPG, MP3, 3PG, OGG, PNG, FLV, WAV, M4A, FLAC, MOV, MP4, MKV, AVI, AAC, UMX, MD5 SU-208GB     OK
  • “Data set 1” is a continuation of the data set mentioned in Write speed tests and Compatibility/playback tests
  • 23, 24, 26, 27 are the most useful test disks.
  • TODO: Need a BB-R and BD-R DL multi file test disk.
  • 25 was a accidental burn because I wasn’t planning on testing that FS combo. However I thought id test it anyway.

Data set 1 read

WriteId Read device Result Readable file types
18 PS3 OK MP4
18 LG DVX392H Err  
19 LG DVX392H Err  
20 LG DVX392H Err  
21 LG DVX392H Err  
22 LG DVX392H OK MP3
23 LG DVX392H OK MP3, JPG, AVI
24 LG DVX392H OK MP3, JPG, AVI
25 LG DVX392H OK MP3, JPG, AVI
26 LG DVX392H OK MP3, JPG, AVI
27 LG DVX392H OK MP3, JPG, AVI

Data set 2 write

Id Date Disc brand Disc type Write software File source IO type Project type File system File type Write device Write speed max Write speed average Result
65 2021-03-03 MediaRange BD-R DL ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet binary BDR-212M 8x 6.6x OK

Data set 2 read

N/A

Data set 3 write

Id Date Disc brand Disc type Write software File source IO type Project type File system File type Write device Write speed max Write speed average Result
1 2021-02-24 Verbatim BD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet MP3 BDR-212M     OK
2 2021-02-24 Verbatim BD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data UDF 1.02 MP4 BDR-212M     OK
3 2021-02-24 Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data UDF 1.02 MKV BDR-212M     OK
5 2021-02-24 Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet MP3 BDR-212M     OK
6 2021-02-24 Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet MP3 BDR-212M     OK
7 2021-02-24 Philips DVD-R ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet MP3 BDR-212M     OK
8 2021-03-08 Kodak DVD+R DL ImgBurn Remote SMB Windows -> VM passthrough -> Windows Data ISO 9660+Joliet MP3 BDR-212M     OK

Data set 3 read

WriteId Read device Result Readable file types Notes
1 Sony BDP-S1500 OK MP3  
2 Sony BDP-S1500 OK MP4 One MP4 file didn’t work: “The file is corrupt or unsupported.”. The rest were working perfect.
3 Sony BDP-S1500 OK MKV Videos with no audio present: “Audio format not supported.”
5 Sony BDP-S1500 OK MP3  
6 Sony BDP-S1500 OK MP3  
7 Sony BDP-S1500 OK MP3  
8 Sony BDP-S1500 OK MP3  

Sony BDP-S1500

  • Sony - Supported File Formats for DLNA® and USB Compatible Blu-ray Disc™ Players and Network Media Players
    • They don’t confirm what formats work on disc however.
    • In my testing it seems to support the same as USB compatible formats.
  • Id:2 seems to be due to 5.1 audio or file size however it’s only 17G and a 11G file worked.
  • Id:3 seems to be due to MKV opus instead of using MP3, MPEG or AAC audio.
    • AC3 apparently works.
      • My friend 2E0EOL says tsMuxer warned him to use AC3.
    • MP4 seems to mostly use aac audio and avc1 video in my sample set.
    • MKV is mostly opus audio in my sample set.

With regards to finding a format for a friends player:

  • MP3 only I would fire out: ISO 9660+Joilet
  • Anything else like MP4, MKV I would fire out UDF.
  • If you’re really worried about compatibility then just use: ISO 9660+Joliet+UDF if that doesn’t work its probably not worth debugging further as you will expend 8 discs for each combination of file type and file system type if you wanted a unique combination test.
  • It seems most systems these days are not worried about file system type but are more worried about codec.
comments powered by Disqus