Writing Many Files to SD

2021-01-26

Some storage mediums are inherently poor at integrity when used in I/O heavy operations. To avoid stressing them, I found myself wanting to only write once to an SD card, despite having many files to write. How do you ensure that you incur the minimum writes when you’re trying to write the whole device? You write an image to regular hard disks, then burn that image directly to the SD without intermediate steps.

Overview

Create a sparse img
Format the image with your desired filesystem (probably exfat)
Make a loop device pointing at the image
Mount the loop device
Write your desired files to it
Unmount / detach loop device 6a. You now have an img file representing a filesystem with all your desired files on it.
Burn the img to your storage medium

Process

Making an empty img is surprisingly not particularly easy. If you’re like me, with no particular knowledge of what separates iso/img, you might think to google how to create iso files. But you don’t want that - iso follows the ISO9660 spec for DVD filesystems, which is absurdly specific and not what you’re after.

Instead, you can use dd to create an empty img file, which is then mounted and used. Great, except you have to physically reserve the correct amount of space on your working device to do so, which is slow and unnecessary. Instead, make a sparse img file with;

dd if=/dev/zero of=./archive.img bs=1 count=0 seek=990G

Where seek dictates your actual physical volume size that you want to write to (in my case, about a terabyte). This should complete almost immediately, since nothing is really happening.

Mounting / loop device

Your instinct, like mine, is probably to mount the file directly. That’s fine, but gets weird if you umount it and try to mount something else. For best results, you need to create a loop device that points to the file, then mount that. It’s not as bad as it sounds.

The actual command to do that is;

losetup -fP ./${archive}.img

Then mount the resulting device

mount "${loopdevice}" "/mnt/someArchive"

To unmount you’ll call losetup -d on the device you created.

Final Script

#!/bin/bash

# $1 is the basename of the img to make.
# will also be used as the mount name.

archive="$1"
targetDevice=/dev/mmcblk0

mkdir -p "/mnt/${archive}"

dd if=/dev/zero of=./${archive}.img bs=1 count=0 seek=990G
mkexfatfs -n ${archive} ./${archive}.img

loopdevice=$(losetup -fP --show ./${archive}.img)
mount "${loopdevice}" "/mnt/${archive}"

echo "mounted '${archive}' as ${loopdevice} at /mnt/${archive}"

# at this point, copy your files as desired.

# to write to medium
dd if=./${archive}.img of=${targetDevice} bs=32M conv=sparse status=progress

# to unmount
umount "/mnt/${archive}"
losetup -d "${loopdevice}"