Adding a New Disk

WARNING: BE CAREFUL! Some commands in this tutorial can cause permanent data loss. We recommend you backup your files before attempting to run these commands. You should also double check every command to make sure you have made no typos or mistakes. A single typo could destroy your data forever!

NOTE: This guide is no substitute for reading the OpenBSD FAQ. In particular, you should read the section on Disk Setup.

NOTE: Before performing the upgrades, you will want to announce the changes to your users and teammates.

On OpenBSD, there are two disk drivers: wd and sd. wd is an IDE-like disk, and sd is a SCSI-like disk (including USB disks). To list all your available disks:

$ dmesg | grep -E '(sd[0-9]|wd[0-9])'
sd0 at scsibus2 targ 0 lun 0: <BUYVM, SLAB, 2.5+> serial.BUYVM_SLAB_VOLUME-4047
sd0: 524288MB, 512 bytes/sector, 1073741824 sectors, thin
sd1 at scsibus3 targ 0 lun 0: <VirtIO, Block Device, >
sd1: 40960MB, 512 bytes/sector, 83886080 sectors

In this example, our server has two SCSI-like disks (a BuyVM slab and a VirtIO block device). In this case, the VirtIO block device is the hard disk the operating system is installed on, and the slab is extra storage we purchased. The disks are numbered by the order in which they are detected at boot time. The first SCSI-like disk will be sd0, the second sd1, and so forth.

The first disk sd0 can store around 500GB of data, the second disk sd1 can store around 40GB.

We will now add the new space from sd0 by creating a new partition for the disk, formatting it, and then mounting it to the filesystem.

WARNING: Double check your commands and make certain you are editing the correct disk. You must substitute the disk in the example for the one you actually want to format. Typing the wrong disk by accident could destroy all your data forever.

Creating a New Partition

The first task is to use fdisk to create fdisk partitions (Master Boot Record partitions). Now, the easy way is to run this one command:

$ doas fdisk -iy sd0

This automatically creates an fdisk partition that spans the entire disk. That said, you can (and should!) learn how to edit this interactively, to fine tune your options.

$ doas fdisk -e sd0
Enter 'help' for information

Type help to get help, manual to read the fdisk man page, and print to view the partitions:

sd0: 1> help
        help            Command help list
        manual          Show entire OpenBSD man page for fdisk
        reinit          Re-initialize loaded MBR (to defaults)
        setpid          Set the identifier of a given table entry
        disk            Edit current drive stats
        edit            Edit given table entry
        flag            Flag given table entry as bootable
        update          Update machine code in loaded MBR
        select          Select extended partition table entry MBR
        swap            Swap two partition entries
        print           Print loaded MBR partition table
        write           Write loaded MBR to disk
        exit            Exit edit of current MBR, without saving changes
        quit            Quit edit of current MBR, saving current changes
        abort           Abort program without saving current changes
sd0: 1> print
Disk: sd0       geometry: 66837/255/63 [1073741824 Sectors]
Offset: 0       Signature: 0x0
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      

This disk sd0 is completely blank, so it is probably safe to edit the fdisk partitions and format the disk.

WARNING: If the disk already has existing partitions, double check to make sure you are not wiping out data!

First, let's reinit. This should automatically create an fdisk partition on the last partition, partition 3, for OpenBSD and use all available space. It will also initialize this as the boot block.

sd0: 1> reinit
Disk: sd0       geometry: 66837/255/63 [1073741824 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
*3: A6      0   1   2 -  66836 254  63 [          64:  1073736341 ] OpenBSD     
Use 'write' to update disk.

Notice that the Signature changes from 0x0 to 0xAA55, and partition #3 is now set to id A6 (the OpenBSD partition type).

This can also be done manually by using the edit command on partition 3:

sd0: 1> edit 3
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
Partition id ('0' to disable) [01 - FF]: [0] (? for help) A6
Do you wish to edit in CHS mode? [n] y
BIOS Starting cylinder [0 - 66836]: [0] 
BIOS Starting head [0 - 254]: [1] 
BIOS Starting sector [1 - 63]: [2] 
BIOS Ending cylinder [0 - 66836]: [0] 66836
BIOS Ending head [0 - 254]: [0] 254
BIOS Ending sector [1 - 63]: [1] 63

We need to edit in Cylinder-Head-Sector (CHS) mode. To use the full space of the disk, you will want to choose the lowest numbers in the range for the starting cylinder, head, and sector; and the highest possible numbers for the ending cylinder, head, and sector.

We now write to disk:

sd0*: 1> write
Writing MBR at offset 0.

We'll print G to see everything in gigabytes:

sd0: 1> print G
Disk: sd0       geometry: 66837/255/63 [1073741824 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 00      0   0   0 -      0   0   0 [           0:           0G] unused      
 1: 00      0   0   0 -      0   0   0 [           0:           0G] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0G] unused      
*3: A6      0   1   2 -  66836 254  63 [          64:         512G] OpenBSD     

If everything looks good,

sd0: 1> quit

Disklabel

Next, we need to use disklabel to manage OpenBSD filesystem partitions (these disklabel partitions exist inside of an fdisk partition).

We should leave the first logical track unused, so the first partition should start at block 64.

Each fdisk partition can be sliced into OpenBSD disklabel partitions with letters as labels. The a partition is your root partition, b is your swap, and c is the entire disk.

If you already have the operating system installed on another disk (like we do here), you just need 1 single partition: the a partition with the same size as the disk.

$ doas disklabel -E sd0
Label editor (enter '?' for help at any prompt)
sd0> ?
Available commands:
 ? | h    - show help                 n [part] - set mount point
 A        - auto partition all space  p [unit] - print partitions
 a [part] - add partition             q        - quit & save changes
 b        - set OpenBSD boundaries    R [part] - resize auto allocated partition
 c [part] - change partition size     r        - display free space
 D        - reset label to default    s [path] - save label to file
 d [part] - delete partition          U        - undo all changes
 e        - edit drive parameters     u        - undo last change
 g [d|u]  - [d]isk or [u]ser geometry w        - write label to disk
 i        - modify disklabel UID      X        - toggle expert mode
 l [unit] - print disk label header   x        - exit & lose changes
 M        - disklabel(8) man page     z        - delete all partitions
 m [part] - modify partition

Suffixes can be used to indicate units other than sectors:
 'b' (bytes), 'k' (kilobytes), 'm' (megabytes), 'g' (gigabytes) 't' (terabytes)
 'c' (cylinders), '%' (% of total disk), '&' (% of free space).
Values in non-sector units are truncated to the nearest cylinder boundary.

To print the partitions in gigabytes:

sd0> p G
OpenBSD area: 64-1073736405; size: 512.0G; free: 512.0G
#                size           offset  fstype [fsize bsize   cpg]
  c:           512.0G                0  unused                    

We want to add the a partition. We'll use defaults for offset, size, and FS type:

sd0> a a
offset: [64] 
size: [1073736341] 
FS type: [4.2BSD] 
sd0*> p G
OpenBSD area: 64-1073736405; size: 512.0G; free: 0.0G
#                size           offset  fstype [fsize bsize   cpg]
  a:           512.0G               64  4.2BSD   4096 32768     1 
  c:           512.0G                0  unused                    

Then write the changes to disk and quit:

sd0*> w
sd0> q
No label changes.

Next, you'll want to format the disk:

$ doas newfs sd0a
/dev/rsd0a: 524285.3MB in 1073736320 sectors of 512 bytes                    
644 cylinder groups of 814.44MB, 26062 blocks, 52224 inodes each             
super-block backups (for fsck -b #) at:                                                 
...

We now have sd0a formatted, but we need to mount it. Let's look at where our system's partitions are currently mounted:

$ mount                 
/dev/sd1a on / type ffs (local)
/dev/sd1k on /home type ffs (local, nodev, nosuid)
/dev/sd1d on /tmp type ffs (local, nodev, nosuid)
/dev/sd1f on /usr type ffs (local, nodev)
/dev/sd1g on /usr/X11R6 type ffs (local, nodev)
/dev/sd1h on /usr/local type ffs (local, nodev, wxallowed)
/dev/sd1j on /usr/obj type ffs (local, nodev, nosuid)
/dev/sd1i on /usr/src type ffs (local, nodev, nosuid)
/dev/sd1e on /var type ffs (local, nodev, nosuid)

We can also see the size of these partitions:

$ df -h                                                                      
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd1a      615M    232M    352M    40%    /
/dev/sd1k      3.7G    2.1G    1.4G    59%    /home
/dev/sd1d      863M   28.0K    820M     0%    /tmp
/dev/sd1f      2.3G    1.6G    600M    74%    /usr
/dev/sd1g      648M    238M    378M    39%    /usr/X11R6
/dev/sd1h      2.3G    2.2G   50.2M    98%    /usr/local
/dev/sd1j      5.2G    2.0K    4.9G     0%    /usr/obj
/dev/sd1i      1.4G    2.0K    1.3G     0%    /usr/src
/dev/sd1e      1.3G    318M    913M    26%    /var

For this example, /home is too small for our server, so we're going to copy all the data from sd1k to sd0, then mount sd0a as the new /home.

$ doas su
# mount /dev/sd0a /mnt
# cd /mnt

Then we dump and restore? to clone the partition:

# dump -0 -a -f - /home | restore -rf -

You will want to replace /home with whatever partition you want to back up on the new disk.

Alternatively, you can copy the files manually by running:

# cp -R /home/* /home/.* /mnt/

Inspect the files in the partition to make sure they copied successfully:

# ls -a /mnt/

Then unmount the partition:

# umount /mnt/
# ^D

(Type ctrl+d or ^D to exit)

The next steps will take the system offline, so before performing any more steps, you will want to announce to your users and teammates that they should save all files and prepare for downtime.

Now we need to automatically mount this partition at bootup. Let's take a look at the current /etc/fstab:

37deb3c1c30cbfbf.b none swap sw
37deb3c1c30cbfbf.a / ffs rw 1 1
37deb3c1c30cbfbf.k /home ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.d /tmp ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.f /usr ffs rw,nodev 1 2
37deb3c1c30cbfbf.g /usr/X11R6 ffs rw,nodev 1 2
37deb3c1c30cbfbf.h /usr/local ffs rw,wxallowed,nodev 1 2
37deb3c1c30cbfbf.j /usr/obj ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.i /usr/src ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.e /var ffs rw,nodev,nosuid 1 2

The string 37deb3c1c30cbfbf is called the disklabel unique identifier (DUID). Each disk partition has its own unique DUID. Let's find the DUID of our new disk partition:

$ sysctl hw.disknames 
hw.disknames=cd0:,sd0:f433d9e11879420f,sd1:37deb3c1c30cbfbf,fd0:

The DUID for sd0 is f433d9e11879420f. So, we need to replace /home's partition 37deb3c1c30cbfbf.k with f433d9e11879420f.a (since we are trying to mount sd0a):

37deb3c1c30cbfbf.b none swap sw
37deb3c1c30cbfbf.a / ffs rw 1 1
f433d9e11879420f.a /home ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.d /tmp ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.f /usr ffs rw,nodev 1 2
37deb3c1c30cbfbf.g /usr/X11R6 ffs rw,nodev 1 2
37deb3c1c30cbfbf.h /usr/local ffs rw,wxallowed,nodev 1 2
37deb3c1c30cbfbf.j /usr/obj ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.i /usr/src ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.e /var ffs rw,nodev,nosuid 1 2

Then, reboot:

$ doas shutdown -r now

If everything was done correctly, the system should automatically mount sd0a as /home upon bootup. The old partition, /dev/sd1k, can still be mounted if you need it:

$ doas mount /dev/sd1k /mnt/

Double check to make sure the files on /home exactly match those on /mnt from sd1k. If they do not, you will need to clone the disk again using dump/restore? or cp.

Once you are certain the files have been copied properly, you can use unmount the partition:

$ doas umount /mnt

Then you can clear out the free space to grow an existing partition? or you can create a new partition with it and mount it elsewhere.

Troubleshooting:

Uh oh, my system won't reboot!

This can happen if you foul up /etc/fstab. For example, suppose I had made this error in /etc/fstab:

37deb3c1c30cbfbf.b none swap sw
37deb3c1c30cbfbf.a / ffs rw 1 1
f433d9e11879420f.b /home ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.d /tmp ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.f /usr ffs rw,nodev 1 2
37deb3c1c30cbfbf.g /usr/X11R6 ffs rw,nodev 1 2
37deb3c1c30cbfbf.h /usr/local ffs rw,wxallowed,nodev 1 2
37deb3c1c30cbfbf.j /usr/obj ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.i /usr/src ffs rw,nodev,nosuid 1 2
37deb3c1c30cbfbf.e /var ffs rw,nodev,nosuid 1 2

For the third line, instead of putting .a, I put .b by accident. When I reboot the system by running doas shutdown -r now, I can no longer ssh into the system. When I log in via serial console (cu for vmm or VNC for BuyVM)

root on sd0a (a0e97b069eed0743.a) swap on sd0b dump on sd0b
Automatic boot in progress: starting file system checks.
/dev/sd0a (a0e97b069eed0743.a): file system is clean; not checking
Can't open d40a0cfcd4551ac7.b: No such file or directory
CAN'T CHECK FILE SYSTEM.
d40a0cfcd4551ac7.b: UNEXPECTED INCONSISTENCY; RUN fsck_ffs MANUALLY.
/dev/sd0k (a0e97b069eed0743.k): file system is clean; not checking
/dev/sd0d (a0e97b069eed0743.d): file system is clean; not checking
/dev/sd0f (a0e97b069eed0743.f): file system is clean; not checking
/dev/sd0g (a0e97b069eed0743.g): file system is clean; not checking
/dev/sd0h (a0e97b069eed0743.h): file system is clean; not checking
/dev/sd0j (a0e97b069eed0743.j): file system is clean; not checking
/dev/sd0i (a0e97b069eed0743.i): file system is clean; not checking
/dev/sd0e (a0e97b069eed0743.e): file system is clean; not checking
THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
        ffs: d40a0cfcd4551ac7.b (/home)
Automatic file system check failed; help!
Enter pathname of shell or RETURN for sh: 

In order to boot up, you'll need to fix fstab. So, press the enter key to enter into the shell. Then, to view your /etc/fstab, type:

# cat /etc/fstab

Check how your partitions are mounted:

# mount
root_device on / type ffs (local, read-only)

You'll notice that only / is mounted (so you can't use programs from /usr or view files in /home). This means you'll lack basic utilities like less and text editors like vi and mg. You'll also be unable to edit and save /etc/fstab, since / is mounted read-only. So first, let's fix that.

# mount -rw /
# mount -rw /dev/sd0f /usr

Our fstab says a0e97b069eed0743.f is used for /usr so we mount /dev/sd0f to /usr. This will allow us to use text editors like vi to edit (you could, of course, use ed?).

# export TERM=xterm
# vi /etc/fstab

Replace f433d9e11879420f.b with f433d9e11879420f.a, save and quit, then reboot the system:

# shutdown -r now