SWGEmu – Using a RAM Drive to Reduce SSD Wear When Compiling

If we could go back nearly twenty years to the internet of yore, we could read a post I made where I dreamed about having enough RAM to do something useful with a RAM drive in Windows 2000. Since then I think the most use I have gotten out RAM drives has been their use in Linux distributions such as Damn Small Linux and Knoppix. That was until I recently upgraded to 24GB of RAM from 12GB, because the price on DDR3 finally dropped to sane levels and I really wanted to give this a whirl.

What Are We Doing?
We’re using our extra system RAM as a “scratch pad” file system so the GCC compiler can put its junk there while it does its thing.

Why Are Doing This?
The main reason is because SSDs (solid state drives) wear out a little bit every time they are written to and the process of compiling the SWGEmu server (Core3) writes a hell of a lot files to the drive, thereby reducing the drive’s operational life time. This is likely not significant for enterprise MLC based SSDs, but it absolutely is important for consumer grade TLC and 3D NAND drives, as they are designed to be more for reading data than constant writing of data – Compiling large software is an edge-case, abnormal usage for this kind of hardware. The other reason is that doing this is significantly faster, because it eliminates the write and seek speed bottle necks that exist even when using a fast SSD (well, probably not significantly over an NVME M.2 SSD, but I don’t have one to test that assumption).

Requirements
– A VirtualBox based SWGEmu development environment running Debian/Devuan Linux.
– 20GB RAM will just barely squeeze by with a Windows 10 host OS, so more than that would be ideal.
– A solid state drive and a rotational hard drive.

The Setup
Being completely honest here, this setup is a little complicated, but it makes sense when you look at it on the whole, because the configurations tries to play to strengths of all the hardware that is available. We use the SSD to store the development environment, because that allows it to boot quickly and dramatically improves the performance of Grep searches of the code base and other read-heavy activities. We use a standard old rotational hard drive to store the RAM disk image, log files, and Berkeley database files, because they are frequently written to, but in such a way that the slowness of the drive isn’t really a problem for a testing/development environment. And of course, we use the RAM Drive as the place where GCC spits out its object code and does its executable linking, because that’s an epic ass load of disk writes for what are essentially a bunch of temporary files.

The first thing you will need to do is create the empty virtual disk where you will be storing the RAM drive images. On the settings panel for your VM in VirtualBox go to Storage > Sata Controller > Plus Sign and follow the steps to create a new 20GB VDI image somewhere handy on your rotational hard drive (I put mine in S:\Rob\VirtualBoxVMs\ for instance). In the VirtualBox world, this is the same thing as plugging an empty hard drive into the machine, so the next time you boot up your VM you will need to manually configure the new drive. The steps for that are,

1. Use “sudo cfdisk /dev/sdb” in a terminal to create a partition table that has a single 20GB linux partition. It’s visual and pretty straight forward.

2. Format the partition using “sudo mkfs.ext4 /dev/sdb1”.

3. Create a mount point for the drive.
mkdir /home/swgemu/hd

4. Add the partition to the /etc/fstab file so it will automatically be mounted at startup.
sudo nano /etc/fstab
Add the following line to the file and save it:
/dev/sdb1 /home/swgemu/hd ext3 errors=remount-ro 0 1

Next we need to setup VirtualBox so it can handle the RAM drive. This is where your system specs are important. Compiling Core3 requires approx 850MB of RAM per core used, the Xfce based dev environment needs about 600MB, Windows 10 needs roughly 2GB, and the RAM drive will need a whopping 10GB to house Core3’s crazy volume of object files. On my system I compile using 6 of the 8 cores available on my system and I like to round up to 1GB/core to be on the safe side. So that means that on the System page of settings for my virtual machine I allocate 16,384MB of RAM to the VM, ensuring that the system has enough RAM to avoid ever using the swap file. If you use more or fewer cores, adjust your RAM value accordingly.

Side Note: In the past, Core3 used only 650MB per core and the compiled objects only took up about 4GB on the drive. How times have changed!

Having allocated the RAM, we now need to create a mount point for the RAM drive and add it to fstab so that it will also be ready on boot up.
1. Create a mount point for the drive.
mkdir /home/swgemu/ramdrive

2. Use fstab to create the RAM drive.
sudo nano /etc/fstab
Add the following line to the file and save it:
tmpfs /home/[yourUserName]/ramdrive tmpfs nodev,nosuid,noexec,nodiratime,size=10240M 0 0

Before you reboot, go ahead and open /etc/fstab so you can add the following “mount binds” which will seamlessly allow Core3 to read/write the log and Berkeley database files to different physical hardware than what is normally mounted at these locations. Such is the power of *NIX file systems!

/home/swgemu/hd/logs /home/swgemu/workspace/Core3/MMOCoreORB/bin/log none defaults,bind 0 0
/home/swgemu/hd/databases/ /home/swgemu/workspace/Core3/MMOCoreORB/bin/databases/ none defaults,bind 0 0

Finally, before you reboot you will need to copy the following files from the /home/swgemu/workspace/Core3/MMOCoreORB/build/unix directory to your documents folder, as they are required by the build process and will end up being hidden by a mount bind later on.

/home/swgemu/workspace/Core3/MMOCoreORB/build/unix/conf3400.sh
/home/swgemu/workspace/Core3/MMOCoreORB/build/unix/stamp-h1
/home/swgemu/workspace/Core3/MMOCoreORB/build/unix/config/config.h.in

Now is the time to reboot!

After the system has loaded, you will need to a “first build and save” to prime the system, after which you will be able to simply use the ramdrive.sh bash script to load and save your RAM drive image.

1. Setup the RAM drive environment by creating some mound points and bind in the RAM drive. It’s important to keep in mind that everything in ~/ramdrive gets deleted when the system reboots.
cd ~/ramdrive
mkdir tmp
mkdir build
cd build
mkdir unix
sudo mount –bind /tmp /home/swgemu/ramdrive/tmp
sudo mount –bind /home/swgemu/ramdrive/build/unix /home/swgemu/workspace/Core3/MMOCoreORB/build/unix

2. Copy the above saved files into the ramdrive directory
/home/swgemu/ramdrive/build/unix/conf3400.sh
/home/swgemu/ramdrive/build/unix/stamp-h1
/home/swgemu/ramdrive/build/unix/config/config.h.in

3. Build the server as you would normally. I use a simple build script that tells me how long the compilation took, because that’s fun for me.

4. Run the “./ramdrive.sh save” command to copy the contents of /home/swgemu/ramdrive/build to a Tarball image that can be restored each time you reboot the system. The image gets saved to /home/swgemu/hd/ with a file name like so, ram-image-2019_05_23-.tar.gz.

Here is a copy of ramdrive.sh

Usage:
./ramdrive.sh load
./ramdrive.sh save

Both of those uses should self explanatory. When you start the system, first thing you should do is open an terminal and run ./ramdrive.sh load to configure the RAM drive and populate it with the previously compiled object code. Any time you have made significant C++ changes, you will want to save a new RAM drive image so that the compiler doesn’t have to catch up compiling changes from way back when, as well as the ones you’re currently working on. As complex as it this whole thing is, it’s really that simple to use it once it’s all setup!

Unfortunately the WordPress theme I am using butchers code, which is why I put link to a copy of the file above, but here’s the code anyway:


#!/bin/bash
# Moves SWGEmu build files to and from hard drive back / ram drive

# Bail if not passed argument
if [ -z "$1" ]
then
echo "Usage: ramdrive "
exit 1
fi

ARG=$1
DIR="/home/swgemu/ramdrive/build"
HD="/home/swgemu/hd"
STAMP=$(date +"%Y_%m_%d")

if [[ $ARG == *"load"* ]]
then
echo "Mounting /tmp to RAM drive..."
cd ~/ramdrive
mkdir tmp
mkdir build
cd build
mkdir unix
sudo mount --bind /tmp /home/swgemu/ramdrive/tmp
echo "Loading most recent RAM disk image...."
cd $HD
tar -zxvf ram-image-*.tar.gz -C $DIR
echo "Mounting Core3 build/unix to RAM drive..."
sudo mount --bind /home/swgemu/ramdrive/build/unix /home/swgemu/workspace/Core3/MMOCoreORB/build/unix
echo "RAM Drive ready for use!"
exit 1
elif [[ $ARG == *"save"* ]]
then
echo "Saving...."

cd $HD

if [ -f "ram-"* ]
then
echo "Archiving previous file..."
FILENAME=$(ls ram-*)
mv $FILENAME old_$FILENAME
fi

cd $DIR
tar -zcvf $HD/ram-image-$STAMP-.tar.gz unix
echo "Contents of RAM Drive saved to hard drive!"
exit 1
else
echo "Usage: ramdrive "
exit 1
fi

With all that said, I’m not sure how practical this is for the general SWGEmu modding community to use, but here it is anyway! Mostly, I just wanted to share this, because the younger me inside thinks it’s “so cool!” to be able to do these kinds of things now lol… However, using this setup cut my compilation times down by about 300 seconds while I was working on mods for Tarkin’s Revenge and it definitely saves my 250GB SK Hynix SSD from a whole lot write cycles, which should increase its life span as my Windows 10 system disk and the place where I keep my favorite games and projects.

Advertisements