I'm going to keep this one short and sweet. S3 cloud object storage is going to be less expensive in most cases than upping a VPS to a bigger drive. For instance, UpCloud offers $5 a month for 250GB, compared to an UpCloud server with 8GB of ram, 4 CPU Cores, and 160Gb of storage for $48/mo. There's a good chance you will not need that much server to run a PDS, but if you have a lot of users on it you'll need the disk space for user blobs.

Table of Contents

• PDS Setup
• Using rclone to migrate from a blobstore on disk to S3

PDS Setup

I'm not going to go over the setup of a cloud S3 object store since UpCloud does here and @yehuda.turtleis.land does a great job here for DigitalOcean Spaces.

1. Open your pds.env file on your PDS, usually at /pds/pds.env.
2. Remove the env PDS_BLOBSTORE_DISK_LOCATION (Can't have both set)
3. Set some new env variables in the pds.env as below

For UpCloud

PDS_BLOBSTORE_S3_BUCKET={What you named the bucket}
PDS_BLOBSTORE_S3_REGION={Bottom left of Overview top section}
PDS_BLOBSTORE_S3_ENDPOINT={S3 endpoint under Public access}
PDS_BLOBSTORE_S3_ACCESS_KEY_ID={Under users tab}
PDS_BLOBSTORE_S3_SECRET_ACCESS_KEY={Under users tab}
PDS_BLOBSTORE_S3_FORCE_PATH_STYLE=false

Digital ocean gives an Origin endpoint url like thishttps://{bucketname}.{region}.digitaloceanspaces.com that holds most of the info. Can always look here for a much more indepth setup for Digital Ocean.

PDS_BLOBSTORE_S3_BUCKET=bucketname
PDS_BLOBSTORE_S3_REGION=region
PDS_BLOBSTORE_S3_ENDPOINT=https://{region}.digitaloceanspaces.com
PDS_BLOBSTORE_S3_ACCESS_KEY_ID={Settings -> Access Keys}
PDS_BLOBSTORE_S3_SECRET_ACCESS_KEY={Settings -> Access Keys}
PDS_BLOBSTORE_S3_FORCE_PATH_STYLE=false

4. Run docker compose down & docker compose up -d in the /pds directory and if everything is right your PDS should be now storing and reading blobs from the object store! But this doesn't move the already existing blobs over, for that check out the next section.

Using rclone to migrate from a blobstore on disk to S3

When I first started looking into doing this, I was wrongly assuming that I would need to write a script or a simple Rust binary to help users migrate to an S3 object storage. Thankfully, @bnewbold.net pointed out I was trying to reinvent the wheel.

I recommend before doing this making sure you have a backup of your server. Data lose is not expected, but you never know

1. You'll want to turn of the PDS while you do this with docker compose down in the PDS directory (usually /pds)
2. Install rclone. I used sudo -v ; curl https://rclone.org/install.sh | sudo bash
3. Create a ~/.config/rclone/rclone.conf

[pds-blobs]
type = s3
provider = {UpCloud or DigitalOcean}
env_auth = false
access_key_id = {your_access_key_id}
secret_access_key = {your_secret_key
endpoint = {S3 endpoint minus the https so like atl1.digitaloceanspaces.com}
acl = private

A couple of notes before you run the command to move the blobs over

• If you have a bunch of blobs, it will take a while. If you have a lot of users probably want to inform them ahead of time
• There are a couple of rclone options, may find one you prefer. My example uses sync. But there is also move, copy, and more commands here
• You want to make sure the PDS is not running by doing the docker compose down.

4. Once you're good with everything and prepared to take the time to move the blobs, you can run this command.

rclone sync /pds/blocks pds-blobs:{bucket-name}/blocks -P

To break that command down a bit

• rclone - the tool
• sync - the command
• /pds/blocks - the local location of your blobs on the PDS currently
• pds-blobs: - The name you set in ~/.config/rclone/rclone.conf
• /blocks - folder location on the object storage. The PDS expects there to be a /blocks folder when reading
• -P - Shows the progress of it

5. Once the upload is done can start your pds backup with docker compose up -d in the /pds directory

And that's it! Your PDS should now be using the S3 object storage for blobs, saving hard drive space and some money. If you're wanting to try out UpCloud you can use my referral link to sign up for $25 of free credits or if you want to go with Digital Ocean you can use my referral link to get $200 in credit over 60 days.