Garage S3 Setup

Garage is an open-source, lightweight distributed S3-compatible storage system designed to be self-hosted on commodity hardware, bare-metal servers, or home labs with minimal resource usage.

---

Best Practices: Bucket & Cluster Setup

Follow these recommendations when configuring Garage for CI/CD caching:

1. Cluster Layout & Replication

• Single-Node CI Runner Cache: Set replication_factor = 1 in garage.toml if running on a single build server or local runner node to maximize available storage capacity.
• Multi-Node Cluster: Use replication_factor = 3 across multiple physical nodes or availability zones for high availability and zero-downtime runner caching.

2. Disk Space Protection & Bucket Quotas

Continuous integration workloads can quickly consume hundreds of gigabytes of storage if left unchecked. Garage provides built-in quota controls:

# Set a hard storage quota on the CI cache bucket (e.g. 50 GB)
garage bucket set --max-size 50G ci-cache

# Set a limit on the total number of cache objects (e.g. 10,000 objects)
garage bucket set --max-objects 10000 ci-cache

3. Periodic Garbage Collection

Run periodic garbage collection on your Garage cluster to prune stale blocks and reclaim disk space:

# Check block storage status and run GC
garage gc

4. Network Security

• The Garage S3 API port (3900) and Admin port (3902) should remain on internal networks, private VPCs, or encrypted VPN overlays (e.g. Tailscale / WireGuard).
• If exposing Garage across networks, place it behind an SSL/TLS reverse proxy (e.g. Traefik, Caddy, Nginx).

---

Credentials & Key Management

Garage provides a clean CLI for managing credentials and granting least-privilege bucket access:

1. Create a Dedicated API Key

# Create an access key named 'ci-runner-key'
garage key create ci-runner-key

Garage will output the Key ID (GK...) and Secret Key.

2. Create the Cache Bucket & Authorize Key

# Create the bucket
garage bucket create ci-cache

# Grant read and write permissions to the key
garage bucket allow ci-cache --read --write --key ci-runner-key

3. Verify Key Permissions

# Inspect key details and bound buckets at any time
garage key info ci-runner-key

4. Configure GitHub Secrets

Save the credentials in your repository or organization secrets:

Secret Name	Source
GARAGE_ACCESS_KEY	Output of garage key info ci-runner-key (Key ID)
GARAGE_SECRET_KEY	Output of garage key info ci-runner-key (Secret Key)

---

Workflow Configuration

- name: Cache dependencies using self-hosted Garage
  uses: xSAVIKx/cloud-cache-action@v1
  with:
    bucket: ci-cache
    endpoint: http://garage.internal:3900 # Replace with your Garage S3 endpoint
    provider: garage
    access-key: ${{ secrets.GARAGE_ACCESS_KEY }}
    secret-key: ${{ secrets.GARAGE_SECRET_KEY }}
    key: ${{ runner.os }}-build-${{ hashFiles('**/lock') }}
    path: build/

---

Running Garage Locally for Integration Testing

You can spin up Garage locally using the repository's docker-compose.test.yml:

docker compose -f docker-compose.test.yml up -d garage

docker-compose.test.yml (Click to view Compose definition)

name: cloud-cache-test

services:
  # Garage S3 compatible storage (https://garagehq.deuxfleurs.fr)
  garage:
    image: dxflrs/garage:v2.4.1
    container_name: cloud-cache-garage
    ports:
      - '3900:3900' # S3 API
      - '3902:3902' # Admin API
    volumes:
      - ./tests/fixtures/garage.toml:/etc/garage.toml:ro
    environment:
      - RUST_LOG=info
    restart: unless-stopped

  # SeaweedFS S3 compatible storage (https://github.com/seaweedfs/seaweedfs)
  seaweedfs:
    image: chrislusf/seaweedfs:4.46
    container_name: cloud-cache-seaweedfs
    ports:
      - '8333:8333' # S3 API
      - '9333:9333' # Master
    command: 'server -s3 -s3.port=8333'
    restart: unless-stopped

  # MinIO S3 compatible storage (Alternative / Local testing)
  minio:
    image: minio/minio:RELEASE.2025-09-07T16-13-09Z
    container_name: cloud-cache-minio
    ports:
      - '9000:9000'
      - '9001:9001'
    environment:
      - MINIO_ROOT_USER=minioadmin
      - MINIO_ROOT_PASSWORD=minioadmin
    command: server /data --console-address ":9001"
    restart: unless-stopped

The Garage service requires a configuration file mounted at /etc/garage.toml. You can use the repository's tests/fixtures/garage.toml:

tests/fixtures/garage.toml (Click to view Garage configuration)

metadata_dir = "/tmp/garage/meta"
data_dir = "/tmp/garage/data"
db_engine = "sqlite"

replication_factor = 1

rpc_bind_addr = "0.0.0.0:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f"

[s3_api]
s3_region = "garage"
api_bind_addr = "0.0.0.0:3900"
root_domain = ".s3.garage"

[s3_web]
bind_addr = "0.0.0.0:3902"
root_domain = ".web.garage"

[admin]
api_bind_addr = "0.0.0.0:3903"
admin_token = "garageadmintoken123456"

Initializing Garage S3 Locally

After starting the container, initialize the layout and credentials:

# 1. Assign layout to the local node
NODE_ID=$(docker exec cloud-cache-garage /garage status | awk '/Node/ {print $2}')
docker exec -ti cloud-cache-garage /garage layout assign -z dc1 -c 10G $NODE_ID
docker exec -ti cloud-cache-garage /garage layout apply --version 1

# 2. Create access key and bucket
docker exec -ti cloud-cache-garage /garage key create ci-cache-key
docker exec -ti cloud-cache-garage /garage bucket create ci-cache
docker exec -ti cloud-cache-garage /garage bucket allow ci-cache --read --write --key ci-cache-key