Driver interface and rclone implementation

The Driver interfaceCopy Anchor Link

store/driver defines exactly one thing: store bytes in some backend, read them back, inspect, delete, and list them.

type Driver interface {
	Put(ctx context.Context, key string, r io.Reader, size int64) error
	Open(ctx context.Context, key string) (io.ReadCloser, error)
	Stat(ctx context.Context, key string) (*ObjectInfo, error)
	Delete(ctx context.Context, key string) error
	List(ctx context.Context, dir string) ([]ObjectInfo, error)
}

Rules every implementation follows:

• Safe for concurrent use: one driver instance may be used from many goroutines at once.
• Honors ctx: every method respects cancellation and deadlines on ctx.
• Detect missing objects with errors.Is: implementations wrap ErrNotFound with context such as the key, so never compare with ==.
• Keys use / as the separator, like photos/2026/a.png, relative to the driver's root path.

ObjectInfo is what Stat and List return:

The rclone implementationCopy Anchor Link

store/rclone implements Driver on top of rclone backends. It builds rclone's fs.Fs directly in-process: parameters are passed to the backend as a map, with no rclone config file, no connection string, and no rclone subprocess.

Opening a driverCopy Anchor Link

d, err := rclone.Open(ctx, rclone.Options{
	Backend: "s3",
	Params: map[string]string{
		"provider":          "Cloudflare",
		"access_key_id":     "<access key>",
		"secret_access_key": "<secret key>",
		"endpoint":          "https://<account id>.r2.cloudflarestorage.com",
	},
	Path: "my-bucket/images",
})
if err != nil {
	return err
}
defer func() { _ = d.Close(ctx) }()

Fields of rclone.Options:

Errors returned by Open never carry the parameter map, so credentials in it cannot leak into logs through error messages.

Content-Type on writeCopy Anchor Link

Put streams the data; the driver itself never buffers the whole object. It first reads up to 512 bytes from the start of the data, determines the Content-Type with net/http.DetectContentType (the WHATWG content sniffing algorithm), hands it to the backend along with the object, and then writes the remaining bytes. Object keys need no file extension: objects in S3 and other object stores get their real type (such as image/png) instead of application/octet-stream, so opening the store's public link directly in a browser displays the image instead of downloading it.

• Encrypted drivers do not pass the type on to the store: rclone's crypt layer deliberately hides the plaintext type, and the stored object is ciphertext.
• Backends that do not record a Content-Type (such as local) ignore it.

s3 skips the bucket check by defaultCopy Anchor Link

When Backend is s3 and Params has no no_check_bucket, the driver adds no_check_bucket=true. An explicit value is never overridden.

The reason: S3-compatible stores such as R2 and MinIO are often accessed with credentials scoped to a bucket, without permission to create buckets. By default rclone tries to create the bucket before uploading; when the bucket exists but the credentials may not create it, the request fails with 403 and the write fails, while a read-only health check (see Multi-store failover) still passes. The store looks healthy but rejects writes. The cost of skipping the check is that the bucket must already exist.

EnsureBucketCopy Anchor Link

When your program needs to create the bucket, use (*rclone.Driver).EnsureBucket. It makes the root directory, which for s3 creates the bucket named in Path; an existing bucket is not an error.

b, err := rclone.Open(ctx, rclone.Options{
	Backend: "s3",
	Params: map[string]string{
		"provider":          "Cloudflare",
		"access_key_id":     "<access key>",
		"secret_access_key": "<secret key>",
		"endpoint":          "https://<account id>.r2.cloudflarestorage.com",
		"no_check_bucket":   "false", // required, otherwise bucket creation is skipped and EnsureBucket does nothing
	},
	Path: "my-bucket",
})
if err != nil {
	return err
}
defer func() { _ = b.Close(ctx) }()
if err := b.EnsureBucket(ctx); err != nil {
	return err // usually the credentials lack permission to create buckets
}

EnsureBucket is not part of the Driver interface because it only makes sense for object storage. The application opens a temporary driver with no_check_bucket=false just for this call, and keeps using a driver with the default settings for everyday reads and writes.

Encoding password parametersCopy Anchor Link

rclone expects some parameters in the encoded form produced by rclone obscure, for example pass for webdav, sftp, and ftp. Which parameters these are is declared by each backend's option metadata. The driver hands Params to the backend unchanged, so these values must be encoded beforehand: a plaintext value left unencoded is decoded by the backend into garbage, and authentication fails even though the credential is correct. Not every secret is in this category; s3's secret_access_key, for instance, is passed as-is.

The package provides two functions for the encoding. Both pick the password parameters from the backend's metadata, modify the map in place, skip empty values, and return an error for an unknown backend name:

ObscurePlainSecrets decides "already encoded" the same way rclone does: a value rclone can decode counts as encoded, matching rclone config create run without --obscure / --no-obscure. If it misjudges a plaintext value, encode it with rclone obscure first and pass the result. The application uses ObscureSecrets for its console template forms (the values are known to be plaintext) and ObscurePlainSecrets for custom parameters (both forms are accepted).

params := map[string]string{
	"url":    "https://dav.example.com/remote.php/dav/files/me",
	"vendor": "nextcloud",
	"user":   "me",
	"pass":   "plaintext password",
}
if err := rclone.ObscureSecrets("webdav", params); err != nil {
	return err
}
d, err := rclone.Open(ctx, rclone.Options{Backend: "webdav", Params: params, Path: "images"})

A related helper, SensitiveParams(backend), returns the names of the parameters a backend declares as passwords or sensitive (IsPassword or Sensitive in its option metadata), sorted and without duplicates, by the same rule as rclone config redacted: besides passwords, keys and tokens, identity information such as access key IDs and account names is included. Use it to decide which values to mask when echoing a configuration and which to scrub from error messages; the application's edit read-back and error scrubbing rely on it.

OAuth token write-back and OnChangeCopy Anchor Link

Cloud drive backends such as Google Drive, OneDrive, and Dropbox use OAuth. The library has no authorization flow: run rclone authorize "drive" on your machine (substituting the backend name), and put the resulting JSON token in Params["token"]. If you authorized with your own app, put its client_id and client_secret in Params as well: a token can only be renewed by the app that issued it.

Open only constructs the backend; it does not run the configuration steps rclone config performs after authorization. Some backends need parameters written by those steps — OneDrive, for example, needs drive_id and drive_type. Configure the remote once with rclone config on your machine, then copy those values from rclone.conf into Params.

When the token is about to expire, rclone refreshes it itself and writes the new value back as a parameter. On a write-back, the driver first updates its in-memory parameters and then calls OnChange(key, value).

• With a nil OnChange, the new value lives only in memory, and after a restart the process starts again from the old token. For providers that rotate the refresh token on every refresh, the old token is already void by then, so authorization breaks after the restart. With cloud drive backends, always supply OnChange and persist the value.
• The callback may run on any goroutine, including before Open returns (write-backs during construction). It runs synchronously on rclone's call path, so return quickly and make it safe for concurrent use.
• The callback has no return value; if persisting fails, logging it is up to you.
• Do not run a separate token refresh loop. Two refreshers invalidate each other's refresh tokens. Calling a read-only operation periodically (for example failover.Probe) is enough to keep rclone refreshing on time.

d, err := rclone.Open(ctx, rclone.Options{
	Backend: "drive",
	Params:  map[string]string{"token": tokenJSON},
	Path:    "imgyoung",
	OnChange: func(key, value string) {
		if err := saveParam(key, value); err != nil { // saveParam: your persistence function
			log.Printf("persist parameter %s: %v", key, err)
		}
	},
})

The crypt layerCopy Anchor Link

When Crypt is non-nil, the driver stacks rclone's crypt layer on top of the backend: object content is encrypted as it streams, and file names are encrypted with rclone's standard mode. Reads and writes through the driver encrypt and decrypt transparently, and List returns decrypted keys; the storage provider only ever sees ciphertext and encrypted names.

Under the hood, Open first builds the inner backend, registers it in rclone's process-wide fs cache under a random name (like imgyoung-0123abcd), and then builds the crypt layer with that name as its remote.

crypt is a public rclone format, so standard rclone can decrypt the data without this library: generate a configuration file with ConfigFile, described in the next section. The application's channel encryption is built on this same layer, with passwords derived from the master key; see Channel encryption.

Exporting an rclone configuration fileCopy Anchor Link

func ConfigFile(opts Options, remote, cryptRemote string) (string, error)

ConfigFile produces the text of an rclone configuration file (rclone.conf) equivalent to the driver Open(opts) builds, so standard rclone can read and write the same data with it:

• The backend section is named remote, and its parameters are exactly what the driver hands to the backend, including the defaults this package adds (such as s3's no_check_bucket = true). Empty values are omitted, because both the driver and rclone treat an empty value as unset. Password parameters are already in rclone's encoded form by the Options convention and are written as they are.
• When opts.Crypt is non-nil, a second crypt section named cryptRemote is written: its remote points to <remote>:<opts.Path>, password and password2 are the encoded Password and Salt, and filename_encryption = standard, matching the driver's crypt layer. Reads and writes through it are plaintext.
• Without encryption, the backend section carries no path; use it as <remote>:<opts.Path>. OnChange plays no part.
• Equivalence is guaranteed by construction: ConfigFile and Open share the same default filling and crypt parameter generation, so they never compute things separately.

text, err := rclone.ConfigFile(rclone.Options{
	Backend: "s3",
	Params: map[string]string{
		"provider":          "Cloudflare",
		"access_key_id":     "<access key>",
		"secret_access_key": "<secret key>",
		"endpoint":          "https://<account id>.r2.cloudflarestorage.com",
	},
	Path:  "my-bucket/images",
	Crypt: &rclone.CryptOptions{Password: "<password>", Salt: "<salt>"},
}, "mystore", "mystore-crypt")

The resulting text looks like this (encoding uses a random IV, so password and password2 differ on every call, yet always decode to the same values):

[mystore]
type = s3
access_key_id = <access key>
endpoint = https://<account id>.r2.cloudflarestorage.com
no_check_bucket = true
provider = Cloudflare
secret_access_key = <secret key>

[mystore-crypt]
type = crypt
filename_encryption = standard
password = <encoded Password>
password2 = <encoded Salt>
remote = mystore:my-bucket/images

rclone --config rclone.conf ls mystore-crypt:

The text is serialized with the same INI library rclone uses for its configuration files, then read back the way rclone reads it and checked. If a parameter cannot be written into the file as it is, the function returns an error wrapping ErrUnrepresentable instead of producing a file that is not equivalent to the driver. The error text names only the section and the parameter, never the value (the text itself is in Chinese):

Other errors: an unknown backend name, a section name that breaks rclone's naming rules (only letters, digits, _, -, ., +, @ and spaces), or a failure to encode a password.

CloseCopy Anchor Link

Always call Close(ctx) when you are done with a driver. If the backend provides a shutdown hook (rclone's Shutdowner), it stops the backend's background tasks and connection pools. An encrypted driver also removes the inner backend from rclone's fs cache, and the cache closes the inner backend as it is removed. Do not use a driver after Close.

Close is a method on *rclone.Driver, not part of the Driver interface. If you pass drivers around as driver.Driver, keep the concrete pointer as well so you can close it.

Complete exampleCopy Anchor Link

This is store/rclone/example_test.go from the repository. It runs with go test ./store/rclone, and each example prints exactly what its Output comment lists (the comments in the source are in Chinese): ExampleSensitiveParams queries the sensitive parameters of the webdav backend, and Example walks through write, stat, read back, and delete on rclone's memory backend.

package rclone_test

import (
	"context"
	"errors"
	"fmt"
	"io"
	"strings"

	"github.com/x-haose/imgyoung/store/driver"
	"github.com/x-haose/imgyoung/store/rclone"
)

// ExampleSensitiveParams 查询 webdav 后端的敏感参数:密码(IsPassword)与账号名、令牌(Sensitive)都在其中。
func ExampleSensitiveParams() {
	names, err := rclone.SensitiveParams("webdav")
	if err != nil {
		fmt.Println("sensitive params:", err)
		return
	}
	fmt.Println(strings.Join(names, " "))
	// Output:
	// bearer_token pass user
}

// Example 用 rclone 的内存后端走一遍驱动的基本操作:写入、查询、读回、删除。
func Example() {
	ctx := context.Background()
	d, err := rclone.Open(ctx, rclone.Options{Backend: "memory", Path: "bucket"})
	if err != nil {
		fmt.Println("open:", err)
		return
	}
	defer func() { _ = d.Close(ctx) }()

	if err := d.Put(ctx, "hello.txt", strings.NewReader("hello"), 5); err != nil {
		fmt.Println("put:", err)
		return
	}
	info, err := d.Stat(ctx, "hello.txt")
	if err != nil {
		fmt.Println("stat:", err)
		return
	}
	fmt.Println(info.Key, info.Size)

	rc, err := d.Open(ctx, "hello.txt")
	if err != nil {
		fmt.Println("open object:", err)
		return
	}
	data, err := io.ReadAll(rc)
	_ = rc.Close()
	if err != nil {
		fmt.Println("read:", err)
		return
	}
	fmt.Println(string(data))

	if err := d.Delete(ctx, "hello.txt"); err != nil {
		fmt.Println("delete:", err)
		return
	}
	_, err = d.Stat(ctx, "hello.txt")
	fmt.Println(errors.Is(err, driver.ErrNotFound))
	// Output:
	// hello.txt 5
	// hello
	// true
}

The memory and local backends are well suited to integration tests: the first keeps data in process memory only, the second pairs with t.TempDir(), and neither needs a cloud account.

Writing your own driverCopy Anchor Link

Without rclone, implement driver.Driver yourself and it works with store/failover as is. Key points:

1. Return ErrNotFound for missing objects: when Open, Stat, or Delete cannot find the object, return an error that errors.Is(err, driver.ErrNotFound) matches. failover.Open relies on it to tell "object does not exist" from "store is broken", and failover.Probe treats ErrNotFound from Stat as "store reachable".
2. Put reads exactly size bytes and returns an error if it gets fewer; do not close r. After a failure the caller may retry with the same key, and the new Put should completely replace whatever was left behind.
3. List is recursive: an empty dir means all objects; keys are full paths relative to the root; a missing directory returns an empty result and nil.
4. Fill ModTime with the real modification time: callers use it to judge how old an object is. The application's reconciliation, for example, only reclaims unknown objects whose modification time is older than its grace period.
5. Be safe for concurrent use and honor ctx. Check that the implementation is complete with a compile-time assertion: var _ driver.Driver = (*MemDriver)(nil).

Here is an implementation that keeps objects in process memory, ready to use in tests:

package memstore

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"strings"
	"sync"
	"time"

	"github.com/x-haose/imgyoung/store/driver"
)

// MemDriver is a Driver that keeps objects in process memory.
type MemDriver struct {
	mu      sync.RWMutex
	objects map[string]memObject
}

type memObject struct {
	data    []byte
	modTime time.Time
}

var _ driver.Driver = (*MemDriver)(nil)

func New() *MemDriver {
	return &MemDriver{objects: map[string]memObject{}}
}

func (m *MemDriver) Put(ctx context.Context, key string, r io.Reader, size int64) error {
	data, err := io.ReadAll(io.LimitReader(r, size))
	if err != nil {
		return fmt.Errorf("read %q: %w", key, err)
	}
	if int64(len(data)) != size {
		return fmt.Errorf("write %q: want %d bytes, got %d: %w", key, size, len(data), io.ErrUnexpectedEOF)
	}
	if err := ctx.Err(); err != nil {
		return err
	}
	m.mu.Lock()
	defer m.mu.Unlock()
	m.objects[key] = memObject{data: data, modTime: time.Now()}
	return nil
}

func (m *MemDriver) Open(ctx context.Context, key string) (io.ReadCloser, error) {
	if err := ctx.Err(); err != nil {
		return nil, err
	}
	m.mu.RLock()
	defer m.mu.RUnlock()
	o, ok := m.objects[key]
	if !ok {
		return nil, fmt.Errorf("object %q: %w", key, driver.ErrNotFound)
	}
	return io.NopCloser(bytes.NewReader(o.data)), nil
}

func (m *MemDriver) Stat(ctx context.Context, key string) (*driver.ObjectInfo, error) {
	if err := ctx.Err(); err != nil {
		return nil, err
	}
	m.mu.RLock()
	defer m.mu.RUnlock()
	o, ok := m.objects[key]
	if !ok {
		return nil, fmt.Errorf("object %q: %w", key, driver.ErrNotFound)
	}
	return &driver.ObjectInfo{Key: key, Size: int64(len(o.data)), ModTime: o.modTime}, nil
}

func (m *MemDriver) Delete(ctx context.Context, key string) error {
	if err := ctx.Err(); err != nil {
		return err
	}
	m.mu.Lock()
	defer m.mu.Unlock()
	if _, ok := m.objects[key]; !ok {
		return fmt.Errorf("object %q: %w", key, driver.ErrNotFound)
	}
	delete(m.objects, key)
	return nil
}

func (m *MemDriver) List(ctx context.Context, dir string) ([]driver.ObjectInfo, error) {
	if err := ctx.Err(); err != nil {
		return nil, err
	}
	prefix := ""
	if dir != "" {
		prefix = strings.TrimSuffix(dir, "/") + "/"
	}
	m.mu.RLock()
	defer m.mu.RUnlock()
	var out []driver.ObjectInfo
	for k, o := range m.objects {
		if strings.HasPrefix(k, prefix) {
			out = append(out, driver.ObjectInfo{Key: k, Size: int64(len(o.data)), ModTime: o.modTime})
		}
	}
	return out, nil
}