# Driver interface and rclone implementation (/en/docs/library/driver)
The Driver contract, rclone.Open options, password parameter encoding, OAuth token write-back, the crypt layer, and writing your own driver.
## The Driver interface [#the-driver-interface]
`store/driver` defines exactly one thing: store bytes in some backend, read them back, inspect, delete, and list them.
```go
type Driver interface {
Put(ctx context.Context, key string, r io.Reader, size int64) error
Open(ctx context.Context, key string) (io.ReadCloser, error)
Stat(ctx context.Context, key string) (*ObjectInfo, error)
Delete(ctx context.Context, key string) error
List(ctx context.Context, dir string) ([]ObjectInfo, error)
}
```
| Method | Semantics | When the object is missing |
|---|---|---|
| `Put` | Streams `size` bytes from `r` into `key`. `size` must be known up front; the caller owns and closes `r` | — |
| `Open` | Returns a reader for the object; the caller must `Close` it | Returns an error wrapping `ErrNotFound` |
| `Stat` | Returns the object's metadata | Returns an error wrapping `ErrNotFound` |
| `Delete` | Deletes the object | Returns an error wrapping `ErrNotFound`; the caller decides whether that counts as success |
| `List` | Recursively lists every object under `dir`; an empty `dir` means the whole store. Returned keys are full paths relative to the store root | A missing directory yields an empty result, not an error |
Rules every implementation follows:
- **Safe for concurrent use**: one driver instance may be used from many goroutines at once.
- **Honors `ctx`**: every method respects cancellation and deadlines on `ctx`.
- **Detect missing objects with `errors.Is`**: implementations wrap `ErrNotFound` with context such as the key, so never compare with `==`.
- **Keys use `/` as the separator**, like `photos/2026/a.png`, relative to the driver's root path.
`ObjectInfo` is what `Stat` and `List` return:
## The rclone implementation [#the-rclone-implementation]
`store/rclone` implements `Driver` on top of rclone backends. It builds rclone's `fs.Fs` directly in-process: parameters
are passed to the backend as a map, with no rclone config file, no connection string, and no rclone subprocess.
### Opening a driver [#opening-a-driver]
```go
d, err := rclone.Open(ctx, rclone.Options{
Backend: "s3",
Params: map[string]string{
"provider": "Cloudflare",
"access_key_id": "",
"secret_access_key": "",
"endpoint": "https://.r2.cloudflarestorage.com",
},
Path: "my-bucket/images",
})
if err != nil {
return err
}
defer func() { _ = d.Close(ctx) }()
```
Fields of `rclone.Options`:
Errors returned by `Open` never carry the parameter map, so credentials in it cannot leak into logs through error messages.
### Content-Type on write [#content-type-on-write]
`Put` streams the data; the driver itself never buffers the whole object. It first reads up to 512 bytes from the start of the
data, determines the Content-Type with `net/http.DetectContentType` (the WHATWG content sniffing algorithm), hands it to
the backend along with the object, and then writes the remaining bytes. Object keys need no file extension: objects in
S3 and other object stores get their real type (such as `image/png`) instead of `application/octet-stream`, so opening
the store's public link directly in a browser displays the image instead of downloading it.
- Encrypted drivers do not pass the type on to the store: rclone's crypt layer deliberately hides the plaintext type, and the stored object is ciphertext.
- Backends that do not record a Content-Type (such as `local`) ignore it.
### s3 skips the bucket check by default [#s3-skips-the-bucket-check-by-default]
When `Backend` is `s3` and `Params` has no `no_check_bucket`, the driver adds `no_check_bucket=true`. An explicit value is never overridden.
The reason: S3-compatible stores such as R2 and MinIO are often accessed with credentials scoped to a bucket, without
permission to create buckets. By default rclone tries to create the bucket before uploading; when the bucket exists but
the credentials may not create it, the request fails with 403 and the write fails, while a read-only health check (see
[Multi-store failover](./failover.mdx)) still passes. The store looks healthy but rejects writes. The cost of skipping
the check is that **the bucket must already exist**.
### EnsureBucket [#ensurebucket]
When your program needs to create the bucket, use `(*rclone.Driver).EnsureBucket`. It makes the root directory, which
for s3 creates the bucket named in `Path`; an existing bucket is not an error.
```go
b, err := rclone.Open(ctx, rclone.Options{
Backend: "s3",
Params: map[string]string{
"provider": "Cloudflare",
"access_key_id": "",
"secret_access_key": "",
"endpoint": "https://.r2.cloudflarestorage.com",
"no_check_bucket": "false", // required, otherwise bucket creation is skipped and EnsureBucket does nothing
},
Path: "my-bucket",
})
if err != nil {
return err
}
defer func() { _ = b.Close(ctx) }()
if err := b.EnsureBucket(ctx); err != nil {
return err // usually the credentials lack permission to create buckets
}
```
`EnsureBucket` is not part of the `Driver` interface because it only makes sense for object storage. The application
opens a temporary driver with `no_check_bucket=false` just for this call, and keeps using a driver with the default
settings for everyday reads and writes.
### Encoding password parameters [#encoding-password-parameters]
rclone expects some parameters in the encoded form produced by `rclone obscure`, for example `pass` for webdav, sftp,
and ftp. Which parameters these are is declared by each backend's option metadata. The driver hands `Params` to the
backend unchanged, so these values must be **encoded beforehand**: a plaintext value left unencoded is decoded by the
backend into garbage, and authentication fails even though the credential is correct. Not every secret is in this
category; s3's `secret_access_key`, for instance, is passed as-is.
The package provides two functions for the encoding. Both pick the password parameters from the backend's metadata,
modify the map in place, skip empty values, and return an error for an unknown backend name:
| | `ObscureSecrets` | `ObscurePlainSecrets` |
|---|---|---|
| Which values it encodes | Every password parameter | Password parameters that are not already encoded |
| Use it when | The values are known to be plaintext, such as form input | The source is unknown: plaintext, or an encoded value copied from rclone.conf |
| When it guesses wrong | An already-encoded value gets encoded twice; authentication fails | A plaintext value that happens to look encoded passes through unchanged; authentication fails |
`ObscurePlainSecrets` decides "already encoded" the same way rclone does: a value rclone can decode counts as encoded,
matching `rclone config create` run without `--obscure` / `--no-obscure`. If it misjudges a plaintext value, encode it
with `rclone obscure` first and pass the result. The application uses `ObscureSecrets` for its console template forms
(the values are known to be plaintext) and `ObscurePlainSecrets` for custom parameters (both forms are accepted).
```go
params := map[string]string{
"url": "https://dav.example.com/remote.php/dav/files/me",
"vendor": "nextcloud",
"user": "me",
"pass": "plaintext password",
}
if err := rclone.ObscureSecrets("webdav", params); err != nil {
return err
}
d, err := rclone.Open(ctx, rclone.Options{Backend: "webdav", Params: params, Path: "images"})
```
A related helper, `SensitiveParams(backend)`, returns the names of the parameters a backend declares as passwords or
sensitive (`IsPassword` or `Sensitive` in its option metadata), sorted and without duplicates, by the same rule as
`rclone config redacted`: besides passwords, keys and tokens, identity information such as access key IDs and account
names is included. Use it to decide which values to mask when echoing a configuration and which to scrub from error
messages; the application's edit read-back and error scrubbing rely on it.
### OAuth token write-back and OnChange [#oauth-token-write-back-and-onchange]
Cloud drive backends such as Google Drive, OneDrive, and Dropbox use OAuth. The library has no authorization flow: run
`rclone authorize "drive"` on your machine (substituting the backend name), and put the resulting JSON token in
`Params["token"]`. If you authorized with your own app, put its `client_id` and `client_secret` in `Params` as well:
a token can only be renewed by the app that issued it.
`Open` only constructs the backend; it does not run the configuration steps `rclone config` performs after
authorization. Some backends need parameters written by those steps — OneDrive, for example, needs `drive_id` and
`drive_type`. Configure the remote once with `rclone config` on your machine, then copy those values from rclone.conf into `Params`.
When the token is about to expire, rclone refreshes it itself and writes the new value back as a parameter. On a
write-back, the driver first updates its in-memory parameters and then calls `OnChange(key, value)`.
- With a nil `OnChange`, the new value lives only in memory, and after a restart the process starts again from the old
token. For providers that rotate the refresh token on every refresh, the old token is already void by then, so
authorization breaks after the restart. With cloud drive backends, always supply `OnChange` and persist the value.
- The callback may run on any goroutine, including before `Open` returns (write-backs during construction). It runs
synchronously on rclone's call path, so return quickly and make it safe for concurrent use.
- The callback has no return value; if persisting fails, logging it is up to you.
- Do not run a separate token refresh loop. Two refreshers invalidate each other's refresh tokens. Calling a read-only
operation periodically (for example [`failover.Probe`](./failover.mdx)) is enough to keep rclone refreshing on time.
```go
d, err := rclone.Open(ctx, rclone.Options{
Backend: "drive",
Params: map[string]string{"token": tokenJSON},
Path: "imgyoung",
OnChange: func(key, value string) {
if err := saveParam(key, value); err != nil { // saveParam: your persistence function
log.Printf("persist parameter %s: %v", key, err)
}
},
})
```
### The crypt layer [#the-crypt-layer]
When `Crypt` is non-nil, the driver stacks rclone's crypt layer on top of the backend: object content is encrypted as
it streams, and file names are encrypted with rclone's `standard` mode. Reads and writes through the driver encrypt and
decrypt transparently, and `List` returns decrypted keys; the storage provider only ever sees ciphertext and encrypted names.
Under the hood, `Open` first builds the inner backend, registers it in rclone's process-wide fs cache under a random
name (like `imgyoung-0123abcd`), and then builds the crypt layer with that name as its `remote`.
Password and Salt are the only way to decrypt. Without them, the objects in the store cannot be recovered.
crypt is a public rclone format, so standard rclone can decrypt the data without this library: generate a
configuration file with `ConfigFile`, described in the next section. The application's channel encryption is built on
this same layer, with passwords derived from the master key; see [Channel encryption](../channels/encryption.mdx).
### Exporting an rclone configuration file [#config-file]
```go
func ConfigFile(opts Options, remote, cryptRemote string) (string, error)
```
`ConfigFile` produces the text of an rclone configuration file (rclone.conf) equivalent to the driver `Open(opts)`
builds, so standard rclone can read and write the same data with it:
- The backend section is named `remote`, and its parameters are exactly what the driver hands to the backend,
including the defaults this package adds (such as s3's `no_check_bucket = true`). Empty values are omitted, because
both the driver and rclone treat an empty value as unset. Password parameters are already in rclone's encoded form
by the `Options` convention and are written as they are.
- When `opts.Crypt` is non-nil, a second crypt section named `cryptRemote` is written: its `remote` points to
`:`, `password` and `password2` are the encoded `Password` and `Salt`, and
`filename_encryption = standard`, matching the driver's crypt layer. Reads and writes through it are plaintext.
- Without encryption, the backend section carries no path; use it as `:`. `OnChange` plays no part.
- Equivalence is guaranteed by construction: `ConfigFile` and `Open` share the same default filling and crypt
parameter generation, so they never compute things separately.
```go
text, err := rclone.ConfigFile(rclone.Options{
Backend: "s3",
Params: map[string]string{
"provider": "Cloudflare",
"access_key_id": "",
"secret_access_key": "",
"endpoint": "https://.r2.cloudflarestorage.com",
},
Path: "my-bucket/images",
Crypt: &rclone.CryptOptions{Password: "", Salt: ""},
}, "mystore", "mystore-crypt")
```
The resulting text looks like this (encoding uses a random IV, so `password` and `password2` differ on every call,
yet always decode to the same values):
```ini title="rclone.conf"
[mystore]
type = s3
access_key_id =
endpoint = https://.r2.cloudflarestorage.com
no_check_bucket = true
provider = Cloudflare
secret_access_key =
[mystore-crypt]
type = crypt
filename_encryption = standard
password =
password2 =
remote = mystore:my-bucket/images
```
```bash title="Terminal"
rclone --config rclone.conf ls mystore-crypt:
```
The text is serialized with the same INI library rclone uses for its configuration files, then read back the way
rclone reads it and checked. If a parameter cannot be written into the file as it is, the function returns an error
wrapping `ErrUnrepresentable` instead of producing a file that is not equivalent to the driver. The error text names
only the section and the parameter, never the value (the text itself is in Chinese):
| Situation | Error text |
|---|---|
| A parameter value contains a line break (`\n` or `\r`) | `参数值无法写进 rclone 配置文件: 远端 "" 的参数 "" 含换行` |
| The generated text cannot be parsed the way rclone reads it | `参数值无法写进 rclone 配置文件: 生成的文本按 rclone 的读法无法解析` |
| A parameter reads back as something other than its original value (leading or trailing whitespace, a `%(name)s` style reference, and so on) | `参数值无法写进 rclone 配置文件: 远端 "" 的参数 ""` |
Other errors: an unknown backend name, a section name that breaks rclone's naming rules (only letters, digits, `_`,
`-`, `.`, `+`, `@` and spaces), or a failure to encode a password.
The configuration file contains backend credentials and authorization tokens; rclone's encoding is reversible, not
encryption. For encrypted channels it also contains the decryption password. Keep it offline, to the same standard as the master key.
### Close [#close]
Always call `Close(ctx)` when you are done with a driver. If the backend provides a shutdown hook (rclone's
`Shutdowner`), it stops the backend's background tasks and connection pools. An encrypted driver also removes the inner
backend from rclone's fs cache, and the cache closes the inner backend as it is removed. Do not use a driver after `Close`.
`Close` is a method on `*rclone.Driver`, not part of the `Driver` interface. If you pass drivers around as
`driver.Driver`, keep the concrete pointer as well so you can close it.
### Complete example [#complete-example]
This is `store/rclone/example_test.go` from the repository. It runs with `go test ./store/rclone`, and each example
prints exactly what its `Output` comment lists (the comments in the source are in Chinese): `ExampleSensitiveParams`
queries the sensitive parameters of the webdav backend, and `Example` walks through write, stat, read back, and delete
on rclone's `memory` backend.
```go title="store/rclone/example_test.go"
package rclone_test
import (
"context"
"errors"
"fmt"
"io"
"strings"
"github.com/x-haose/imgyoung/store/driver"
"github.com/x-haose/imgyoung/store/rclone"
)
// ExampleSensitiveParams 查询 webdav 后端的敏感参数:密码(IsPassword)与账号名、令牌(Sensitive)都在其中。
func ExampleSensitiveParams() {
names, err := rclone.SensitiveParams("webdav")
if err != nil {
fmt.Println("sensitive params:", err)
return
}
fmt.Println(strings.Join(names, " "))
// Output:
// bearer_token pass user
}
// Example 用 rclone 的内存后端走一遍驱动的基本操作:写入、查询、读回、删除。
func Example() {
ctx := context.Background()
d, err := rclone.Open(ctx, rclone.Options{Backend: "memory", Path: "bucket"})
if err != nil {
fmt.Println("open:", err)
return
}
defer func() { _ = d.Close(ctx) }()
if err := d.Put(ctx, "hello.txt", strings.NewReader("hello"), 5); err != nil {
fmt.Println("put:", err)
return
}
info, err := d.Stat(ctx, "hello.txt")
if err != nil {
fmt.Println("stat:", err)
return
}
fmt.Println(info.Key, info.Size)
rc, err := d.Open(ctx, "hello.txt")
if err != nil {
fmt.Println("open object:", err)
return
}
data, err := io.ReadAll(rc)
_ = rc.Close()
if err != nil {
fmt.Println("read:", err)
return
}
fmt.Println(string(data))
if err := d.Delete(ctx, "hello.txt"); err != nil {
fmt.Println("delete:", err)
return
}
_, err = d.Stat(ctx, "hello.txt")
fmt.Println(errors.Is(err, driver.ErrNotFound))
// Output:
// hello.txt 5
// hello
// true
}
```
The `memory` and `local` backends are well suited to integration tests: the first keeps data in process memory only,
the second pairs with `t.TempDir()`, and neither needs a cloud account.
## Writing your own driver [#writing-your-own-driver]
Without rclone, implement `driver.Driver` yourself and it works with `store/failover` as is. Key points:
1. **Return `ErrNotFound` for missing objects**: when `Open`, `Stat`, or `Delete` cannot find the object, return an
error that `errors.Is(err, driver.ErrNotFound)` matches. `failover.Open` relies on it to tell "object does not
exist" from "store is broken", and `failover.Probe` treats `ErrNotFound` from `Stat` as "store reachable".
2. **`Put` reads exactly `size` bytes** and returns an error if it gets fewer; do not close `r`. After a failure the
caller may retry with the same key, and the new `Put` should completely replace whatever was left behind.
3. **`List` is recursive**: an empty `dir` means all objects; keys are full paths relative to the root; a missing
directory returns an empty result and nil.
4. **Fill `ModTime` with the real modification time**: callers use it to judge how old an object is. The application's
reconciliation, for example, only reclaims unknown objects whose modification time is older than its grace period.
5. **Be safe for concurrent use and honor `ctx`.** Check that the implementation is complete with a compile-time
assertion: `var _ driver.Driver = (*MemDriver)(nil)`.
Here is an implementation that keeps objects in process memory, ready to use in tests:
```go
package memstore
import (
"bytes"
"context"
"fmt"
"io"
"strings"
"sync"
"time"
"github.com/x-haose/imgyoung/store/driver"
)
// MemDriver is a Driver that keeps objects in process memory.
type MemDriver struct {
mu sync.RWMutex
objects map[string]memObject
}
type memObject struct {
data []byte
modTime time.Time
}
var _ driver.Driver = (*MemDriver)(nil)
func New() *MemDriver {
return &MemDriver{objects: map[string]memObject{}}
}
func (m *MemDriver) Put(ctx context.Context, key string, r io.Reader, size int64) error {
data, err := io.ReadAll(io.LimitReader(r, size))
if err != nil {
return fmt.Errorf("read %q: %w", key, err)
}
if int64(len(data)) != size {
return fmt.Errorf("write %q: want %d bytes, got %d: %w", key, size, len(data), io.ErrUnexpectedEOF)
}
if err := ctx.Err(); err != nil {
return err
}
m.mu.Lock()
defer m.mu.Unlock()
m.objects[key] = memObject{data: data, modTime: time.Now()}
return nil
}
func (m *MemDriver) Open(ctx context.Context, key string) (io.ReadCloser, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
m.mu.RLock()
defer m.mu.RUnlock()
o, ok := m.objects[key]
if !ok {
return nil, fmt.Errorf("object %q: %w", key, driver.ErrNotFound)
}
return io.NopCloser(bytes.NewReader(o.data)), nil
}
func (m *MemDriver) Stat(ctx context.Context, key string) (*driver.ObjectInfo, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
m.mu.RLock()
defer m.mu.RUnlock()
o, ok := m.objects[key]
if !ok {
return nil, fmt.Errorf("object %q: %w", key, driver.ErrNotFound)
}
return &driver.ObjectInfo{Key: key, Size: int64(len(o.data)), ModTime: o.modTime}, nil
}
func (m *MemDriver) Delete(ctx context.Context, key string) error {
if err := ctx.Err(); err != nil {
return err
}
m.mu.Lock()
defer m.mu.Unlock()
if _, ok := m.objects[key]; !ok {
return fmt.Errorf("object %q: %w", key, driver.ErrNotFound)
}
delete(m.objects, key)
return nil
}
func (m *MemDriver) List(ctx context.Context, dir string) ([]driver.ObjectInfo, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
prefix := ""
if dir != "" {
prefix = strings.TrimSuffix(dir, "/") + "/"
}
m.mu.RLock()
defer m.mu.RUnlock()
var out []driver.ObjectInfo
for k, o := range m.objects {
if strings.HasPrefix(k, prefix) {
out = append(out, driver.ObjectInfo{Key: k, Size: int64(len(o.data)), ModTime: o.modTime})
}
}
return out, nil
}
```