# Architecture (/en/docs/architecture)
Single-instance design, layers and dependency direction, the data model, upload and access data flows, the outbox fence, periodic tasks, and why rclone is integrated in-process.
## Single-instance design [#single-instance-design]
ImgYoung is deliberately designed as a single instance: one process, one data directory.
- **The metadata database is one SQLite file** (`imgyoung.db` in the data directory, in WAL mode). Writes queue on a
single connection and reads use a pool; the driver is the pure-Go `modernc.org/sqlite`.
- **The task queue lives in the same database**: asynchronous work runs on River, whose job tables sit in the same
SQLite file as the business tables, so no separate queue service is needed.
- **All other state is local**: the upload concurrency gate, per-user upload rate limits, login rate limiting, and access
activity aggregation live in process memory, and the variant cache lives on local disk. There is no Redis.
- **One static binary**: everything is built with `CGO_ENABLED=0`, and the admin console's frontend assets and the
OpenAPI document are compiled into the binary.
As a result, ImgYoung does not support several processes sharing one data directory, nor horizontal scaling. The path
to high availability is replacing the database with PostgreSQL (River supports PostgreSQL itself, so the task queue
moves along with it) and moving the in-process shared state out at that point; the current code only implements SQLite.
## Layers and dependency direction [#layers]
```text
http ──→ core ──→ infra
│ └────→ store
└─────────────→ infra
```
| Layer | Directory | Responsibility |
|---|---|---|
| store | `store/` | The public storage engine: the `driver` interface, the `rclone` implementation, and the `failover` primitives. See [Using it as a Go library](./library/index.mdx) |
| infra | `internal/infra/` | Infrastructure and outbound adapters: `config` (bootstrap and runtime settings), `db` (schema, queries, migrations, sqlc-generated code), `secret` (master key and encryption), `id` (short IDs), `imaging` (image processing), `cdn`, `urlfetch` (fetching for URL imports), `observability` (metrics), `version` |
| core | `internal/core/` | Business core: `channel` (channels and health checks), `replication` (replication, reconciliation, draining, CDN purge), `upload`, `variants` (on-demand processing), `dedup`, `lifecycle`, `reliability` (backup and restore), `auth`, `audit`, `referer`, `usage`, `alert` |
| http | `internal/http/` | Inbound HTTP: `publicapi` (uploads and image access), `s3api` (the S3 protocol surface), `adminapi` (admin console), `middleware`, `httpx`, `router` |
| Composition root | `cmd/imgyoung/` | `build.go` constructs the object graph by hand in the order infrastructure → services → HTTP, `serve.go` orchestrates startup and shutdown with oklog/run, and `loops.go` holds the periodic loops |
`store` and `infra` are two parallel leaves: `store` depends on no `internal/` package, and `infra` depends on neither
core, store, nor http; `core` depends on both. The dependency direction is enforced by golangci-lint's depguard rules,
and nothing merges while CI is red:
| Rule | Must not import | Why |
|---|---|---|
| `store/**` | Any package under `internal/` | store is a public, reusable storage engine and cannot depend on application code |
| `internal/infra/**` | `internal/core`, `store`, `internal/http` | infra is the infrastructure leaf |
| `internal/core/**` | `internal/http` | The business core knows nothing about transport and can be tested without HTTP |
| `internal/http/**` | `store` | Channel selection and failover belong to core; the transport layer only uses the results core gives it |
Two more conventions:
- **Interfaces are defined by their consumers**: each package declares small interfaces with only the methods it uses
(such as `upload.ChannelSelector` and `publicapi.BlobChannels`), satisfied implicitly by concrete types.
`store/driver.Driver` is the only first-class abstraction.
- **Identity is passed explicitly**: middleware resolves the authenticated principal into the request context, and the
handler takes it out and passes it to core as an argument; core never reads identity from the context, and per-user
isolation is enforced in core.
HTTP handlers and River workers are thin adapters that translate an external trigger into a service call. Periodic
tasks are `Run(ctx, wait)` loops owned by each service and orchestrated by the composition root; there is no separate scheduler layer.
## Data model [#data-model]
At the heart of the data model is the **separation of physical images from upload references**:
```text
images (references) ──→ blobs (physical images) ──→ blob_channels (replicas per channel) ──→ channels
N : 1 1 : N N : 1
```
- **`blobs`** are the bytes actually stored. Every upload creates a new blob; after background deduplication merges
byte-identical blobs, one blob can be referenced by several images. A blob records the `sha256` of the sanitized
bytes, the perceptual hash `phash`, the object key inside channels `storage_key` (of the form
`{short_id}_{first 8 hex chars of sha256}`), MIME type, size, dimensions, the reference count `ref_count`, and the deletion markers.
- **`images`** are the references created by uploads; the public 10-character short ID is the primary key. Owner,
source token, S3 object key alias, expiry time and access activity, the cold-image label, and cleanup exemption all
live on the reference, not on the blob.
- **`blob_channels`** records a blob's replica on a channel and doubles as the state of its replication task:
`pending → uploading → done / failed`, plus the attempt count and the last error.
Keeping the two layers apart puts each concern in its place: merging duplicates just repoints references at the
surviving blob; TTL, cold images, and quotas are computed per reference; replicas, replication, and physical deletion work per blob.
**Reference counting**: every change to `ref_count` is an atomic `UPDATE … SET ref_count = ref_count ± 1`. Deleting a
reference and decrementing the count happen in the same transaction. A blob whose count reaches zero is marked for
deletion with a physical deletion time based on the runtime setting `delete_grace_minutes`. When that time comes, the
cleanup task first removes its variant cache, then deletes the business rows in one transaction while re-checking the
deletion condition (if the blob was referenced again in the meantime, the deletion is abandoned), and finally queues an
object deletion job for every channel holding a replica.
The remaining tables:
| Table | Contents |
|---|---|
| `channels` | Channels: rclone backend name, parameter JSON encrypted with the master key (AES-GCM), priority, health flag and latest check result, whether encrypted, whether direct links are allowed, the being-cleaned-up flag; the creation time `created_at` (which, together with the channel ID, feeds key derivation for encrypted channels and never changes) and the binding time `bound_at` (see [reconciliation GC](#reconcile-gc)) |
| `variants` | Index of the variant cache: one row per blob and processing parameter set; the bytes live on local disk and are evicted LRU by last access |
| `settings` | Runtime settings in a single-row wide table; each column's DEFAULT is the factory value. See [Configuration](./configuration.mdx) |
| `users`, `tokens`, `sessions` | Users and quotas, upload tokens (hashes only), admin sessions (only a hash of the session id plus its bound CSRF token) |
| `idempotency_keys` | Upload idempotency keys, scoped per user |
| `dedup_candidates` | Near-duplicate candidate pairs, the work queue for manual decisions |
| `referer_whitelist`, `audit_log`, `alerts` | The hotlink protection allowlist, audit log of admin actions, and alert records |
| `master_key_fingerprint` | A single row holding a fingerprint derived from the master key (it cannot be reversed into the key); the master key is checked against it at every start, and a missing or mismatched key refuses to start. See [Master key](./configuration.mdx#master-key) |
The schema is defined by declarative SQL under `internal/infra/db/schema/`, one file per table, with a comment on every
table and every column. Versioned migrations are generated from it by Atlas, embedded in the binary, and applied in
order at startup; applied versions are recorded in the `schema_revisions` table, so upgrading across several versions
also applies every migration in between.
Migrations cannot be downgraded: if the database records a migration version this binary does not know (meaning a newer
binary has already migrated it), the process refuses to start, and the error names the latest migration version in the
database and the latest one this binary knows. The only way back is "old binary + a snapshot taken before the
upgrade"; see [Upgrading](./operations/upgrade.mdx).
## Upload data flow [#upload-data-flow]
Data plane uploads, admin console uploads, URL imports, and S3 PutObject differ only in where the bytes come from;
once the bytes are in hand they all go through the same pipeline. The upload path does not deduplicate: every upload
creates a new blob and deduplication happens in the background, which keeps the upload path a straight line with no branches.
**Entry and guards**: authenticate (upload token, session, or S3 signature) and pass two gates: the per-user upload
rate limit (`upload_rate_per_min`) and the global upload concurrency gate (`max_concurrent_uploads`, which returns 429
immediately when full). The image bytes are then read fully into memory; no upload of any size is written to a
temporary file. The per-image limit `max_upload_mb` applies to the image itself: a file exactly at the limit is
accepted, and anything larger returns 413. A multipart request body is additionally capped at "per-image limit + 64 KiB",
and only three parts are read — `file`, `exif_policy`, and `ttl`. `exif_policy` and `ttl` are taken only from form
fields (never from URL query parameters), at most 1 KiB each, or the request gets a 400.
**Idempotency**: if `Idempotency-Key` is present and the same user has already used that key, the existing image is returned.
**Format and pixel budget**: the real format is decided by the magic bytes at the start of the file, and only JPEG,
PNG, GIF, and WebP are accepted. Only the image header is read to get the dimensions, and they are cross-checked
against the sniffed format; if width × height exceeds `max_pixels_mp` the upload is rejected without a full decode.
Decoded memory depends on resolution, not on file size, so this gate has to come before decoding.
**Quota pre-check**: quota is charged in full per reference; exceeding the user's quota returns 403 and nothing is written to channels.
**Sanitizing**: with the EXIF policy `strip`, metadata is removed (JPEG, PNG, WebP; GIF has no standard metadata segment and passes through unchanged).
For JPEG, a minimal Exif segment holding only the orientation is added back, so photos that rely on the orientation
tag to display upright do not end up rotated; GPS, camera model, timestamps, and every other field are still removed,
and nothing is added back when the orientation is the default.
**Hashing**: one full decode computes the perceptual hash (pHash), then SHA-256 is computed over the bytes that will be
stored. Sanitizing happens before hashing, so `sha256` always matches the bytes actually stored in channels.
**Writing to channels**: the object key is `{short_id}_{first 8 hex chars of sha256}`; embedding the short ID is the
last resort for rebuilding from storage if the metadata database is lost entirely. `failover.Write` writes to the
healthy channels in priority order until `min_success` copies succeed, with a 10-minute timeout per write, moving on to
the next channel on failure.
**Recording**: one transaction writes the blob, the reference, a replica row for every active channel (`done` for the
ones written, `pending` for the rest), and the idempotency key.
**Queuing replication**: replication jobs are queued for the `pending` channels and the upload returns. If queuing
fails, it is only logged; reconciliation picks it up.
The whole upload happens in memory without temporary files. Peak memory is roughly "concurrency × max(per-image size
limit, decoded memory at the pixel budget)", so lowering these settings adapts ImgYoung to small machines. If the
channel write succeeds but recording fails, an orphan object is left behind; reconciliation GC reclaims it, and the
channel write is never rolled back.
## Access data flow [#access-data-flow]
`GET` / `HEAD /i/{id}` (and the S3 object key alias `/o/{user_id}/{key}`; the admin console preview
`/api/admin/images/{id}/thumb` uses the same pipeline but is never counted as an access):
1. Hotlink protection middleware allows the request or returns 403 based on the Referer allowlist (admin console previews skip this step).
2. The reference is looked up; if it cannot be found, 404 is returned. A reference past its expiry time is treated as
missing right away, without waiting for the cleanup task to reclaim it, and an access never renews it.
3. Processing parameters are validated: a `w` outside the width presets or an explicit `format` outside the allowed
formats returns 400 (GIF originals are never processed, so both parameters are ignored for them).
4. Access is counted: only GET requests on the public endpoints add to the in-memory activity, flushed to the database
in batches every `activity_flush_seconds`; HEAD requests, requests with invalid parameters, and admin console previews are not counted.
5. On-demand processing kicks in when the image needs resizing or converting: a `w` parameter, an explicit `format`, or
no format with a browser that accepts WebP, WebP allowed by the settings, and an original that is not WebP already.
GIFs are always served as originals. A variant cache hit is returned directly; on a miss the image is processed
within the `process_workers` concurrency limit and cached, with concurrent requests for the same parameters merged
into one processing run. When processing is busy or fails, the original is served instead, with `Cache-Control: no-store`.
A HEAD request fetches or generates the variant the same way; it just returns no body.
6. Otherwise the original is served. Candidate channels are those that are healthy, not being cleaned up, and hold a
`done` replica of the image, in priority order. In `redirect` access mode, if a candidate allows direct links, is not
encrypted, and has a public URL configured, the response is a 302 to that channel, sent with
`Cache-Control: no-store` (the target changes with channel health and configuration, so it must not be cached). A
HEAD request gets the response headers only and no channel object is read; otherwise `failover.Open` proxies the
read, switching to the next channel as soon as one cannot be read.
Caching depends on the endpoint:
| Endpoint | Cache-Control | Why |
|---|---|---|
| `/i/{id}` | `public, max-age=31536000, immutable` | Content behind a short ID never changes; the URL is the version |
| `/o/{user_id}/{key}` | `public, no-cache` | Re-uploading the same key repoints the alias at a new image; caches may store the response but must revalidate with the ETag before each use, getting 304 when unchanged |
On both endpoints originals and variants carry a strong ETag (the content hash for originals, the hash plus processing
parameters for variants), and a request whose `If-None-Match` matches it gets a 304. The 304 carries the `ETag`,
`Cache-Control`, and `Vary` the 200 response would have, so caches can keep using their stored copy. Responses whose
format was negotiated via `Accept` carry `Vary: Accept`; a fallback original is `no-store` and never answered with a 304.
If a database lookup fails, the response is a plain-text 500 that can be retried.
## The outbox fence: business tables are the single source of truth [#outbox]
Asynchronous replication runs on River, but River jobs are only **idempotent triggers**. The single source of truth for
replica state is the business table `blob_channels`:
- A replication job means "make sure blob X has a replica on channel Y". The worker reads the business table first: if
the replica is already `done` it returns; before copying it sets `uploading`; on failure it goes back to `pending` and
River retries; if the last attempt still fails it becomes `failed`; on success it becomes `done`.
- Jobs are deduplicated by (blob, channel), counting only unfinished jobs, so the same replica never appears in the queue twice.
- **Reconciliation** runs at startup and every 10 minutes after that: it queues drain jobs for channels being cleaned
up, seeds missing replica rows for every channel, resets `failed` replicas on healthy channels to `pending`, and then
pages through every `pending` / `uploading` replica with a primary-key cursor and queues it again; replicas turning
`done` during the scan cannot shift later pages and cause rows to be skipped.
So even if River's job tables are lost or corrupted, the queue can be rebuilt completely from the business table; a job
that was running when the process was killed is rescheduled by River once it has been stuck in the running state for
16 minutes. When a channel is created, `pending` replica rows are seeded for every existing blob, and the same
machinery brings them in line, with no separate backfill logic.
The other asynchronous jobs follow the same principle: deleting objects from a channel (`delete_object`), draining a
channel being cleaned up (`drain_channel`: it first deletes the replica objects row by row, so its progress is simply the
remaining replica rows; then deletes the system objects under `_imgyoung/` on that channel, metadata snapshots included,
along with deep check leftovers, leaving objects written by other programs alone; and finally deletes the channel record), and purging CDN caches
(`cdn_purge`). All of them are idempotent and retried by River with backoff: orphans left by an object deletion that
finally fails are reclaimed by reconciliation GC; the first time a drain job cannot delete an object, it writes the
reason into the channel's check result and raises a `drain_failed` alert, then keeps retrying with backoff — once its
attempts run out, reconciliation queues it again, so it never stops (the job's own timeout or a shutdown cancellation
does not count as a failure); the way out is **Remove now** on the channel in the admin console (remote objects are kept). When a CDN purge finally
fails, the edge cache expires once the CDN's own cache lifetime runs out.
## Periodic tasks [#periodic-tasks]
Periodic tasks are loops owned by each service and started and stopped by the composition root. For intervals that come
from runtime settings, a change in the admin console reschedules the current wait immediately instead of waiting for the old interval to finish.
| Task | Interval | What it does |
|---|---|---|
| Health check | `health_check_seconds` (default 60 seconds) | Retry loading channels that previously failed to load, then light checks on all channels in parallel; an unhealthy channel whose light check passes recovers only after a deep check confirms it. See [Multi-store failover](./library/failover.mdx) |
| Alert evaluation | Same as the health check | Unhealthy channels, failed replication, channel usage over the threshold, failed drains, failed backups; can be pushed to a webhook |
| Lifecycle cleanup | `cleanup_sweep_seconds` (default 60 seconds) | Reclaim expired references → physically delete blobs past their grace period → label cold images → reclaim expired idempotency keys → reclaim audit entries past retention |
| Activity flush | `activity_flush_seconds` (default 60 seconds) | Write public access times and counts to the database in batches; variants' last-access times are flushed on the same interval |
| Deduplication | `dedup_scan_seconds` (default 300 seconds) | Merge exact duplicates (same SHA-256) automatically, then scan for pHash near-duplicates and queue them for manual decisions |
| Usage report | `usage_report_seconds` (default 300 seconds) | Compute storage usage per channel and update metrics |
| Metadata database backup | `backup_interval_hours` (default 24 hours) | Take one consistent snapshot with `VACUUM INTO` and write it to `_imgyoung/backup/` on every channel that is healthy and not being cleaned up, each channel keeping its own `backup_keep` copies; if any channel is not written, the round counts as failed (`backup_failed` alert) and the channels that were written keep the snapshot. At startup, a backup runs right away if any healthy channel lacks a snapshot within the backup interval. See [Backup and restore](./operations/backup.mdx) |
| Auth maintenance | 1 minute / 1 hour (fixed) | Write back token last-used times in batches / reclaim expired sessions |
| Replication reconciliation | 10 minutes (fixed) | See the previous section |
| Reconciliation GC | 24 hours (fixed) | List every healthy channel and compare with the replica records in the database in both directions: reclaim orphan objects, and reset phantom replicas to `pending` to be replicated again. Criteria below |
### Reconciliation GC criteria [#reconcile-gc]
What reconciliation GC handles (the system directory `_imgyoung/` and the light check key are excluded from the comparison):
- **Orphans**: an object is reclaimed only if all of these hold — its key has this service's object key format
`{short_id}_{first 8 hex chars of sha256}`; the database has no replica record of it on that channel in any state;
its modification time is not earlier than the channel's binding time; and it is past the 48-hour grace period
(which covers uploads already written to the channel but not yet recorded). Every other object without a record —
files written by other programs, objects in an old bucket attached by a new instance during disaster recovery
before the metadata is restored, or objects written to the channel before a snapshot restore that the restored
database has no record of — is only counted, never reclaimed, and appears as `foreign` in the reconciliation log.
- **Deep check leftovers**: `.imgyoung-probe-` temporary objects left when a deep check fails to delete them are
deleted once past the 48-hour grace period, regardless of when they were written, and are not counted in the
reconciliation's object and byte totals.
- **Phantoms**: replicas recorded as `done` but missing from the channel are reset to `pending` and queued for replication again.
The **binding time** (`channels.bound_at`) marks when the channel record started to correspond to its current storage
location; this service only writes objects there after that moment:
- A new channel's binding time equals its creation time. When a new instance attaches an old bucket, every object
already in it is older and is only counted, never reclaimed.
- Editing a channel in the admin console resets it to the time of the edit if the backend name or any configuration
parameter changes; typing the same password again counts as a change (rclone encodes passwords with a random IV).
Changing only the name, priority, or direct-link switch does not reset it, and neither does a token refresh write-back.
- After a restore from a snapshot, every channel is reset to the time of the first start after the restore: before
swapping the snapshot in, a marker file `imgyoung.db.restored` is written to the data directory; after migrations and
before the master key is loaded, the marker triggers the reset and is deleted on success, with the log line
`换入快照后已把渠道绑定时刻重置为当前时刻,快照之后写入渠道的对象不会被对账回收`. Objects uploaded after the snapshot,
which the restored database has no record of, are therefore never deleted as orphans. The marker is written when the
staged file `/imgyoung.db.restore` is swapped in, and a restore from the admin console stages exactly that
file; a manual restore must also put the snapshot there and then start, not overwrite `imgyoung.db` directly,
otherwise no marker is written and the binding times are not reset.
`created_at` does not change along with the binding time: together with the channel ID it feeds key derivation for encrypted channels.
Reconciliation only trusts this database's replica records: objects another instance writes to the same path have the
right key format but no record here, so once past the grace period they are deleted as orphans.
### Shutdown [#shutdown]
Shutdown (on SIGINT / SIGTERM, or a restart requested after restoring a backup snapshot in the admin console) happens
in order: new requests stop being accepted first, and in-flight requests get at most 6 seconds before the remaining
connections are closed; then the background loops and in-flight replication jobs are cancelled and the replication
subsystem gets at most 2 seconds, while each loop writes out data still held in memory before it exits; finally the
channel drivers and the database are closed. The two stages total 8 seconds, within Docker's default 10-second kill
timeout, and the sample configurations under `deploy/` leave extra headroom (`stop_grace_period: 15s` in compose,
`TimeoutStopSec=15s` for systemd). Cancelled replication jobs keep their state in the business table and are queued
again by reconciliation on the next start.
## Technology choices [#tech-stack]
Every choice serves the same premise: a single instance, a single static binary, and no external services needed to
deploy apart from the storage channels. Exact dependency versions are whatever the repository's `go.mod` says.
### Language and build [#language-and-build]
- **Go**: compiles to a single static binary with a small Docker image; I/O orchestration such as multi-channel writes,
asynchronous replication, health checks, and the task queue is straightforward with goroutines; the ecosystem for
object storage, HTTP, and metrics is mature, and simple syntax and fast builds keep the bar for contributors low. The
deciding factor is that rclone itself is written in Go, and only Go can call its backends directly in-process; see
[Why rclone is integrated in-process](#rclone).
- **`CGO_ENABLED=0` throughout**: without a C toolchain or system libraries, the release archives can be cross-compiled
for Linux, macOS, and Windows on two architectures each, and shipped in a distroless image holding nothing but the
binary. Every dependency below is pure Go or works without cgo. All builds use `-tags nodynamic`: by default the WebP
library tries to load the system libwebp at runtime, and only with that path switched off is the Linux binary free of
the dynamic linker.
### Dependencies [#dependencies]
| Area | Choice | Why |
|---|---|---|
| HTTP routing | chi | Handlers are plain standard-library `http.Handler`s with no private concepts; uploads (streaming request reads) and image proxying (streaming response writes) both work directly with `io.Reader` / `io.Writer`, so a thin router fits best |
| SQLite driver | `modernc.org/sqlite` | A pure-Go implementation; no cgo |
| Database access | sqlc | Queries are written in SQL and compiled into type-safe Go code. No ORM: an image host's queries are all simple, exact lookups, so an ORM's abstraction buys nothing |
| Migrations | Atlas | The declarative schema is the single source of truth; during development Atlas diffs it into versioned migrations, and at runtime its pure-Go executor applies them on startup, so deployments need no migration tool |
| Task queue | River | A persistent task queue with retries, backoff, and concurrency control built in; on its SQLite driver the job tables share a database with the business tables, so there is no extra service. Jobs are only idempotent triggers and state lives in the business tables; see [the outbox fence](#outbox) |
| Periodic tasks | A `Run(ctx, wait)` loop owned by each service | Periodic tasks only need "run every so often", with the interval taken from runtime settings and rescheduled the moment it changes; no cron library |
| Wiring | Assembled by hand in `cmd/imgyoung/build.go` | The set of components is fixed and their dependencies are static, so calling constructors in order is enough: searchable, navigable, with no reflection or runtime container. No DI framework |
| Lifecycle | oklog/run | HTTP, River, and each background loop are actors; when any of them exits, the whole group shuts down in order. Harder to get wrong than hand-rolled `WaitGroup`s and channels |
| Architecture guard | depguard (built into golangci-lint) | The direction of dependencies between layers is a lint rule enforced by CI, not a matter of discipline; see [Layers and dependency direction](#layers) |
| Perceptual hash | goimagehash | Pure Go; takes the decoded `image.Image`, so it plugs straight into the upload's decode step |
| Image codecs | Standard library, `golang.org/x/image`, gen2brain/webp | None needs cgo; gen2brain/webp encodes WebP by running libwebp in a pure-Go WebAssembly runtime |
| Cryptography | Standard-library AES-256-GCM and HKDF; argon2id from `golang.org/x/crypto` | Encrypting channel configuration and credentials, deriving channel keys and S3 secrets, and hashing login passwords respectively. Encryption of channel data itself is left to rclone crypt |
| Rate limiting | `golang.org/x/time/rate` | Upload rate limits and login throttling are both in-process token buckets |
| Logging and metrics | `log/slog`, Prometheus client_golang | The de facto standards in their areas |
| Short IDs | jaevor/go-nanoid | Public short IDs come from a mature implementation; internal primary keys are SQLite auto-increment rowids and never exposed |
The project is released under the MIT license: the same as rclone, with the lowest barrier for contributors, and a
self-hosted tool has no realistic concern about cloud vendors reselling it as a hosted service.
### What is deliberately left out [#what-is-deliberately-left-out]
- **Redis**: it has no place in a single instance. What needs atomicity, such as idempotency keys and reference counts,
relies on SQLite constraints and atomic updates; rate limits are in-process token buckets; the variant cache lives on
local disk. In a single instance the in-process versions are strictly better than Redis: no network round trips, no
serialization, no extra service to run. It only becomes worth reconsidering when moving to multiple instances.
- **Configuration files and configuration libraries**: environment variables only get the process started; all other
runtime settings live in the database's `settings` table and are edited in the admin console, so "where does this
value come from" always has exactly one answer. See [Configuration](./configuration.mdx).
- **ORMs, DI frameworks, cron libraries**: see the table above for why.
### SQLite and versioned migrations [#sqlite-and-versioned-migrations]
SQLite is a single file with no external dependencies, and a consistent snapshot is just one more file, which suits
self-hosting best and fits the single-instance design. Versioned migrations have been there from day one: a
long-running self-hosted instance goes through many version upgrades, and without versioned migrations the schema would
quickly get out of hand. Migrations only go forward; see [Data model](#data-model).
### Two upgrade seams [#two-upgrade-seams]
Only two choices are expected to be replaced; everything else is fixed. Go's thin libraries all follow the shape of the
standard library, so replacing one would be mechanical work, and reserving flexibility for it would buy nothing.
- **Image processing engine**: the single entry point for on-demand processing is `Processor.Process(src, params)` in
`internal/infra/imaging`. The current implementation is pure Go: the standard library decodes JPEG / PNG / GIF,
gen2brain/webp encodes WebP, output is limited to WebP, JPEG, and PNG, and GIFs are stored and served as-is without
processing. AVIF and HEIF are not included: they see little use, and their encoders (libaom and the like) are heavy.
More formats or faster processing (an implementation based on libvips, for example) would mean replacing this processor.
- **Database**: currently SQLite. The schema is defined in declarative SQL, migrations are generated by Atlas, and query
code is generated by sqlc; moving to PostgreSQL means regenerating the query code and migrations for that dialect and
switching River to its PostgreSQL driver.
Each seam currently has a single implementation, and no interface type has been defined for either in advance.
## Why rclone is integrated in-process [#rclone]
The storage layer follows one rule: rclone only stores bytes in some backend, retrieves, inspects, lists, and deletes
them. Primary/backup switching, replication, reference counting, deduplication, TTL, and health decisions are all
ImgYoung's own code, separated from rclone by the `store/driver.Driver` interface.
The integration imports rclone's `fs` and backend packages directly and builds an `fs.Fs` in-process from a parameter
map, instead of calling the rclone command line, librclone's RPC interface, or an rclone config file:
- **Streaming, no temporary files**: uploads write in-memory bytes straight through `fs.Fs.Put`, and reads stream the
object straight into the HTTP response.
- **Configuration stays in ImgYoung's own database**: channel parameters are stored in the `channels` table encrypted
with the master key and decrypted into a map for the backend at runtime. The parameters contain credentials, so they
are never logged and never carried in error chains.
- **One token refresher**: rclone refreshes OAuth tokens itself and hands them to the application through the
parameter write-back hook, which encrypts and stores them immediately; the application runs no refresh loop of its own.
- **One handle for everything**: reads, writes, checks, listing, and deletion all go through the same `fs.Fs`, with
one set of error mappings (missing objects always become `driver.ErrNotFound`).
- **Channel encryption with no extra code**: encrypted channels stack rclone's crypt layer on top of the backend, and
the upload and access paths stay unchanged; crypt is a public format, so with the password the data can be decrypted
by standard rclone on its own. See [Channel encryption](./channels/encryption.mdx).
- **Configuration exportable as-is**: the admin console can export an rclone configuration file for every channel, with
exactly the parameters the driver hands to the backend, plus a crypt remote for encrypted channels. It is generated by
`ConfigFile` in `store/rclone`, which shares its parameter handling with driver construction, so standard rclone can
read and write the channel's data without ImgYoung. See [Driver interface and rclone implementation](./library/driver.mdx#config-file).
- **Tests need no cloud account**: rclone's `local` and `memory` backends let the whole storage path run as integration tests on one machine.
Registering every rclone backend (`backend/all`) means any storage rclone supports can be used as a channel directly,
and rclone upgrades bring new backends with them; the cost is a binary on the order of 100 MB. This relies on rclone's
internal APIs, which make no stability promise, so `go.mod` pins the rclone version and upgrades are done by hand with the full test suite.
The `Driver` interface does not assume rclone exists, which is also the way out if rclone's internal APIs change:
swap in another `Driver` implementation and the business code stays the same. Splitting the interface and the rclone
implementation into separate packages also lets Go projects outside the application reuse this storage engine on its
own; see [Using it as a Go library](./library/index.mdx).