Architecture

Single-instance designCopy Anchor Link

ImgYoung is deliberately designed as a single instance: one process, one data directory.

• The metadata database is one SQLite file (imgyoung.db in the data directory, in WAL mode). Writes queue on a single connection and reads use a pool; the driver is the pure-Go modernc.org/sqlite.
• The task queue lives in the same database: asynchronous work runs on River, whose job tables sit in the same SQLite file as the business tables, so no separate queue service is needed.
• All other state is local: the upload concurrency gate, per-user upload rate limits, login rate limiting, and access activity aggregation live in process memory, and the variant cache lives on local disk. There is no Redis.
• One static binary: everything is built with CGO_ENABLED=0, and the admin console's frontend assets and the OpenAPI document are compiled into the binary.

As a result, ImgYoung does not support several processes sharing one data directory, nor horizontal scaling. The path to high availability is replacing the database with PostgreSQL (River supports PostgreSQL itself, so the task queue moves along with it) and moving the in-process shared state out at that point; the current code only implements SQLite.

Layers and dependency directionCopy Anchor Link

http ──→ core ──→ infra
  │        └────→ store
  └─────────────→ infra

store and infra are two parallel leaves: store depends on no internal/ package, and infra depends on neither core, store, nor http; core depends on both. The dependency direction is enforced by golangci-lint's depguard rules, and nothing merges while CI is red:

Two more conventions:

• Interfaces are defined by their consumers: each package declares small interfaces with only the methods it uses (such as upload.ChannelSelector and publicapi.BlobChannels), satisfied implicitly by concrete types. store/driver.Driver is the only first-class abstraction.
• Identity is passed explicitly: middleware resolves the authenticated principal into the request context, and the handler takes it out and passes it to core as an argument; core never reads identity from the context, and per-user isolation is enforced in core.

HTTP handlers and River workers are thin adapters that translate an external trigger into a service call. Periodic tasks are Run(ctx, wait) loops owned by each service and orchestrated by the composition root; there is no separate scheduler layer.

Data modelCopy Anchor Link

At the heart of the data model is the separation of physical images from upload references:

images (references) ──→ blobs (physical images) ──→ blob_channels (replicas per channel) ──→ channels
        N : 1                      1 : N                                 N : 1

• blobs are the bytes actually stored. Every upload creates a new blob; after background deduplication merges byte-identical blobs, one blob can be referenced by several images. A blob records the sha256 of the sanitized bytes, the perceptual hash phash, the object key inside channels storage_key (of the form {short_id}_{first 8 hex chars of sha256}), MIME type, size, dimensions, the reference count ref_count, and the deletion markers.
• images are the references created by uploads; the public 10-character short ID is the primary key. Owner, source token, S3 object key alias, expiry time and access activity, the cold-image label, and cleanup exemption all live on the reference, not on the blob.
• blob_channels records a blob's replica on a channel and doubles as the state of its replication task: pending → uploading → done / failed, plus the attempt count and the last error.

Keeping the two layers apart puts each concern in its place: merging duplicates just repoints references at the surviving blob; TTL, cold images, and quotas are computed per reference; replicas, replication, and physical deletion work per blob.

Reference counting: every change to ref_count is an atomic UPDATE … SET ref_count = ref_count ± 1. Deleting a reference and decrementing the count happen in the same transaction. A blob whose count reaches zero is marked for deletion with a physical deletion time based on the runtime setting delete_grace_minutes. When that time comes, the cleanup task first removes its variant cache, then deletes the business rows in one transaction while re-checking the deletion condition (if the blob was referenced again in the meantime, the deletion is abandoned), and finally queues an object deletion job for every channel holding a replica.

The remaining tables:

The schema is defined by declarative SQL under internal/infra/db/schema/, one file per table, with a comment on every table and every column. Versioned migrations are generated from it by Atlas, embedded in the binary, and applied in order at startup; applied versions are recorded in the schema_revisions table, so upgrading across several versions also applies every migration in between.

Migrations cannot be downgraded: if the database records a migration version this binary does not know (meaning a newer binary has already migrated it), the process refuses to start, and the error names the latest migration version in the database and the latest one this binary knows. The only way back is "old binary + a snapshot taken before the upgrade"; see Upgrading.

Upload data flowCopy Anchor Link

Data plane uploads, admin console uploads, URL imports, and S3 PutObject differ only in where the bytes come from; once the bytes are in hand they all go through the same pipeline. The upload path does not deduplicate: every upload creates a new blob and deduplication happens in the background, which keeps the upload path a straight line with no branches.

The whole upload happens in memory without temporary files. Peak memory is roughly "concurrency × max(per-image size limit, decoded memory at the pixel budget)", so lowering these settings adapts ImgYoung to small machines. If the channel write succeeds but recording fails, an orphan object is left behind; reconciliation GC reclaims it, and the channel write is never rolled back.

Access data flowCopy Anchor Link

GET / HEAD /i/{id} (and the S3 object key alias /o/{user_id}/{key}; the admin console preview /api/admin/images/{id}/thumb uses the same pipeline but is never counted as an access):

1. Hotlink protection middleware allows the request or returns 403 based on the Referer allowlist (admin console previews skip this step).
2. The reference is looked up; if it cannot be found, 404 is returned. A reference past its expiry time is treated as missing right away, without waiting for the cleanup task to reclaim it, and an access never renews it.
3. Processing parameters are validated: a w outside the width presets or an explicit format outside the allowed formats returns 400 (GIF originals are never processed, so both parameters are ignored for them).
4. Access is counted: only GET requests on the public endpoints add to the in-memory activity, flushed to the database in batches every activity_flush_seconds; HEAD requests, requests with invalid parameters, and admin console previews are not counted.
5. On-demand processing kicks in when the image needs resizing or converting: a w parameter, an explicit format, or no format with a browser that accepts WebP, WebP allowed by the settings, and an original that is not WebP already. GIFs are always served as originals. A variant cache hit is returned directly; on a miss the image is processed within the process_workers concurrency limit and cached, with concurrent requests for the same parameters merged into one processing run. When processing is busy or fails, the original is served instead, with Cache-Control: no-store. A HEAD request fetches or generates the variant the same way; it just returns no body.
6. Otherwise the original is served. Candidate channels are those that are healthy, not being cleaned up, and hold a done replica of the image, in priority order. In redirect access mode, if a candidate allows direct links, is not encrypted, and has a public URL configured, the response is a 302 to that channel, sent with Cache-Control: no-store (the target changes with channel health and configuration, so it must not be cached). A HEAD request gets the response headers only and no channel object is read; otherwise failover.Open proxies the read, switching to the next channel as soon as one cannot be read.

Caching depends on the endpoint:

On both endpoints originals and variants carry a strong ETag (the content hash for originals, the hash plus processing parameters for variants), and a request whose If-None-Match matches it gets a 304. The 304 carries the ETag, Cache-Control, and Vary the 200 response would have, so caches can keep using their stored copy. Responses whose format was negotiated via Accept carry Vary: Accept; a fallback original is no-store and never answered with a 304. If a database lookup fails, the response is a plain-text 500 that can be retried.

The outbox fence: business tables are the single source of truthCopy Anchor Link

Asynchronous replication runs on River, but River jobs are only idempotent triggers. The single source of truth for replica state is the business table blob_channels:

• A replication job means "make sure blob X has a replica on channel Y". The worker reads the business table first: if the replica is already done it returns; before copying it sets uploading; on failure it goes back to pending and River retries; if the last attempt still fails it becomes failed; on success it becomes done.
• Jobs are deduplicated by (blob, channel), counting only unfinished jobs, so the same replica never appears in the queue twice.
• Reconciliation runs at startup and every 10 minutes after that: it queues drain jobs for channels being cleaned up, seeds missing replica rows for every channel, resets failed replicas on healthy channels to pending, and then pages through every pending / uploading replica with a primary-key cursor and queues it again; replicas turning done during the scan cannot shift later pages and cause rows to be skipped.

So even if River's job tables are lost or corrupted, the queue can be rebuilt completely from the business table; a job that was running when the process was killed is rescheduled by River once it has been stuck in the running state for 16 minutes. When a channel is created, pending replica rows are seeded for every existing blob, and the same machinery brings them in line, with no separate backfill logic.

The other asynchronous jobs follow the same principle: deleting objects from a channel (delete_object), draining a channel being cleaned up (drain_channel: it first deletes the replica objects row by row, so its progress is simply the remaining replica rows; then deletes the system objects under _imgyoung/ on that channel, metadata snapshots included, along with deep check leftovers, leaving objects written by other programs alone; and finally deletes the channel record), and purging CDN caches (cdn_purge). All of them are idempotent and retried by River with backoff: orphans left by an object deletion that finally fails are reclaimed by reconciliation GC; the first time a drain job cannot delete an object, it writes the reason into the channel's check result and raises a drain_failed alert, then keeps retrying with backoff — once its attempts run out, reconciliation queues it again, so it never stops (the job's own timeout or a shutdown cancellation does not count as a failure); the way out is Remove now on the channel in the admin console (remote objects are kept). When a CDN purge finally fails, the edge cache expires once the CDN's own cache lifetime runs out.

Periodic tasksCopy Anchor Link

Periodic tasks are loops owned by each service and started and stopped by the composition root. For intervals that come from runtime settings, a change in the admin console reschedules the current wait immediately instead of waiting for the old interval to finish.

Reconciliation GC criteriaCopy Anchor Link

What reconciliation GC handles (the system directory _imgyoung/ and the light check key are excluded from the comparison):

• Orphans: an object is reclaimed only if all of these hold — its key has this service's object key format {short_id}_{first 8 hex chars of sha256}; the database has no replica record of it on that channel in any state; its modification time is not earlier than the channel's binding time; and it is past the 48-hour grace period (which covers uploads already written to the channel but not yet recorded). Every other object without a record — files written by other programs, objects in an old bucket attached by a new instance during disaster recovery before the metadata is restored, or objects written to the channel before a snapshot restore that the restored database has no record of — is only counted, never reclaimed, and appears as foreign in the reconciliation log.
• Deep check leftovers: .imgyoung-probe- temporary objects left when a deep check fails to delete them are deleted once past the 48-hour grace period, regardless of when they were written, and are not counted in the reconciliation's object and byte totals.
• Phantoms: replicas recorded as done but missing from the channel are reset to pending and queued for replication again.

The binding time (channels.bound_at) marks when the channel record started to correspond to its current storage location; this service only writes objects there after that moment:

• A new channel's binding time equals its creation time. When a new instance attaches an old bucket, every object already in it is older and is only counted, never reclaimed.
• Editing a channel in the admin console resets it to the time of the edit if the backend name or any configuration parameter changes; typing the same password again counts as a change (rclone encodes passwords with a random IV). Changing only the name, priority, or direct-link switch does not reset it, and neither does a token refresh write-back.
• After a restore from a snapshot, every channel is reset to the time of the first start after the restore: before swapping the snapshot in, a marker file imgyoung.db.restored is written to the data directory; after migrations and before the master key is loaded, the marker triggers the reset and is deleted on success, with the log line 换入快照后已把渠道绑定时刻重置为当前时刻,快照之后写入渠道的对象不会被对账回收. Objects uploaded after the snapshot, which the restored database has no record of, are therefore never deleted as orphans. The marker is written when the staged file <data dir>/imgyoung.db.restore is swapped in, and a restore from the admin console stages exactly that file; a manual restore must also put the snapshot there and then start, not overwrite imgyoung.db directly, otherwise no marker is written and the binding times are not reset.

created_at does not change along with the binding time: together with the channel ID it feeds key derivation for encrypted channels.

ShutdownCopy Anchor Link

Shutdown (on SIGINT / SIGTERM, or a restart requested after restoring a backup snapshot in the admin console) happens in order: new requests stop being accepted first, and in-flight requests get at most 6 seconds before the remaining connections are closed; then the background loops and in-flight replication jobs are cancelled and the replication subsystem gets at most 2 seconds, while each loop writes out data still held in memory before it exits; finally the channel drivers and the database are closed. The two stages total 8 seconds, within Docker's default 10-second kill timeout, and the sample configurations under deploy/ leave extra headroom (stop_grace_period: 15s in compose, TimeoutStopSec=15s for systemd). Cancelled replication jobs keep their state in the business table and are queued again by reconciliation on the next start.

Technology choicesCopy Anchor Link

Every choice serves the same premise: a single instance, a single static binary, and no external services needed to deploy apart from the storage channels. Exact dependency versions are whatever the repository's go.mod says.

Language and buildCopy Anchor Link

• Go: compiles to a single static binary with a small Docker image; I/O orchestration such as multi-channel writes, asynchronous replication, health checks, and the task queue is straightforward with goroutines; the ecosystem for object storage, HTTP, and metrics is mature, and simple syntax and fast builds keep the bar for contributors low. The deciding factor is that rclone itself is written in Go, and only Go can call its backends directly in-process; see Why rclone is integrated in-process.
• CGO_ENABLED=0 throughout: without a C toolchain or system libraries, the release archives can be cross-compiled for Linux, macOS, and Windows on two architectures each, and shipped in a distroless image holding nothing but the binary. Every dependency below is pure Go or works without cgo. All builds use -tags nodynamic: by default the WebP library tries to load the system libwebp at runtime, and only with that path switched off is the Linux binary free of the dynamic linker.

DependenciesCopy Anchor Link

The project is released under the MIT license: the same as rclone, with the lowest barrier for contributors, and a self-hosted tool has no realistic concern about cloud vendors reselling it as a hosted service.

What is deliberately left outCopy Anchor Link

• Redis: it has no place in a single instance. What needs atomicity, such as idempotency keys and reference counts, relies on SQLite constraints and atomic updates; rate limits are in-process token buckets; the variant cache lives on local disk. In a single instance the in-process versions are strictly better than Redis: no network round trips, no serialization, no extra service to run. It only becomes worth reconsidering when moving to multiple instances.
• Configuration files and configuration libraries: environment variables only get the process started; all other runtime settings live in the database's settings table and are edited in the admin console, so "where does this value come from" always has exactly one answer. See Configuration.
• ORMs, DI frameworks, cron libraries: see the table above for why.

SQLite and versioned migrationsCopy Anchor Link

SQLite is a single file with no external dependencies, and a consistent snapshot is just one more file, which suits self-hosting best and fits the single-instance design. Versioned migrations have been there from day one: a long-running self-hosted instance goes through many version upgrades, and without versioned migrations the schema would quickly get out of hand. Migrations only go forward; see Data model.

Two upgrade seamsCopy Anchor Link

Only two choices are expected to be replaced; everything else is fixed. Go's thin libraries all follow the shape of the standard library, so replacing one would be mechanical work, and reserving flexibility for it would buy nothing.

• Image processing engine: the single entry point for on-demand processing is Processor.Process(src, params) in internal/infra/imaging. The current implementation is pure Go: the standard library decodes JPEG / PNG / GIF, gen2brain/webp encodes WebP, output is limited to WebP, JPEG, and PNG, and GIFs are stored and served as-is without processing. AVIF and HEIF are not included: they see little use, and their encoders (libaom and the like) are heavy. More formats or faster processing (an implementation based on libvips, for example) would mean replacing this processor.
• Database: currently SQLite. The schema is defined in declarative SQL, migrations are generated by Atlas, and query code is generated by sqlc; moving to PostgreSQL means regenerating the query code and migrations for that dialect and switching River to its PostgreSQL driver.

Each seam currently has a single implementation, and no interface type has been defined for either in advance.

Why rclone is integrated in-processCopy Anchor Link

The storage layer follows one rule: rclone only stores bytes in some backend, retrieves, inspects, lists, and deletes them. Primary/backup switching, replication, reference counting, deduplication, TTL, and health decisions are all ImgYoung's own code, separated from rclone by the store/driver.Driver interface.

The integration imports rclone's fs and backend packages directly and builds an fs.Fs in-process from a parameter map, instead of calling the rclone command line, librclone's RPC interface, or an rclone config file:

• Streaming, no temporary files: uploads write in-memory bytes straight through fs.Fs.Put, and reads stream the object straight into the HTTP response.
• Configuration stays in ImgYoung's own database: channel parameters are stored in the channels table encrypted with the master key and decrypted into a map for the backend at runtime. The parameters contain credentials, so they are never logged and never carried in error chains.
• One token refresher: rclone refreshes OAuth tokens itself and hands them to the application through the parameter write-back hook, which encrypts and stores them immediately; the application runs no refresh loop of its own.
• One handle for everything: reads, writes, checks, listing, and deletion all go through the same fs.Fs, with one set of error mappings (missing objects always become driver.ErrNotFound).
• Channel encryption with no extra code: encrypted channels stack rclone's crypt layer on top of the backend, and the upload and access paths stay unchanged; crypt is a public format, so with the password the data can be decrypted by standard rclone on its own. See Channel encryption.
• Configuration exportable as-is: the admin console can export an rclone configuration file for every channel, with exactly the parameters the driver hands to the backend, plus a crypt remote for encrypted channels. It is generated by ConfigFile in store/rclone, which shares its parameter handling with driver construction, so standard rclone can read and write the channel's data without ImgYoung. See Driver interface and rclone implementation.
• Tests need no cloud account: rclone's local and memory backends let the whole storage path run as integration tests on one machine.

Registering every rclone backend (backend/all) means any storage rclone supports can be used as a channel directly, and rclone upgrades bring new backends with them; the cost is a binary on the order of 100 MB. This relies on rclone's internal APIs, which make no stability promise, so go.mod pins the rclone version and upgrades are done by hand with the full test suite.

The Driver interface does not assume rclone exists, which is also the way out if rclone's internal APIs change: swap in another Driver implementation and the business code stays the same. Splitting the interface and the rclone implementation into separate packages also lets Go projects outside the application reuse this storage engine on its own; see Using it as a Go library.