158 lines
8.7 KiB
Markdown
158 lines
8.7 KiB
Markdown
# Recovery — The Target Architecture
|
|
|
|
This document describes where writeonce is going: a single, self-contained binary that owns its own storage, serves its own content, and pushes updates to connected clients in real-time — with no external database, no cloud pipeline, and no separate API server.
|
|
|
|
## Guiding Principle
|
|
|
|
**Everything in one process.** The database, the server logic, and the client-facing interface all live in a single codebase and ship as a single executable. If you can run the binary, you have the full platform.
|
|
|
|
## Own Database
|
|
|
|
The current PostgreSQL instance is a derived cache — it stores JSONB copies of files that already exist as the source of truth. The recovery architecture eliminates this indirection entirely.
|
|
|
|
### What Changes
|
|
|
|
- **No external database.** No PostgreSQL, no Diesel ORM, no connection pooling, no migrations.
|
|
- **Local file storage.** Markdown files and JSON metadata files are stored in a local directory, just as they are today in `writeonce-articles-s3/`. The file system *is* the database.
|
|
- **Custom storage segments (.seg files).** Research area: segment files that provide efficient read access, indexing, and potentially append-only writes for content. Think of these as a lightweight, purpose-built storage layer — not a general-purpose database engine, but enough to support indexed lookups by `blog-title` and ordered listing by date.
|
|
- **Indexed by blog-title.** The `sys_title` / blog-title field remains the primary key for content retrieval. The embedded storage must support O(1) or O(log n) lookups by this field.
|
|
|
|
### What Stays the Same
|
|
|
|
- Articles are still structured as JSON metadata + Markdown content pairs.
|
|
- The `sys_title`, `published`, `tags`, `author`, and section structure remain the content model.
|
|
- Content is still the source of truth — but now it's read directly from local storage instead of being derived through a sync pipeline.
|
|
|
|
## No AWS Infrastructure
|
|
|
|
The current architecture uses S3 as a file host and Lambda as a sync trigger. In the target architecture, there is nothing to sync *to* — the files are already where they need to be.
|
|
|
|
### What Gets Removed
|
|
|
|
| Current Component | Why It Existed | Why It's No Longer Needed |
|
|
|---|---|---|
|
|
| S3 bucket | Remote file storage | Files live locally alongside the binary |
|
|
| Lambda function (Go) | Watch S3 for changes, call API | No remote store to watch — file changes are local |
|
|
| aws-infra service (Rust) | Bridge to AWS S3/EC2 APIs | No AWS dependency |
|
|
| Pulumi IaC | Manage Lambda + S3 resources | No cloud resources to manage |
|
|
|
|
### What Replaces It
|
|
|
|
The binary watches its own content directory. When a file changes (new article, updated metadata), the embedded database re-indexes and notifies subscribers. The deployment model becomes:
|
|
|
|
```
|
|
1. Place the binary on a server
|
|
2. Point it at a content directory
|
|
3. It serves
|
|
```
|
|
|
|
No credentials, no IAM roles, no SDK configuration.
|
|
|
|
## No Separate API
|
|
|
|
Today, `writeonce-api` is a standalone Actix-web server that mediates between the frontend and the database. In the target architecture, the server logic is embedded in the same process as the database and the content renderer.
|
|
|
|
### What This Means
|
|
|
|
- **No HTTP hop between database and server.** Queries go directly from the request handler to the storage engine in-process. No network serialization, no connection pool, no ORM layer.
|
|
- **Single codebase.** No multi-repo coordination. A new article field is added once — in the content model — and it flows through storage, indexing, and rendering in the same compilation unit.
|
|
- **Single deployment.** One binary, one container, one process. No docker-compose orchestrating API + database + infra services.
|
|
|
|
The binary still exposes HTTP endpoints — it's still a web server. But it's a web server with an embedded database, not a web server that talks to an external one.
|
|
|
|
## Real-Time Subscriptions Without WebSocket
|
|
|
|
The current architecture has no mechanism for pushing content updates to connected clients. The target architecture adds real-time subscriptions, but explicitly without WebSocket.
|
|
|
|
### Why Not WebSocket
|
|
|
|
WebSocket adds connection state management, heartbeat logic, reconnection handling, and protocol upgrade complexity. For a content platform where updates are infrequent (articles are published, not streamed), the overhead isn't justified.
|
|
|
|
### Subscription Model
|
|
|
|
The target is a subscription mechanism where:
|
|
|
|
- A client subscribes to a content query (e.g., "all published articles" or "article with sys_title X")
|
|
- When the underlying data changes, the server pushes the relevant diff to the subscriber
|
|
- No polling from the client side
|
|
|
|
Candidate approaches to research:
|
|
|
|
- **Server-Sent Events (SSE)** — unidirectional push over HTTP. Simple, well-supported, no protocol upgrade. Natural fit for infrequent content updates.
|
|
- **SpacetimeDB-style subscriptions** — clients register queries, the engine tracks which rows match, and only sends diffs when the result set changes. This is the aspirational model.
|
|
- **Long polling** — fallback option. Simple but less efficient than SSE for multiple subscribers.
|
|
|
|
The key constraint: the subscription mechanism must work without requiring clients to maintain persistent bidirectional connections.
|
|
|
|
## Target Architecture
|
|
|
|
```
|
|
content directory
|
|
(JSON + MD files, .seg index)
|
|
|
|
|
| file watch + re-index
|
|
v
|
|
+---------------------------+
|
|
| writeonce binary |
|
|
| |
|
|
| +-------------------+ |
|
|
| | embedded storage | | .seg files, blog-title index
|
|
| | (read/write/index)| |
|
|
| +-------------------+ |
|
|
| | |
|
|
| +-------------------+ |
|
|
| | server logic | | route handlers, content queries
|
|
| | (HTTP endpoints) | |
|
|
| +-------------------+ |
|
|
| | |
|
|
| +-------------------+ |
|
|
| | subscription mgr | | SSE / query-based push
|
|
| | (real-time push) | |
|
|
| +-------------------+ |
|
|
| |
|
|
+---------------------------+
|
|
|
|
|
HTTP / SSE
|
|
|
|
|
v
|
|
+-------------------+
|
|
| frontend app | Angular or successor
|
|
| (browser client) |
|
|
+-------------------+
|
|
```
|
|
|
|
## Single Repository
|
|
|
|
The five current repos collapse into one:
|
|
|
|
```
|
|
writeonce/
|
|
content/ # articles (JSON + MD), images, assets
|
|
storage/ # embedded database engine (.seg files, indexing)
|
|
server/ # HTTP handlers, subscription manager
|
|
frontend/ # client application
|
|
writeonce.toml # configuration (port, content dir, index settings)
|
|
```
|
|
|
|
One repo. One build. One deploy artifact.
|
|
|
|
## What Needs Research
|
|
|
|
| Area | Question | Notes |
|
|
|------|----------|-------|
|
|
| **.seg file format** | What storage format gives efficient indexed reads over JSON+MD content? | Look at LSM trees, append-only logs, SQLite's page format for inspiration |
|
|
| **File watching** | How to efficiently detect content changes on Linux/macOS? | `inotify` on Linux, `kqueue` on macOS, or cross-platform via `notify` crate |
|
|
| **SSE vs alternatives** | Is SSE sufficient for the subscription model, or is something custom needed? | SSE handles the "push diffs to subscribers" case well for low-frequency updates |
|
|
| **Index structure** | What index structure supports `blog-title` lookup + date-ordered listing? | B-tree or hash index for title, sorted set for date ordering |
|
|
| **Language choice** | Continue with Rust for the unified binary? | Rust fits: single binary output, no runtime, strong typing, existing team knowledge |
|
|
| **Frontend coupling** | Should the frontend be embedded in the binary (serve static assets) or remain separate? | Embedding simplifies deployment; separate allows independent frontend iteration |
|
|
|
|
## Migration Path
|
|
|
|
The transition from current to target doesn't have to be all-or-nothing:
|
|
|
|
1. **Phase 1** — Build the embedded storage engine. Read JSON+MD files from a local directory, index by `blog-title`, serve via HTTP. No AWS, no PostgreSQL. This alone replaces `writeonce-api` + `aws-infra` + `lambda-function` + PostgreSQL.
|
|
2. **Phase 2** — Add real-time subscriptions (SSE). Clients subscribe to content queries and receive push updates when files change.
|
|
3. **Phase 3** — Collapse repositories. Move frontend into the unified codebase. Ship as a single binary that serves both API and static assets.
|
|
|
|
Each phase produces a working system. The current architecture can run in parallel until the new one is ready.
|