Blogpipe
Autonomous blog pipeline with quality gates, from topic to published post
- Pipeline stages
- 6 + publish
- Job states
- 13
- Quality gates
- 4
- Drafts produced
- 140+
- Vendored skills
- 15
- CLI commands
- 18
Blogpipe turns a topic string, a source file, or an editorial calendar row into a finished, published blog post without a person in the loop. Each job moves through a fixed set of stages, each run as a scoped Claude Agent SDK session, and cannot reach publish until it passes score gates and a deterministic structural validator. A Streamlit dashboard enqueues work, shows state and cost, and offers a one-click approval gate before publish.
The problem
Before Blogpipe, publishing a post meant a person typing a prompt, watching a terminal stream for several minutes, eyeballing the result, and clicking publish. Nothing acted on quality scores, a failed run left no record of what to resume, headless runs needed blanket permissions, and nothing capped spend. Content for a small company has to ship on a schedule whether or not someone is at a laptop, and every post has to meet the same structural contract (frontmatter, schema markup, images on disk, internal links) or the live site breaks in subtle ways.
How it is built
The pipeline is a state machine executed by a Python runner. A job goes QUEUED, WRITING, ENRICHING, VERIFYING, REVISING (looping back to verify as needed), FINALIZING, READY, and then PUBLISHED after approval; any stage can exit to FAILED or NEEDS_HUMAN. Rewrite jobs skip the write stage, and Search Console directed rewrites of live posts enter at a dedicated OPTIMIZE stage. Every exit path writes a report next to the draft, so a stuck job is reviewable without digging through scratch state.
Each stage is a Claude Agent SDK session that invokes the existing blog skill suite unchanged. Autonomy comes from scoping rather than trust: per-stage tool allowlists, an in-process PreToolUse hook that fences writes to allowlisted directories and blocks dangerous shell patterns, per-stage and per-day cost budgets, and a pacing module that backs off before the account's rolling usage window fills.
Quality is enforced in code. The verify stage produces a content score, an AI citation readiness score, and a fact-check report; a deterministic Python validator checks the frontmatter contract, JSON-LD types, image files, unresolved placeholders, internal links, and answer-first structure. A draft below the bar loops through revise a bounded number of times, then stops for a human.
All state lives in PostgreSQL: jobs, stage runs, events, scores, artifacts, a link registry, and Search Console snapshots. Deployment is one Docker image in two roles, a dashboard and a worker. Because they share no process tree, the dashboard writes run and stop intents to a control table and the worker, which owns the process ids, claims and executes them with SKIP LOCKED semantics. Draft images are mirrored to object storage, and publishing upserts by slug into the headless CMS.
What it taught me
- Enforcing quality in deterministic code rather than in prompts made the pipeline debuggable: a failed gate names the exact check, and almost-valid model JSON is repaired by a lenient parser instead of invalidating a whole report.
- Splitting the dashboard and worker into separate containers broke every assumption about process ids, so process control moved to a database control table claimed with SKIP LOCKED.
- Uncoordinated agent sessions exhausted a shared account quota all at once; a lock-slot concurrency cap plus pacing against the rolling usage window fixed it before the limit rather than after.
Architecture
A Streamlit dashboard and a pipeline worker run as two containers from one image and share a PostgreSQL database. The worker claims jobs and control requests, runs each stage as a scoped agent session, records scores and events, mirrors draft images to object storage, and publishes finished posts to the CMS. Search Console snapshots feed optimization jobs back into the queue, and job outcomes go out over a webhook.
Layer 1
DashboardClient
Streamlit; enqueue, preview, approve
→ PostgreSQL (writes intents) → Search Console (fetches snapshots)
Worker dispatcherWorker
claims jobs, forks stage runs
→ PostgreSQL (claims jobs) → Stage sessions (runs stages) → Object storage (mirrors images) → Headless CMS (publishes) → Webhook (reports outcomes)
Layer 2
Search ConsoleExternal
page and query snapshots
Stage sessionsModel
scoped Claude Agent SDK runs
→ PostgreSQL (scores, events)
Object storageStore
draft image backup
Headless CMSExternal
upsert by slug
WebhookExternal
job outcome alerts
Layer 3
PostgreSQLStore
jobs, events, scores, artifacts
Stack
Backend
- Python 3.13
- Claude Agent SDK
- Streamlit
- Node.js (publish script)
Data
- PostgreSQL
- Cloudflare R2
- Sanity CMS
- Google Search Console API
Infrastructure
- Docker Compose
- Coolify
- Traefik
- Cloud VM (arm64)
AI
- Claude models via Agent SDK
- Gemini image generation over MCP