Skip to content
Internal202606 / 12

Blogpipe

Autonomous blog pipeline with quality gates, from topic to published post

Pipeline stages
6 + publish
Job states
13
Quality gates
4
Drafts produced
140+
Vendored skills
15
CLI commands
18

Blogpipe turns a topic string, a source file, or an editorial calendar row into a finished, published blog post without a person in the loop. Each job moves through a fixed set of stages, each run as a scoped Claude Agent SDK session, and cannot reach publish until it passes score gates and a deterministic structural validator. A Streamlit dashboard enqueues work, shows state and cost, and offers a one-click approval gate before publish.

My part. Designed and built the whole system solo: the stage runner, quality gates, PostgreSQL state model, dashboard, and the two-container deployment.

The problem

Before Blogpipe, publishing a post meant a person typing a prompt, watching a terminal stream for several minutes, eyeballing the result, and clicking publish. Nothing acted on quality scores, a failed run left no record of what to resume, headless runs needed blanket permissions, and nothing capped spend. Content for a small company has to ship on a schedule whether or not someone is at a laptop, and every post has to meet the same structural contract (frontmatter, schema markup, images on disk, internal links) or the live site breaks in subtle ways.

How it is built

The pipeline is a state machine executed by a Python runner. A job goes QUEUED, WRITING, ENRICHING, VERIFYING, REVISING (looping back to verify as needed), FINALIZING, READY, and then PUBLISHED after approval; any stage can exit to FAILED or NEEDS_HUMAN. Rewrite jobs skip the write stage, and Search Console directed rewrites of live posts enter at a dedicated OPTIMIZE stage. Every exit path writes a report next to the draft, so a stuck job is reviewable without digging through scratch state.

Each stage is a Claude Agent SDK session that invokes the existing blog skill suite unchanged. Autonomy comes from scoping rather than trust: per-stage tool allowlists, an in-process PreToolUse hook that fences writes to allowlisted directories and blocks dangerous shell patterns, per-stage and per-day cost budgets, and a pacing module that backs off before the account's rolling usage window fills.

Quality is enforced in code. The verify stage produces a content score, an AI citation readiness score, and a fact-check report; a deterministic Python validator checks the frontmatter contract, JSON-LD types, image files, unresolved placeholders, internal links, and answer-first structure. A draft below the bar loops through revise a bounded number of times, then stops for a human.

All state lives in PostgreSQL: jobs, stage runs, events, scores, artifacts, a link registry, and Search Console snapshots. Deployment is one Docker image in two roles, a dashboard and a worker. Because they share no process tree, the dashboard writes run and stop intents to a control table and the worker, which owns the process ids, claims and executes them with SKIP LOCKED semantics. Draft images are mirrored to object storage, and publishing upserts by slug into the headless CMS.

What it taught me

  • Enforcing quality in deterministic code rather than in prompts made the pipeline debuggable: a failed gate names the exact check, and almost-valid model JSON is repaired by a lenient parser instead of invalidating a whole report.
  • Splitting the dashboard and worker into separate containers broke every assumption about process ids, so process control moved to a database control table claimed with SKIP LOCKED.
  • Uncoordinated agent sessions exhausted a shared account quota all at once; a lock-slot concurrency cap plus pacing against the rolling usage window fixed it before the limit rather than after.

Architecture

A Streamlit dashboard and a pipeline worker run as two containers from one image and share a PostgreSQL database. The worker claims jobs and control requests, runs each stage as a scoped agent session, records scores and events, mirrors draft images to object storage, and publishes finished posts to the CMS. Search Console snapshots feed optimization jobs back into the queue, and job outcomes go out over a webhook.

  1. Layer 1

    • DashboardClient

      Streamlit; enqueue, preview, approve

      → PostgreSQL (writes intents) → Search Console (fetches snapshots)

    • Worker dispatcherWorker

      claims jobs, forks stage runs

      → PostgreSQL (claims jobs) → Stage sessions (runs stages) → Object storage (mirrors images) → Headless CMS (publishes) → Webhook (reports outcomes)

  2. Layer 2

    • Search ConsoleExternal

      page and query snapshots

    • Stage sessionsModel

      scoped Claude Agent SDK runs

      → PostgreSQL (scores, events)

    • Object storageStore

      draft image backup

    • Headless CMSExternal

      upsert by slug

    • WebhookExternal

      job outcome alerts

  3. Layer 3

    • PostgreSQLStore

      jobs, events, scores, artifacts

ClientWorkerModelStoreExternal

Stack

Backend

  • Python 3.13
  • Claude Agent SDK
  • Streamlit
  • Node.js (publish script)

Data

  • PostgreSQL
  • Cloudflare R2
  • Sanity CMS
  • Google Search Console API

Infrastructure

  • Docker Compose
  • Coolify
  • Traefik
  • Cloud VM (arm64)

AI

  • Claude models via Agent SDK
  • Gemini image generation over MCP