Skip to content
Internal202611 / 12

Actors Management Portal

One dashboard to sync, build, edit, and monitor Apify actors across accounts

REST endpoints
~100
Dashboard pages
13
Database migrations
12
Agent steps
9
LLM providers
4
Credential storage
Encrypted

A portal for running a fleet of Apify actors spread across several accounts. It links actors that share code into templates, pushes shared files from a master actor to every linked copy, triggers builds in bulk, and lets you edit source in the browser with a diff before saving. It also watches token health, pulls per-account analytics, manages pricing templates, and includes an agent that probes an API, generates an actor, tests it locally, reviews it, and publishes it after approval.

My part. Designed and built the whole portal solo: the FastAPI backend and Celery workers, the credential vault, the template sync engine, the in-browser editor, analytics, and the actor-building agent.

The problem

The same scraper logic often lives in several Apify accounts, and keeping those copies in step by hand is slow and error-prone: shared helper files drift, dependency versions diverge, and a fix landed in one account is forgotten in another. Account tokens expire silently and break automations, and there is no single view of whether every actor is healthy. Managing many accounts needs one place to see status, one way to push a change everywhere, and a safe store for credentials.

How it is built

The backend is FastAPI with async SQLAlchemy on PostgreSQL. Account credentials are encrypted at rest with Fernet and never leave the server; the frontend only sees account labels and health. A health check validates tokens on load and surfaces expired ones with a guided re-authentication flow.

Templates group actors across accounts. A template declares shared file patterns; a sync picks a source actor and pushes matching files to every linked actor, optionally regenerating a per-account README. Auto-detect proposes template mappings by matching actor names across accounts. Build-all triggers builds for the whole group, and bulk maintenance toggles run as Celery tasks with rate limits and retries.

An in-browser editor fetches an actor's source, marks unsaved changes, shows a diff, and saves straight back to Apify. Analytics pulls per-account actor statistics with a cache whose TTL depends on whether the month is still open. Pricing templates define pay-per-event tiers that can be attached to actors.

The actor-building agent runs probe, generate, validate, local test with a bounded fix loop, code review, and UX review, then pauses at an approval gate before publishing and running a remote test. Every step appends to a job log that the UI polls. The LLM layer is switchable across four providers, and image generation produces README covers.

What it taught me

  • A master-actor-to-linked-actors sync keyed on file patterns replaced dozens of per-account edits with one action and a visible diff.
  • Expired tokens fail quietly; checking every token on page load and showing a one-click re-auth path turned silent outages into a badge.
  • Model responses truncated mid-JSON until output limits and timeouts were raised per provider; where a code snippet already states the URL and params, parse it directly instead of asking the model to infer it.

Architecture

A Next.js dashboard calls a FastAPI backend, which holds encrypted account credentials in PostgreSQL and talks to the Apify API on behalf of each account. Bulk operations run through Redis-backed Celery workers. The actor-building agent uses a provider-switchable LLM layer and a local runner to test generated actors before pushing them to Apify.

  1. Layer 1

    • DashboardClient

      Next.js; templates, editor, agent

      → FastAPI backend (REST)

  2. Layer 2

    • FastAPI backendService

      accounts, sync, builds, analytics

      → PostgreSQL (read, write) → Credential vault (decrypt per call) → Redis (enqueue bulk ops) → Apify accounts (sync, edit, analytics) → LLM providers (generate, review) → Local actor runner (local test)

  3. Layer 3

    • Credential vaultService

      Fernet-encrypted tokens

    • PostgreSQLStore

      accounts, actors, templates, jobs

    • RedisQueue

      Celery broker

      → Celery worker (dispatch)

    • Local actor runnerWorker

      runs generated actors for tests

    • LLM providersModel

      four vendors, switchable

  4. Layer 4

    • Celery workerWorker

      bulk maintenance, rate-limited

      → Apify accounts (toggle, build)

  5. Layer 5

    • Apify accountsExternal

      many accounts via API

ClientServiceStoreQueueWorkerModelExternal

Stack

Backend

  • Python 3.12
  • FastAPI
  • SQLAlchemy (async)
  • Alembic
  • Celery
  • Playwright

Data

  • PostgreSQL
  • Redis

Frontend

  • Next.js 16
  • React 19
  • Tailwind CSS v4
  • Recharts

AI

  • OpenAI, Anthropic, Google, Cloudflare Workers AI
  • Gemini image generation

Infrastructure

  • Docker
  • Render blueprint
  • Apify API