Datamagnet Signals

Sales and recruiting teams want to act the moment a champion changes jobs or a prospect engages with a relevant post, but polling an API yourself means building a scheduler, snapshot storage, diffing, de-duplication, ICP filtering, retries and delivery. The hard parts are operational: a run can need more work than one serverless invocation allows, every upstream call is billed, nobody wants the same person reported twice, and a crash between finding a lead and sending it must not lose the lead.

Control plane. A FastAPI service on AWS App Runner owns job CRUD, metrics and the customer-facing leads read, authenticated with an API key. Jobs live in a scheduler PostgreSQL database with a next-run timestamp. The main Datamagnet API proxies its public /signal endpoints to this service, resolving the submitted profile URLs concurrently under a deadline so one slow lookup cannot hold the whole request.

Dispatch and execution. An EventBridge rule fires a Lambda every 15 minutes. The dispatcher locks due jobs with FOR UPDATE SKIP LOCKED, parks each one past a buffer window, and invokes one asynchronous Lambda per job, so two invocations can never run the same job at once and a worker that finds its job already ran skips it rather than billing it twice. The API can also trigger a baseline run immediately after a signal is created.

Detection. Profile signals fetch each tracked profile through the Datamagnet API, hash the response and compare it to the stored snapshot; only a changed profile reaches the LLM, which writes a plain-language summary and a confidence score. Engagement signals collect the people who liked or commented on matching posts, de-duplicate them against per-signal seen-profile state, filter and score them against the customer's ICP on a 1 to 3 scale, store them, then deliver. A second watcher database holds snapshots, detected changes, leads and de-duplication state, separate from the scheduler's own tables.

Reliability and operations. Every run has a time and call budget: when it runs out the run ends as partial, delivers what it gathered, records which posts and groups it covered, and the next run starts with what was covered least recently. Webhooks are HMAC-signed and retried on a 7-attempt schedule from a delivery queue; leads are stored as undelivered before sending and a daily sweep re-sends any a crash left behind. Run history is kept for 14 days. Terraform provisions Lambda, App Runner, EventBridge, SNS, ECR and Secrets Manager; GitHub Actions builds and ships both images on push; a Next.js dashboard on Vercel shows success rates and failures, and an SNS alarm emails on Lambda errors.