Async Infra

03 / 08
PythonTemporalPostgreSQLAWSFastAPIpgvectorVertex AI

Date

May 2026

Contribution

Team project

Type

Work project, LLM call-analysis pipeline

Async Infra is an LLM pipeline for call centres. The goal was simple to say and hard to build. Take every recorded call between a customer and an agent, work out what happened on it, and show managers where agents coach well, where they slip on compliance, and what customers actually care about.

The pipeline runs end to end without anyone reading a transcript. A call comes out the other side as a summary, a set of behavioural features, compliance verdicts, scored coaching opportunities, and open-ended themes, all queryable from a dashboard.

The flow at a glance

  1. Call data ingestion

    CRM

    • Client CRM
    • Call metadata (customers, agents, etc.)

    ↓ Call recordings

  2. Transcription

    Transcription

    • Transcribe audio (STT)
    • Diarisation, redaction, translation

    ↓ Transcripts (UAE region)

  3. Dispatch calls

    Dispatcher

    • Seeds calls, agents, customers
    • → Core API

    ↓ One job per call

  4. Queue jobs

    Queue

    • One job per call, waiting for a worker

    ↓ Next job

  5. Process in Temporal

    LLM

    • LLM Orchestrator (Temporal)
    • Binder (pipelines, prompts, mappings as code)

    ↓ Step outputs

  6. Pipeline outputs

    Outputs

    • Pipeline outputs
    • → doc-etl (emergent taxonomy)

    ↓ Loaded

  7. Store results

    Database

    • Tenant Postgres + pgvector

    ↓ Results

  8. API

    Core API

    • Results available via API

    ↓ View in dashboard

  9. Dashboard

    UI

    • Dashboard
    • View and analyse results

The components

Transcription

Turn call recordings into clean, translated text.

Recordings go through speech-to-text with speaker diarisation, PII redaction, and translation of Arabic and Hindi into English. The output is a list of timed segments, each with a speaker label. The automated phone menu gets its own system label, so nothing downstream mistakes the IVR for the agent.

STTDiarisationPII redactionArabic and Hindi to EnglishUAE region

Dispatcher

Pull call data and queue one analysis job per call.

The front door, a FastAPI service. It pulls call metadata from the CRM for a date range and seeds calls, agents and customers through Core API, with a Redis cache for the lookups. Once transcripts land, a sweep records each one against its call and puts one job per call on the orchestrator's SQS queue. Every step is idempotent, so a sweep can be rerun safely after any failure.

FastAPISQSRedis cacheRerun-safe

Binder

Configure every pipeline as code.

Each tenant gets a folder of YAML: connections and storage roots, CRM field mappings, and every analysis pipeline with its steps, prompts, output schemas and load rules. A GitHub Action syncs it to S3 on merge, and services read it at runtime behind a short cache, so a new prompt, mapping or pipeline goes live without redeploying anything.

YAMLGitHub ActionsS3 syncNo redeploy

LLM Orchestrator

Run the analysis on Temporal.

For each call, one entry workflow writes a short summary, then fans out seven child workflows in parallel:

  • Behavioural features
  • Compliance
  • Name usage
  • Consultative profiling
  • Retention effort
  • Objection handling
  • Emergent features

Temporal does the hard parts: retries, durable state, and resetting a single failed workflow without touching anything else.

Temporal Cloud7 parallel workflows81 behaviours8 emergent categories

Pipelines

Chain the steps and write the results.

Each pipeline is a chain of steps. Some are LLM calls with a strict JSON output schema. Some are plain code, like marking the opportunity moments in the transcript before scoring. Some are gates, like skipping the name check on calls of two minutes or less. Every step's output goes to a versioned S3 bucket, and most pipelines end by loading their results through Core API.

JSON schemasCode stepsGatesVersioned S3

Core API

Serve results per tenant, securely.

A multi-tenant Flask API in front of one Postgres database per tenant. Requests carry a Cognito token and a tenant header. Reads are cached in Redis. It deploys to ECS from a version tag.

FlaskPostgres per tenantCognitoRedisECS

doc-etl and taxonomy

Turn loose emergent features into a browsable taxonomy.

Emergent features arrive as tens of thousands of loosely worded instances. doc-etl groups them into definitions using LLM clustering, naming and deduplication, then builds a three-level hierarchy: narrow themes, broad themes and cross-category domains. Every definition is embedded into pgvector for semantic search. Later runs attach new calls to the existing taxonomy instead of rebuilding it.

LLM clusteringThree-level hierarchypgvectorIncremental runs

Dashboard

Show agents, coaching and compliance.

A React front end with a small Node backend that aggregates Core API calls into agent rankings, coaching views, compliance rates and customer pages.

ReactNodeAgent rankingsCompliance rates

My part

I owned the dispatcher, the Binder configuration for the client, and the integration between the dispatcher and the orchestrator. In practice that also made me the person who ran the data: the backfills, the reprocessing runs, and the checks that what landed in the database was right.

A few pieces of that work I am proudest of:

Checking the models against the source

Before tuning any prompt, I measured whether each output was grounded in its transcript: whether quotes appeared verbatim, who actually said them, and whether verdicts matched their own reasoning.

That surfaced problems no single call would show. The biggest was that the prompts expected an IVR speaker label the transcripts never had. The fix went upstream into transcription instead of into a pile of prompt workarounds.

Reprocessing without losing anything

When better transcripts arrived, I rebuilt the client's data from them. That meant clearing old outputs, keeping every call's identity and metadata, dropping calls where no agent ever spoke, and seeding new ones.

I ran it as one database transaction with checks before and after: it refused to commit if any kept call, demo record or definition changed. It ran after a verified snapshot and a dry run.

Backfilling at scale over a slow link

A loader bug had left emergent features without timestamps. I recovered them for about 1.7 million rows across two client databases by mapping segment ids back to their transcripts.

Some transcripts had been replaced since the rows were written, so I added a guard that only wrote a time when the row's evidence still matched the segment it cited. Switching from row-by-row updates to set-based ones took the run from hours to under a minute.

Following failures to their cause

One example: a dashboard tab failed for every agent. The data was fine. The real cause was an API that sent dates out as datetimes but only accepted plain dates back. The fix was two lines in the API.