Overview
Panayam runs as two Cloud Run services (an API and a LiveKit agent worker) backed by Cloud SQL Postgres and Google Cloud Storage, with the AI work split across four external providers — Anthropic for scoring + interview turns, ElevenLabs for voice synthesis, Deepgram for English speech recognition, and Google for Tagalog speech recognition. Real-time audio/video runs through LiveKit Cloud.
Recruiters interact via the admin SPA at /admin/.
Candidates interact via the public join page at /web/
and the self-scheduling page. Maria (the AI interviewer) is a
persona implemented inside the agent worker; she joins LiveKit
rooms as a participant when dispatched.
Six safety monitors (added in v0.4.0) watch every call in real time for connectivity issues, silence, abuse, multi-speaker breaches, technical failures, and overall call health.
System architecture
Services, data stores, and external dependencies. Solid arrows are HTTP/REST; dashed arrows are async / event-driven.
flowchart TB
classDef ours fill:#DB0011,stroke:#fff,stroke-width:1px,color:#fff
classDef data fill:#444,stroke:#888,color:#fff
classDef ext fill:#1f3a5f,stroke:#5a8ac8,color:#fff
classDef client fill:#2a2a2a,stroke:#666,color:#eee
subgraph Clients
Admin["Admin SPA
admin/*.html"]:::client Cand["Candidate Web
web/index.html"]:::client SelfSched["Self-Schedule
web/schedule.html"]:::client end subgraph "Google Cloud Run" API["Panayam API
FastAPI"]:::ours Agent["Panayam Agent Worker
LiveKit Worker (Python)"]:::ours end subgraph "Data Plane" DB[("Cloud SQL
PostgreSQL")]:::data GCS[("GCS Bucket
recordings + frames")]:::data SM[("Secret Manager
API keys + DB url")]:::data end subgraph "External Providers" LK["LiveKit Cloud
SFU + Egress"]:::ext Anth["Anthropic Claude
Haiku (interview) + Sonnet (scoring)"]:::ext EL["ElevenLabs
TTS (en + fil)"]:::ext DG["Deepgram
STT (English)"]:::ext GSTT["Google Cloud
STT chirp_2 (Tagalog)"]:::ext SMTP["Gmail SMTP
invitation emails"]:::ext end Admin -->|REST| API Cand -->|REST| API SelfSched -->|REST| API Cand -.WebRTC.-> LK API --> DB API --> SM API -.dispatches.-> Agent Agent --> LK Agent -->|interview turns| Anth API -->|scoring| Anth Agent --> EL Agent --> DG Agent --> GSTT Agent --> DB LK -.recording.-> GCS API --> SMTP
admin/*.html"]:::client Cand["Candidate Web
web/index.html"]:::client SelfSched["Self-Schedule
web/schedule.html"]:::client end subgraph "Google Cloud Run" API["Panayam API
FastAPI"]:::ours Agent["Panayam Agent Worker
LiveKit Worker (Python)"]:::ours end subgraph "Data Plane" DB[("Cloud SQL
PostgreSQL")]:::data GCS[("GCS Bucket
recordings + frames")]:::data SM[("Secret Manager
API keys + DB url")]:::data end subgraph "External Providers" LK["LiveKit Cloud
SFU + Egress"]:::ext Anth["Anthropic Claude
Haiku (interview) + Sonnet (scoring)"]:::ext EL["ElevenLabs
TTS (en + fil)"]:::ext DG["Deepgram
STT (English)"]:::ext GSTT["Google Cloud
STT chirp_2 (Tagalog)"]:::ext SMTP["Gmail SMTP
invitation emails"]:::ext end Admin -->|REST| API Cand -->|REST| API SelfSched -->|REST| API Cand -.WebRTC.-> LK API --> DB API --> SM API -.dispatches.-> Agent Agent --> LK Agent -->|interview turns| Anth API -->|scoring| Anth Agent --> EL Agent --> DG Agent --> GSTT Agent --> DB LK -.recording.-> GCS API --> SMTP
Interview flow (with safety monitors)
What happens from "candidate clicks Join" to "scorecard generated". All six safety monitors are shown.
sequenceDiagram
autonumber
actor Cand as Candidate
participant Web as web/index.html
participant API as Panayam API
participant LK as LiveKit Cloud
participant Agent as Agent Worker (Maria)
participant Mon as Safety Monitors
participant Anth as Anthropic
participant DB as Cloud SQL
Note over Cand,Web: Candidate ticks AI consent + verifies devices
Cand->>Web: clicks Join
Web->>API: GET /api/interviews/{id}/join-token
API->>DB: load interview metadata
API-->>Web: token + livekit_url + bot_present
Cand->>LK: WebRTC connect with signed token
LK->>Agent: participant_connected
Agent->>Mon: start ConnectivityMonitor (slice 1)
Agent->>Mon: start SilenceMonitor (slice 2)
Agent->>LK: session.generate_reply (Maria opens)
loop Each conversational turn
Cand->>Agent: speech via Deepgram or Google STT
Agent->>Anth: Claude Haiku (interviewer reply)
Agent->>LK: ElevenLabs TTS via session.say
LK-->>Cand: Maria speaks
par Background classification
Agent->>Mon: classify abuse (slice 3)
Agent->>Mon: classify multi-speaker (slice 4)
end
opt Safety policy triggered
Mon->>LK: warning announcement
Mon->>Agent: set end_reason (e.g. abuse_high)
end
end
Note over Agent,Mon: On unrecoverable STT/TTS/LLM error
slice 5 sets technical_failure Note over Agent,Mon: Slice 6 writes health_summary on shutdown Agent->>DB: persist transcript + health_summary alt end_reason normal API->>Anth: Claude Sonnet scoring pipeline API->>DB: scores + ranking else end_reason reschedule API->>API: send reschedule email else end_reason abuse / integrity API->>API: log only (no scoring, no reschedule) end
slice 5 sets technical_failure Note over Agent,Mon: Slice 6 writes health_summary on shutdown Agent->>DB: persist transcript + health_summary alt end_reason normal API->>Anth: Claude Sonnet scoring pipeline API->>DB: scores + ranking else end_reason reschedule API->>API: send reschedule email else end_reason abuse / integrity API->>API: log only (no scoring, no reschedule) end
Data model
Core tables and their relationships. Excludes utility tables (alembic_version, feedback).
erDiagram
Role ||--o{ Candidate : has
Role ||--o{ Interview : has
Role ||--o| RoleAnalyticsCache : cached_for
Candidate ||--o{ Interview : has
Interview ||--o{ Score : has
Interview ||--o| Ranking : has
Role {
UUID id PK
string title
string department
text jd_raw
jsonb jd_parsed
jsonb hm_notes
string question_bank_id
string rubric_id
bool is_mass_position
datetime created_at
}
Candidate {
UUID id PK
string name
string email
string phone
UUID role_id FK
text resume_url
jsonb resume_parsed
jsonb resume_match
jsonb interview_questions
string status
datetime created_at
datetime updated_at
}
Interview {
UUID id PK
UUID candidate_id FK
UUID role_id FK
string livekit_room_name
string livekit_egress_id
datetime scheduled_at
string status
datetime started_at
datetime ended_at
int duration_seconds
text recording_url
jsonb transcript
jsonb cost_breakdown
jsonb health_summary
datetime created_at
}
Score {
UUID id PK
UUID interview_id FK
string dimension
decimal score
text assessment
jsonb evidence
jsonb red_flags
}
Ranking {
UUID id PK
UUID role_id FK
UUID candidate_id FK
UUID interview_id FK
decimal composite_score
string recommendation
text executive_summary
int rank_position
}
RoleAnalyticsCache {
UUID role_id PK
datetime generated_at
int completed_interviews_count
jsonb payload
}
Process maps by hiring type
Status: DRAFT for review.
Three distinct hiring journeys today: Non-mass (single-owner, resume-driven), Mass — invited (bulk SMS / WhatsApp invite, lightweight resume optional), Mass — walk-in (office kiosk, no resume). The diagrams below are the target operating model — implementation may lag in places (e.g. the wizard banner currently shows JD/resume steps for walk-in roles; fix shipped 2026-05-31 on dev).
1. Non-mass position (resume-driven, single recruiter owner)
Classic funnel: JD parsed → HM alignment → candidates added with resume → resume scored against JD → question bank per candidate → AI interview → scorecard. One owner_user_id on the role.
flowchart TD
A[Recruiter creates role
process_type=standard
is_mass_position=false] --> B[Upload + parse JD
Claude Sonnet 4.6] B --> C{JD ambiguous?
Claude decides} C -->|Yes| C1[HM clarifying questions
1-10 via email link] C -->|No| D C1 --> D D[Question bank + rubric ready] D --> S1[Recruiter sign-off
POST /signoff/recruiter] S1 --> S2[HM sign-off via email link
POST /hm/signoff/approve] S2 --> E[Add candidates: name + email + CV] E --> F[Resume parsed
Claude Sonnet 4.6] F --> G[Resume scored vs JD
match strength] G --> H[Per-candidate question bank
JD + resume + HM rubric] H --> I[Shortlist email sent
candidate self-schedules] I --> J[AI Interview — English stack
Deepgram + Claude Haiku + ElevenLabs
gated by fully_signed] J --> K[Scorecard generated
visual + transcript + cost] K --> L[Ranking on Candidates tab] classDef sign fill:#1a7f37,stroke:#6cc787,color:#fff; class S1,S2 sign;
process_type=standard
is_mass_position=false] --> B[Upload + parse JD
Claude Sonnet 4.6] B --> C{JD ambiguous?
Claude decides} C -->|Yes| C1[HM clarifying questions
1-10 via email link] C -->|No| D C1 --> D D[Question bank + rubric ready] D --> S1[Recruiter sign-off
POST /signoff/recruiter] S1 --> S2[HM sign-off via email link
POST /hm/signoff/approve] S2 --> E[Add candidates: name + email + CV] E --> F[Resume parsed
Claude Sonnet 4.6] F --> G[Resume scored vs JD
match strength] G --> H[Per-candidate question bank
JD + resume + HM rubric] H --> I[Shortlist email sent
candidate self-schedules] I --> J[AI Interview — English stack
Deepgram + Claude Haiku + ElevenLabs
gated by fully_signed] J --> K[Scorecard generated
visual + transcript + cost] K --> L[Ranking on Candidates tab] classDef sign fill:#1a7f37,stroke:#6cc787,color:#fff; class S1,S2 sign;
2. Mass position — invited (bulk recruitment, may or may not have CV)
Same online journey as non-mass but: Tagalog interview stack, resume is optional (lots of candidates won't have one), multiple recruiter owners via role_collaborators. Candidates invited via bulk channel (SMS / WhatsApp / Darwinbox feed).
flowchart TD
A[Recruiter creates role
process_type=standard
is_mass_position=true] --> B[Upload + parse JD
Claude Sonnet 4.6] B --> C[Optional HM clarifying questions
only if JD ambiguous] C --> D[Author Custom Question bank
uniform per role, not per candidate] D --> S1[Recruiter sign-off] S1 --> S2[HM sign-off via email link] S2 --> E[Bulk import candidates
name + phone + optional CV] classDef sign fill:#1a7f37,stroke:#6cc787,color:#fff; class S1,S2 sign; E --> F{Has CV?} F -->|Yes| G[Resume parsed + scored vs JD] F -->|No| H[Skip resume scoring
use only question bank] G --> I[Bulk SMS / WhatsApp invite
self-schedule link] H --> I I --> J[AI Interview — Tagalog stack
Google STT chirp_2 + Gemini 2.5 Flash + ElevenLabs fil-PH] J --> K[Scorecard generated
scored against custom question rubric only when no CV] K --> L[Ranking on Candidates tab
shared across all role collaborators]
process_type=standard
is_mass_position=true] --> B[Upload + parse JD
Claude Sonnet 4.6] B --> C[Optional HM clarifying questions
only if JD ambiguous] C --> D[Author Custom Question bank
uniform per role, not per candidate] D --> S1[Recruiter sign-off] S1 --> S2[HM sign-off via email link] S2 --> E[Bulk import candidates
name + phone + optional CV] classDef sign fill:#1a7f37,stroke:#6cc787,color:#fff; class S1,S2 sign; E --> F{Has CV?} F -->|Yes| G[Resume parsed + scored vs JD] F -->|No| H[Skip resume scoring
use only question bank] G --> I[Bulk SMS / WhatsApp invite
self-schedule link] H --> I I --> J[AI Interview — Tagalog stack
Google STT chirp_2 + Gemini 2.5 Flash + ElevenLabs fil-PH] J --> K[Scorecard generated
scored against custom question rubric only when no CV] K --> L[Ranking on Candidates tab
shared across all role collaborators]
3. Mass position — walk-in (office kiosk, no resume ever)
Candidate physically walks into a HR office. No JD parse, no HM alignment, no resume. Recruiter at the kiosk registers them with name + a unique ID (govt ID, employee ref, or generated), and Maria interviews them on a fixed time cap (default 25 min) against the role's pre-authored question bank.
flowchart TD
A[Recruiter creates role
process_type=walk_in
is_mass_position=true] --> B[Author Custom Question bank
+ optional max_interview_duration_minutes] B --> S1[Recruiter sign-off] S1 --> S2[HM sign-off via email link
recruiter can override if HM unavailable] S2 --> C[Recruiter opens /admin/walk-in.html
kiosk mode at the office] classDef sign fill:#1a7f37,stroke:#6cc787,color:#fff; class S1,S2 sign; C --> D[Candidate walks in
shows ID] D --> E[Recruiter enters: name + external_id
+ optional phone] E --> F[POST /api/walk-in/start
creates candidate + interview + LiveKit room] F --> G[Kiosk hands candidate the join link
tablet / laptop opens it] G --> H[AI Interview — Tagalog stack
hard time-cap from role.max_interview_duration_minutes] H --> I[Scorecard generated
scored only against custom question bank rubric] I --> J[Recruiter sees ranking immediately
candidate can leave / sit for next steps] classDef key fill:#5e2a8a,stroke:#a464d8,color:#fff; class A,F,H key;
process_type=walk_in
is_mass_position=true] --> B[Author Custom Question bank
+ optional max_interview_duration_minutes] B --> S1[Recruiter sign-off] S1 --> S2[HM sign-off via email link
recruiter can override if HM unavailable] S2 --> C[Recruiter opens /admin/walk-in.html
kiosk mode at the office] classDef sign fill:#1a7f37,stroke:#6cc787,color:#fff; class S1,S2 sign; C --> D[Candidate walks in
shows ID] D --> E[Recruiter enters: name + external_id
+ optional phone] E --> F[POST /api/walk-in/start
creates candidate + interview + LiveKit room] F --> G[Kiosk hands candidate the join link
tablet / laptop opens it] G --> H[AI Interview — Tagalog stack
hard time-cap from role.max_interview_duration_minutes] H --> I[Scorecard generated
scored only against custom question bank rubric] I --> J[Recruiter sees ranking immediately
candidate can leave / sit for next steps] classDef key fill:#5e2a8a,stroke:#a464d8,color:#fff; class A,F,H key;
SIPOC — explicit ownership per step
SIPOC = Supplier · Inputs · Process · Outputs · Customers. The flowcharts above show the sequence; the tables below answer "who triggers what" for each step in each hiring type. Supplier is the actor who initiates the step (recruiter, HM, candidate, system); Customer is the downstream party who consumes that step's output.
SIPOC — Non-mass position
| Step | Supplier (triggers) | Inputs | Process | Outputs | Customer |
|---|---|---|---|---|---|
| 1. Create role | Recruiter | Title, dept, process_type, owner_user_id | POST /api/roles inserts the role row, scoped to owner | roles row, status=draft | Recruiter (their workspace) |
| 2. Parse JD | Recruiter | JD text (paste) | Claude Sonnet 4.6 extracts competencies, experience range, must-haves | roles.jd_parsed JSONB | HM (next step), AI scoring (rubric) |
| 3. HM clarifying questions (only if JD ambiguous) | Claude decides; Recruiter → HM if needed | Parsed JD, recruiter sends tokenized email link | HM answers async on public form; 0-10 questions; HM never tasked unless Claude flags ambiguity | roles.hm_notes | Question bank generation |
| 3a. Recruiter sign-off | Recruiter | Generated question bank + rubric | POST /signoff/recruiter — captures snapshot, stamps recruiter_signed_at; may override "proceed without HM" with mandatory comment | roles.recruiter_signed_at, signoff_snapshot | HM sign-off step / dispatcher gate |
| 3b. HM sign-off | HM via email link | Recruiter-approved snapshot | HM reviews on public page; can edit (resets recruiter sign-off) or approve; approval stamps hm_signed_at | roles.hm_signed_at, audit log | Dispatcher gate (Maria can now run interviews) |
| 4. Add candidate | Recruiter | Name, email, optional CV | POST /api/candidates inserts row; resume upload triggers parse | candidates row, candidates.resume_parsed | Resume scoring (next step) |
| 5. Resume-vs-JD match | Recruiter (clicks "Score") | Parsed JD + parsed resume | Claude Sonnet compares; returns 0-100 match + recommendation | candidates.resume_match JSONB | Recruiter (decides who advances) |
| 6. Question bank generation | Recruiter (clicks "Generate") | JD + HM rubric + resume_parsed | Claude Sonnet builds per-candidate question list (~8-12 Qs) | candidates.interview_questions JSONB | Maria (agent worker reads at session start) |
| 7. Shortlist email | Recruiter (clicks "Send invite") | Candidate email + role title | SMTP send with self-schedule link | Email delivered, candidates.status = invited | Candidate (picks slot) |
| 8. AI interview | Candidate (joins LiveKit room) | Question bank, JD, candidate consent | Maria interviews in English (Deepgram + Claude Haiku + ElevenLabs); safety monitors run | Recording, transcript, interviews.health_summary | Scoring pipeline |
| 9. Scoring + scorecard | System (on session shutdown) | Transcript + visual frames + JD rubric | Claude Sonnet scores each dimension + executive summary; visual presence scored separately | rankings row, scorecard JSON | Recruiter (Candidates tab), HM (shared link) |
SIPOC — Mass position (invited)
| Step | Supplier (triggers) | Inputs | Process | Outputs | Customer |
|---|---|---|---|---|---|
| 1. Create mass role | Recruiter (lead) | Title, is_mass_position=true, multiple owners via role_collaborators | Role inserted; visibility shared across all collaborator recruiters | roles row + collaborator rows | All collaborator recruiters |
| 2. Parse JD (lighter) | Recruiter | JD text | Same as non-mass; HM alignment is usually skipped or quick | roles.jd_parsed | Question bank, recruiter team |
| 3. Author Custom Questions | Recruiter | Recruiter writes 5-10 uniform questions for the role | POST /api/roles/{id}/custom-questions; same set used for every candidate | roles.custom_questions JSONB | Maria (per-candidate question bank == custom questions) |
| 4. Bulk import candidates | Recruiter | CSV / Darwinbox feed (name + phone + optional CV) | POST /api/candidates per row; resume parse only if CV present | N candidate rows | Bulk invite step |
| 5. Bulk SMS / WhatsApp invite | Recruiter | Candidate phones + self-schedule URL | External SMS gateway (not in tool today — recruiter triggers manually) | Invite sent | Candidate |
| 6. AI interview (Tagalog stack) | Candidate (joins) | Custom questions | Maria interviews in Filipino (Google chirp_2 STT + Gemini 2.5 Flash + ElevenLabs fil-PH) | Recording, transcript | Scoring |
| 7. Scoring (question-bank rubric) | System | Transcript + custom question rubric (no JD match weight) | Claude Sonnet scores against the role's custom questions only | Scorecard | All role collaborators (shared visibility) |
SIPOC — Mass position (walk-in kiosk)
| Step | Supplier (triggers) | Inputs | Process | Outputs | Customer |
|---|---|---|---|---|---|
| 1. Create walk-in role | Recruiter (lead) | Title, process_type=walk_in, max_interview_duration_minutes | Role inserted; wizard branch skips JD/HM/match | roles row with walk-in flag | All collaborator recruiters |
| 2. Author Custom Questions | Recruiter | Question list | Same as mass-invited step 3 | roles.custom_questions | Maria |
| 3. Open kiosk page | Recruiter (at office) | Browser tab on /admin/walk-in.html?role_id=… | Page lists today's queue + new-candidate form | Kiosk session | Recruiter (running the kiosk) |
| 4. Register walk-in | Recruiter (at kiosk) | Candidate name + external_id (govt ID / employee ref) + optional phone | POST /api/walk-in/start atomically: create candidate, create interview, create LiveKit room, dispatch agent | candidates+interviews+room provisioned | Kiosk hands candidate the join URL |
| 5. Candidate joins | Candidate (at tablet) | Join URL on a kiosk tablet | Standard /web/ join flow | Candidate in LiveKit room | Maria (agent) |
| 6. AI interview (time-capped) | Candidate | Custom questions + role's max_interview_duration_minutes | Tagalog stack; hard time-cap enforced by silence/connectivity + a duration timer | Recording, transcript | Scoring |
| 7. Scoring + immediate ranking | System (on shutdown) | Transcript + custom question rubric | Same as mass-invited step 7 | Scorecard visible on kiosk page within ~30 s | Recruiter (decides next step in person — second-round, walk-away, etc.) |
Decision matrix — which type fits which hire?
| Hire scenario | process_type | is_mass_position | Owner model | JD parse? | Resume? | Time cap |
|---|---|---|---|---|---|---|
| Single high-skill hire (PM, engineer) | standard | false | 1 owner | Yes | Required | ~35 min |
| Bulk recruitment with CV (sales, ops) | standard | true | Multiple owners | Yes (lighter) | Optional | ~25-30 min |
| Walk-in kiosk (field collections, retail) | walk_in | true | Multiple owners | No | None | Configurable, default 25 min |
To implement (post-review): wizard banner already auto-skips JD/HM/match for walk_in roles (shipped on dev). Still missing: a "Resume optional" toggle on standard + mass roles so candidates can be added without CV, and the per-role question_bank_only scoring path that ignores resume_match weights for the latter two types.
Two-stage sign-off on the interview question bank
Before Maria interviews any candidate for a role, BOTH the recruiter and the hiring manager must approve the question bank + rubric. Enforced server-side at the agent dispatcher — any path that creates an Interview row (recruiter scheduling, candidate self-schedule, walk-in kiosk) checks the gate.
sequenceDiagram
actor R as Recruiter
participant Tool as Panayam
participant Claude
actor HM
participant Disp as Dispatcher
actor C as Candidate
R->>Tool: Parse JD
Tool->>Claude: Are there ambiguities
alt Yes ambiguity
Claude-->>Tool: 1 to 10 clarifying questions
R->>HM: Email link to answer clarifying questions
HM-->>Tool: Answers become hm_notes
else No ambiguity
Claude-->>Tool: zero questions
Note over Tool: HM is NOT contacted for clarification
end
Tool->>Tool: Generate question bank and rubric
R->>Tool: Sign off as recruiter
Note over Tool: state becomes recruiter_signed and snapshot captured
R->>HM: Send HM sign-off email
HM->>Tool: Open signoff link, review questions and rubric
alt HM edits a question
HM->>Tool: PUT api hm signoff
Note over Tool: state rolls back to unsigned, recruiter must re-approve
else HM approves as-is
HM->>Tool: POST api hm signoff approve
Note over Tool: state becomes fully_signed
end
C->>Disp: Try to book or join interview
alt state is fully_signed or recruiter overrode
Disp-->>C: Interview created
else state is not fully_signed
Disp-->>C: 409, Question bank not signed off yet
end
Reset triggers — these wipe both signatures and reset state to unsigned:
editing the parsed JD, editing the HM notes, editing the custom questions.
Recruiter override "proceed without HM" requires a mandatory comment and is logged as proceed_without_hm in role_signoff_events.
Technology stack
| Layer | Technology | Purpose |
|---|---|---|
| Real-time transport | LiveKit Cloud | WebRTC rooms, SFU, recording (Egress to GCS) |
| Agent framework | livekit-agents 1.5.x | Joins room as AI participant; orchestrates STT / LLM / TTS pipeline |
| STT — English | Deepgram nova-3 | Low-latency English speech recognition |
| STT — Tagalog | Google STT chirp_2 | Filipino mass-position interviews (asia-southeast1, fil-PH) |
| LLM — interview turns | Claude Haiku 4.5 | Real-time Q&A reply generation (~1-3s TTFT) |
| LLM — scoring | Claude Sonnet 4.6 | Post-call transcript + visual evaluation against rubric |
| LLM — Tagalog | Gemini 2.5 Flash | Mass-position interview turns (Filipino) |
| TTS | ElevenLabs eleven_multilingual_v2 | Maria's voice — works for both English and Filipino |
| Backend | Python 3.11 + FastAPI | API server, webhook handlers, scoring pipeline |
| Frontend | Vanilla JS + HTML/CSS | Admin SPA + candidate join page (no build step) |
| Database | PostgreSQL 15 (Cloud SQL) | Roles, candidates, interviews, scores, rankings, analytics cache |
| Object storage | Google Cloud Storage | Video recordings, captured frames, exported scorecards |
| Gmail SMTP | Shortlist + invite + reschedule emails | |
| Hosting | Google Cloud Run (asia-southeast1) | Two services: Panayam API + Panayam Agent Worker |
| CI / CD | Cloud Build triggers on main | Auto-build + auto-deploy on every merge |
| Secrets | Google Secret Manager | API keys, DB URL, SMTP credentials |
Safety monitors (v0.4.0)
| # | Monitor | Source / cadence | Warn threshold | End threshold | Action |
|---|---|---|---|---|---|
| 1 | ConnectivityMonitor | LiveKit quality event (continuous) | 15 s sustained POOR / LOST | 60 s sustained | Maria speaks warning at 15 s; end + reschedule email at 60 s (end_reason=connectivity_interrupted) |
| 2 | SilenceMonitor | Poll loop, every 60 s | 60 s since last user turn → check-in #1 | 3 check-ins unanswered (~4 min total) | Each check-in is a spoken nudge; end + reschedule email after 3 (end_reason=prolonged_silence) |
| 3 | AbuseMonitor | Per user turn (event-driven); Haiku ~300-500 ms | LOW: 3 warnings · MED: 2 warnings · HIGH: 0 | Strike limit exceeded | HIGH: immediate end (no reschedule). MED/LOW: warning, then end on next breach. end_reason=abuse_<sev> |
| 4 | MultiSpeakerMonitor | Per user turn (event-driven); Haiku | 1st detection of second-voice → warn | 2nd detection → end | Strike-2 ends without reschedule (end_reason=integrity_breach); ambient noise counted separately, never penalised |
| 5 | Visual frame capture | Poll loop, every 60 s (configurable 10-600 s) | n/a (no real-time action) | n/a — used post-call only | Frames captured to GCS; scored as a separate Claude Sonnet visual-presence dimension after shutdown |
| 6 | Recording (LiveKit Egress) | Started on agent.candidate_joined, ends on session close | n/a | n/a | Full MP4 uploaded to voicescreen_recordings* bucket; URL surfaced on scorecard |
| 7 | Technical-failure handler | Hooked into LiveKit "error" / "unrecoverable_error" events | Any unrecoverable error | Immediate | Set end_reason=technical_failure, attempt apology TTS, send reschedule email |
| 8 | Health summary writer | Shutdown callback | n/a | n/a | Aggregates counters from monitors 1-4 into interviews.health_summary JSONB |
Process maps — when each monitor triggers
Sequence diagrams below show the actual control flow inside the agent worker for each safety monitor. Bold actor labels mark the trigger source; intervals on the arrows are the live values from the code (not aspirational).
ConnectivityMonitor (slice 1)
sequenceDiagram
participant LK as LiveKit Cloud
participant Mon as ConnectivityMonitor
participant Maria
participant CB as Shutdown callback
LK->>Mon: participant quality POOR or LOST, continuous event
Mon->>Mon: start poor-quality timer
Note over Mon: tick every 1s
Mon->>Maria: warning spoken at t 15s, says Im having trouble hearing you
Note over Mon,Maria: warning announced and counter incremented
Mon->>Mon: continue polling
Mon->>CB: end session at t 60s
Note over CB: end_reason set to connectivity_interrupted, reschedule email sent
SilenceMonitor (slice 2)
sequenceDiagram
participant STT as Deepgram / Google STT
participant Mon as SilenceMonitor
participant Maria as Maria (TTS)
participant CB as Shutdown callback
STT-->>Mon: user_turn event (resets timer)
Note over Mon: poll every 60s
Mon->>Maria: check-in #1 at 60s silence
STT-->>Mon: (still silent)
Mon->>Maria: check-in #2 at 120s silence
STT-->>Mon: (still silent)
Mon->>Maria: check-in #3 at 180s silence
STT-->>Mon: (still silent)
Mon->>CB: end session at 240s
Note over CB: end_reason=prolonged_silence
send reschedule email
send reschedule email
AbuseMonitor (slice 3)
sequenceDiagram
participant User as Candidate (speech)
participant STT as STT
participant Mon as AbuseMonitor
participant Haiku as Claude Haiku
participant Maria as Maria (TTS)
participant CB as Shutdown callback
User->>STT: spoken turn
STT-->>Mon: on_user_turn(transcript)
Note right of Mon: fire-and-forget — does NOT gate Maria's reply
Mon->>Haiku: classify(severity, category, rationale)
Haiku-->>Mon: {severity: LOW|MEDIUM|HIGH|NONE}
alt severity == HIGH
Mon->>Maria: end-call message
Mon->>CB: end (end_reason=abuse_high)
else severity == MEDIUM (strike 1 or 2)
Mon->>Maria: warning copy
else severity == MEDIUM (strike 3)
Mon->>Maria: end-call message
Mon->>CB: end (end_reason=abuse_medium)
else severity == LOW (strikes 1-3)
Mon->>Maria: warning copy (escalating tone)
else severity == LOW (strike 4)
Mon->>Maria: end-call message
Mon->>CB: end (end_reason=abuse_low)
end
MultiSpeakerMonitor (slice 4)
sequenceDiagram
participant User as Candidate (speech)
participant STT as STT
participant Mon as MultiSpeakerMonitor
participant Haiku as Claude Haiku
participant Maria as Maria (TTS)
participant CB as Shutdown callback
User->>STT: spoken turn
STT-->>Mon: on_user_turn(transcript)
Note right of Mon: fire-and-forget, parallel to AbuseMonitor
Mon->>Mon: cheap pre-filter (noise tokens, regex)
alt matches noise pattern
Mon-->>Mon: classify=ambient_noise (counter only, no action)
else passes pre-filter
Mon->>Haiku: classify (normal / ambient_noise / second_voice)
Haiku-->>Mon: category
alt second_voice (strike 1)
Mon->>Maria: warning
else second_voice (strike 2)
Mon->>Maria: integrity-breach message
Mon->>CB: end (end_reason=integrity_breach)
else ambient_noise
Mon->>Mon: increment ambient counter (no spoken action)
end
end
Visual frame capture (slice 5)
sequenceDiagram
participant Mon as VisualCaptureLoop
participant LK as LiveKit room video track
participant GCS as GCS frames bucket
participant CB as Shutdown callback
participant Sonnet as Claude Sonnet (vision)
Note over Mon: loop while session alive
loop every 60s (configurable)
Mon->>LK: read latest video frame
Mon->>GCS: upload frame as JPG
Mon->>Mon: record frame metadata (timestamp + GCS key)
end
CB->>Sonnet: score_visual_presence(frames)
Sonnet-->>CB: {score, assessment, evidence}
CB->>CB: write presentation_visual_presence dimension to scorecard
Technical-failure handler + Health summary (slices 7-8)
sequenceDiagram
participant LK as LiveKit session
participant Hook as session error handler
participant Maria
participant CB as Shutdown callback
participant DB as Cloud SQL
LK-->>Hook: unrecoverable_error event
Hook->>Maria: apology spoken, says Im sorry technical issue sending reschedule link
Hook->>CB: end with end_reason set to technical_failure
Note over CB: always runs on shutdown
CB->>DB: write interviews health_summary, monitor counters and end_reason
alt end_reason is abuse or integrity_breach
CB->>CB: skip scoring and reschedule, no reward for breach
else end_reason is connectivity_interrupted or prolonged_silence or technical_failure
CB->>CB: send reschedule email, skip scoring
else end_reason is completed normal
CB->>CB: trigger scoring pipeline, transcript and visual
end