Partner Maturity Assessment Platform: Case Study

Challenge

Sales and partner teams qualifying leads for observability, AIOps, and security tooling typically rely on unstructured discovery calls: slow to scale, inconsistent across reps, and purely extractive for the prospect.

Two personas, one product:

• Seller: a structured, repeatable way to qualify leads with real signal, a maturity read grounded in the prospect's own answers and researched context, not just firmographics.
• Buyer: something genuinely useful in return, a personalized maturity report across five domains (observability, AIOps, AI app observability, ML/foundation model ops, security).

The bet: a well-designed assessment with a genuinely insightful report produces better-qualified leads and a useful artifact for the buyer, value on both sides of the same interaction.

Role & Constraints

Owned end-to-end, solo: product definition, domain modeling, architecture, spec authorship, and implementation (spec-driven, with Claude as build agent). No team; every architectural call and correction was mine.

Self-imposed constraints:

• Single-machine deployable: no dependency on paid model APIs. Defaults to a local model (Ollama/llama3.2), swappable to Anthropic or OpenAI via one env variable, no code changes.
• No hard-coded domain data: pillars, questions, personas addable via DB rows only.
• Executable specs: the spec directory (00-index through 07-build-plan) was a real constraint, not idealized documentation; Claude Code loaded only the files relevant to each task.
• Async, typed, tested: FastAPI async throughout, typed TypeScript (no any), pytest coverage for routes, auth, scoring, and agent fallback behavior.

Approach

Used spec-driven development deliberately over open-ended "vibe coding": separating what the system should do (spec) from how it's built (implementation), keeping both in sync as the source of truth.

Spec structure: eight focused, versioned files (index, mission/outcomes, domain model, tech stack/constraints, data model, architecture/API, question bank, build plan), each scoped to a specific build task. A CLAUDE.md, generated from the specs, lived in the project root with a bootstrap prompt to regenerate it if lost. Specs were read-only during a build task; a spec that looked wrong got flagged, not silently reinterpreted.

System design: defined the user flow first: three roles (Prospect, Internal User, Admin) and a five-stage prospect journey (Landing → Pillar Select → Research Summary → Assessment → Report). Designed a three-agent LangGraph pipeline: research agent (company profile at prospect creation), question-selection agent (personalizes the assessment in the background after pillar choice), and report agent (scores answers, generates the report on submission). Built the LLM abstraction as a single factory function, with per-agent model overrides, so the system runs free and local or swaps to a hosted model via one env variable. Used Claude, working from the specs, to implement in ordered, git-branched tasks: scaffolding → database → auth → seed data → admin → assessment flow → scoring → agents → dashboard → verification.

Execution

The build moved fast once specs were in place. The harder, continuous work was keeping specs, architecture diagrams, and code in agreement as the system evolved.

Hardest parts:

• Consistency drift: code-level decisions (which page calls which endpoint, sync vs. polling) quietly outpaced the specs. Fixed via repeated reconciliation passes, treating any gap report as a cue to cascade-check all dependent files, not just the one flagged.
• New specification outside existing structure: extending the domain model or architecture beyond the original file boundaries required care to avoid duplicating content across files.
• Three-way sync: Claude Code skills, the pytest suite, and the Excalidraw diagrams all had to track the same reality as spec and code, not just each other.
• Correction discipline: when a Claude Code consistency report surfaced a mismatch, the fix was always reverting the spec to match actual behavior, never "cleaning up" the code toward something architecturally nicer. Resisting that took explicit rules, not just intent.

Outcome

A working, standalone product: full prospect-facing assessment flow, an internal dashboard for accounts/prospects, an admin panel for pillars/questions, and a three-agent LLM pipeline generating a structured report (executive summary, strengths, gaps, next steps), all running locally via Docker Compose, no external dependencies required.

Built to extend toward production, not as a finished product:

• New pillars/questions add via database, zero code changes, a constraint held throughout.
• The LLM layer moves from free/local to a hosted frontier model with a one-line env change, no local-only tech debt.
• The Datadog-specific application (a partner-facing sales qualification tool) is the proposed next chapter, currently under discussion, not yet built.

Reflection

As much a case study in working responsibly with an AI coding agent as in building the platform. Carrying forward:

• Define the user journey first, more rigorously. I designed the flow early, but I'd now lock down every screen-to-screen transition and edge case before writing any spec; most reconciliation passes traced back to journey ambiguity left unresolved.
• Spec-driven development works, but staying authoritative over time is still an open problem for me, not solved, just workable. Treating drift as a bug, not a documentation nuisance, was the highest-leverage habit I built.
• Need to learn how to scale vibe coding. What I would need to learn is to create skills, use agents (testing agents, UI consistency check agents, etc.), and define loops so that whole spec updates can be implemented through a smoother and easier process, so I can scale and do this faster.