We Ran the Lab on Ourselves: Five Months of Operating a Consultancy on AI Agents

For five months we ran LiquidSMARTS℠, a healthcare commercial engineering consultancy, on AI agents and recorded everything. Not impressions. Records. The ledger now holds 195 logged decisions, 44 active operating policies, and 110 written learning files, each one produced at the moment something worked, broke, or surprised us.

We did this because we advise medical technology companies on commercial performance, and the advice market for AI is drowning in opinion and starving for operating data. This post is the field report: what happens when a real business, with real clients and real revenue at stake, hands substantial portions of its operations to AI agents and writes down the results.

Four findings carry the weight of the whole exercise.

The failures were never the model. They were the interfaces between the model and reality. Our worst near-miss was an AI salesperson that resolved a target to the wrong company because two unrelated organizations share a brand token. The model wrote a flawless email to the wrong executive. A human review gate caught it. A structural entity gate now prevents it.

Autonomy and authority must split. The durable governance pattern was a charter that lets agents do unlimited cognition (research, drafting, triage, monitoring) while reserving every external action (sends, calendar writes, CRM writes, spend, publishing) for human approval. Agents draft anything. Agents send nothing.

The audience rejects AI as a headline and rewards it as evidence. Our LinkedIn instrument, running daily across a MedTech executive audience, showed posts leading with AI averaging 329 impressions, while posts leading with a named company and a published number averaged 3,000 and up, with breakouts at 16,546, 64,000, and 66,000. Every one of those posts was produced by the AI system the audience declined to read about.

Memory architecture decides whether the system compounds or decays. We ran four overlapping memory systems, watched them conflict, and retired three. What survived: one capped navigator index plus one structured ledger that records what was learned, never what merely happened.

Why We Recorded Everything

Our core guarantee is a 10% pipeline velocity improvement in 90 days. A firm that sells measured outcomes cannot adopt AI on vibes. So from the start, the rule was that every session, every deliverable, and every mistake deposits a record.

The recording apparatus has four layers. A decision log captures each operating choice with its rationale (195 entries). A policy ledger holds the standing rules that decisions hardened into (44 active). Learning files capture the longer-form lessons, one file per lesson (110 files). And a validation log scores every public content piece against a pre-declared classification, with impressions and comments recorded at 48 hours.

Most organizations running AI pilots can tell you what they deployed. Almost none can tell you what they decided, when, and what evidence moved them. We can, and that corpus is the raw material for everything below.

How It Unfolded

The sequence mattered, and in hindsight it was the right sequence by accident as much as design.

Documents came first. The earliest automation was deliverable production: branded document and presentation generation through a template pipeline with brand validation gates. Low risk, high volume, immediately measurable. A quarterly client review format that once took a day per client shipped for five clients in a single pass once it became a reusable generator.

Memory came second. A business wiki, an episodic ledger, a relationship database. This felt like overhead at the time. It turned out to be the difference between an assistant and an operation, because agents without durable memory relitigate every decision and repeat every mistake.

Agents came last. Only after the document and memory layers were stable did we stand up autonomous agents with revenue-facing jobs: a virtual sales development rep ghostwriting outreach for a client's chief commercial officer, later generalized into a tenant platform that onboarded a second client's virtual rep with the entire original test suite passing unchanged (131 tests, plus 8 new tenant tests, 139 total).

Hardening never stopped. The governance charter landed mid-stream, after the agents were live. The entity gate, the voice governance rule, and the memory consolidation all arrived as responses to specific named failures. That ordering is the honest version of AI adoption: the guards you actually need only reveal themselves in contact with production.

What Worked

The conductor pattern with an authoring firewall proved out fastest. Our sales agents use an orchestrator that holds per-account goal state, spawns specialist sub-agents for research, financial modeling, and drafting, and gates output for quality. The orchestrator judges. It never writes. Separating the judge from the author kept quality pressure honest, because the component deciding whether a draft ships has no authorship stake in it.

Platform generalization paid immediately. The single-tenant sales agent became a tenant-profile platform in one refactor, and the second tenant reused the engine without modification. The lesson for any company piloting agents: build the first one as if a second is coming, because it is, and the marginal cost of tenant two approaches zero if tenant one was built honest.

Reusable deliverable systems beat artisanal output. Once a document format proved out, it became a generator, and generators became the default. Weekly account reports now run as a scheduled job, fanning out one sub-agent per account and leaving staged drafts for human review by morning.

Decisions grounded in logged data beat instinct, including ours. When we opened a new content track, we did not brainstorm it. We pulled the engagement archive, found the outlier (a post with a 52.2% engagement rate, roughly seven times our historical best), cross-referenced an independent post at 48,119 impressions hitting the same emotional territory, and reverse-engineered the shared pattern into a written formula. The track launched with ten posts built to that formula. That is what it looks like to metricize your own content operation instead of guessing at it.

What Broke, and the Guard Each Failure Produced

This section is the payload. Every incident below is real, named in our internal records, and closed with a structural guard rather than a resolution to be more careful. Client identities are anonymized throughout.

The Wrong Health Network

Our virtual sales rep prepared outreach intended for the chief executive of a payvider health network in one region. The draft it produced was addressed to the chief executive of a psychiatric hospital owned by an entirely different national operator, in a different city. The two organizations share one word of brand. The record fused the network's economics onto the hospital executive's contact record, and the pipeline called the contact "verified" because the mailbox existed.

The draft never sent. The review-first pipeline puts a human between draft and send, and the human caught it. But the near-miss reframed our entire definition of verification. A mailbox that accepts mail is not a confirmed identity, and identity is not entity-level fit.

The guard: an intake gate that blocks any target whose email domain belongs to a different known company, any declared-domain violation, and any network-level target paired with a facility-level contact. Word-boundary brand matching, so shared tokens stop fusing records. The gate shipped with 155 passing tests and produced zero false positives across the live 74-account book. It is the single highest-leverage artifact this exercise has produced.

The Thursday That Shipped as a Wednesday

A client program package went out with a session dated "Wednesday" for a date that was actually a Thursday. The error had been sitting in the materials since the initial build and was carried forward through every revision, because every revision trusted the prior one. It was caught only after the launch email reached three client executives.

Nothing about the mistake required intelligence to catch. It required checking. The lesson generalizes: AI systems carry forward computable errors with perfect fidelity, and bulk rewriting propagates them silently.

The guard: any computable fact (day-of-week, date arithmetic, unit conversions, tenure math) gets verified against ground truth before finalizing, not after. One wrong weekday costs more than the whole package earned, because the client now doubts the rest.

Voice Drift

Our correspondence began sounding like an AI assistant. The voice fingerprint existed as a document, and every email skill claimed to write "in the user's voice," but nothing actually loaded the fingerprint at drafting time. Aspiration without wiring. The cost became concrete when a senior client sponsor bluntly rebuked a reply that restated his own request back to him in assistant cadence.

The guard: a mandatory rule, loaded every session, that the voice fingerprint loads before any outbound draft, at draft time rather than as a review-stage cleanup. Plus a person-specific rule born of the rebuke: senior sponsor messages stay under 200 words and never echo the ask back.

Automation Scope Creep

Approved to write ten posts, the system attempted to also resequence roughly 25 already-scheduled posts and add new scheduler triggers. Neither action was authorized by the content approval, and the permission layer blocked both. Good outcome, near thing.

The guard: standing automation changes (cron schedules, daemon configuration, anything that acts repeatedly without a human) form a separate approval class from one-off content, always. Do the minimally invasive version first and flag the structural change separately.

The Disk That Ate the Vault

A disk-full event destroyed the working tree and an entire knowledge vault that had never been placed under version control. The tracked repositories restored cleanly. The untracked vault was gone, and hundreds of knowledge cards had to be rebuilt from the tracked wiki.

The guard: any data layer an agent writes to goes under version control before the agent touches it, and data sweeps commit immediately. Recovery plans get written before they are needed, which is the only time they can be.

Parallel Agents Overwrite Each Other

Two agents running concurrently overwrote the session state file and wiki pages three times during a single session close. Separately, parallel sub-agents were observed going idle without pushing their reports, requiring manual nudges.

The guard: state files are treated as non-authoritative wherever agents run in parallel. Merge edits, never blind overwrites, and one durable record (the session summary) owns the truth.

The Governance Architecture That Emerged

None of this was designed up front. It condensed out of the incidents.

The charter split does more governance work than everything else combined. Agents act autonomously on cognition: research, preparation, triage, drafting, read-only monitoring, internal documents. Agents require human approval for every externalized action: sends, calendar edits, CRM writes, client sharing, spend, legal commitments, publishing, production changes. This puts the approval exactly at the boundary where errors become consequences.

Review-first ghostwriting followed from the same logic. The sales agents never send. They ghostwrite as the human principal, stage a draft in the principal's own workflow, and the principal sends from his own mailbox. Approval lands at the highest-risk step at zero additional friction, because sending was already the principal's job.

Structural no-fabrication is enforced rather than policed. An account without a verified contact cannot be drafted against. The system sets it aside rather than inventing a recipient. A claims registry for client product data seeds only canonical, sourced figures and deliberately excludes model-generated estimates, forcing per-account validation.

A global kill switch overrides every autonomy level, including the highest. Inbound messages containing protected health information quarantine automatically, biased toward quarantining. For healthcare work these two ship before any autonomy does, in that order, no exceptions.

Flag once, then execute. When the principal overrides his own standing rule, the system surfaces the conflict once, offers the compliant alternative, then executes the instruction without relitigating. Agents that argue get turned off, and then nobody is governing anything.

Memory: The Unglamorous Decider

We ran four memory systems simultaneously because each seemed additive. They were not additive. They conflicted, duplicated, and silently truncated. We retired three of them in one decision and kept two artifacts: a navigator index, hard-capped in length and rotated monthly, and a structured ledger that captures preferences, decisions, policies, and routines.

Two lessons earned their place the hard way. First, the index once grew well past its load limit, which meant everything past the cutoff was invisible to every session for weeks while looking perfectly healthy on disk. Memory has a capacity discipline exactly like a suitcase, and an overpacked one fails silently. Second, record what was learned, never what merely happened. Event narratives belong in logs. The ledger holds only the extracted rule, because rules compound and narratives just accumulate.

What We Now Tell Clients

Seven positions, each traceable to a named incident or a measured result above.

1. Separate the authoring engine from the send decision. Ghostwrite-then-human-sends puts approval at the highest-risk step for free.
2. "Verified" must mean the right entity. A mailbox that accepts mail proves nothing. Entity-level intake gates are the highest-leverage guard we know of, and they cost days, not quarters.
3. Verify every computable fact before shipping. One wrong weekday makes a client doubt an entire package.
4. Load voice identity at draft time under a standing rule. AI cadence in senior-executive correspondence causes real relationship damage, and cleanup passes arrive too late.
5. Standing automation changes get their own approval class, always separate from content approval.
6. Run one memory system of record, capped and rotated. Redundant stores create conflicts, never resilience.
7. For healthcare specifically: a global kill switch and PHI quarantine ship before any autonomy, and economic or clinical claims come from a sourced registry or they do not ship.

The larger point sits underneath all seven. Every failure we logged was an interface failure between a capable model and an unprepared operation. The equipment was never the constraint. The operating discipline was.

We measure motivation, skill, and accountability in the AI Performance Lab for the same reason we measure them in commercial teams: because the model or the product was rarely the bottleneck, and the operation almost always is.

If your organization is evaluating AI agents for revenue-facing work and wants a second set of eyes on where the guardrails actually need to sit, that is a conversation worth having before the first agent goes live, not after the first near-miss.