Open Problems

what I'm seeing

Every serving stack I tune looks excellent on a fixed prompt mix and degrades the moment real traffic arrives. The benchmark number is a steady-state average over a homogeneous batch. The number that gets someone paged is p99 under a mix of 200-token and 8000-token prompts arriving in the wrong order. Continuous batching improves the average and can actively hurt the tail — one long prefill sits in front of a queue of short decodes and everything behind it waits.

what I've tried

Bucketing the prompt mix and routing each bucket to its own replicas. Capping max sequence length per replica. Chunked prefill, so a long prompt can't monopolize a scheduler step. Measuring p99 at fixed concurrency rather than at saturation, which at least makes the number honest. Each of these moves the tail; none of them tells me what the tail should be for a given shape of traffic.

what I don't know

Whether tail latency in a continuously-batched server is a scheduling problem with a known-good policy that simply hasn't been published, or whether it's irreducible without disaggregating prefill and decode onto separate pools. The two answers imply very different capacity plans and very different bills, and I've had to commit to one without being able to defend it from first principles.

→ if you run a fleet where p99 is a contractual number, I want to know what your scheduler actually does.

what I'm seeing

On paper, concurrency is arithmetic: weights, plus KV cache per sequence, divided into memory. In practice the ceiling arrives earlier and far less predictably, because sequence length is unknown at admission time. You accept a request believing it's short, it isn't, and now you either preempt something mid-flight or swap and pay for it in latency. The cache is the first resource to run out, and it runs out based on a number nobody has yet.

what I've tried

Conservative admission against max_model_len, which leaves most of the throughput on the floor. Optimistic admission with preemption and recompute, which is fine until a burst of long generations lands together. Prefix caching to reclaim the shared portion of a prompt. Predicting output length from the prompt with a small classifier — right often enough to be genuinely dangerous.

what I don't know

Whether output-length prediction can ever be reliable enough to schedule against, or whether the correct move is to stop predicting entirely and make preemption cheap enough that mispredicting doesn't matter. Paged attention made the memory side tractable; the admission side still feels pre-paging to me.

→ if you've built admission control that survives a burst of long generations, I'd like to read it.

what I'm seeing

Scientific pipelines are several different programs wearing a single name. Profiling protein-folding workloads end-to-end, the stages have almost nothing in common: MSA search is I/O- and CPU-bound against large sequence databases, the Evoformer stack is the GPU-bound part everyone assumes is the whole job, and the structure module is short and neither. Size the cluster for the GPU stage and the search starves it. Size it for the search and you rent idle accelerators.

what I've tried

Profiling each stage in isolation and provisioning separately — the right answer with the wrong ergonomics, because the scheduler then has to carry state across stages. Running search on a CPU pool and staging results to fast storage. Batching many targets so one stage's idle overlaps another's peak, which works best and is the hardest thing to explain to a customer reading a single per-job runtime.

what I don't know

Whether the correct unit of scheduling here is the job or the stage, and whether any of the standard HPC schedulers express that well enough to rely on. Every team I've worked with has rebuilt the same stage-aware orchestration by hand, which suggests a missing abstraction rather than an unpopular one.

→ if you run mixed CPU/GPU scientific pipelines at cluster scale, I want to compare how you schedule them.

what I'm seeing

At loop depths past 10, every agent I've built sits on a bad trade-off. Truncate the trace aggressively and the agent forgets why it started. Carry the full trace and it gets confused by its own prior errors — re-trying the failed call, or worse, over-correcting in a direction the user never asked for. Reflection steps help, but they add latency we can't afford in any of the production paths I care about.

what I've tried

Structured error summaries injected at fixed intervals. A separate "error memory" channel that gets compacted on every Nth turn. Smaller models acting as summarizers between turns. Hard caps on retries-per-tool-per-thread. Each helps a bit; none of them feels like the right primitive.

what I don't know

Whether this is a model capability gap that will close in the next generation, or a loop-architecture gap that the labs are shipping the wrong abstractions for. I lean architecture — but I'm not confident, and the answer changes what enterprise agent platforms should build now.

→ if you've shipped an agent that holds up at depth 20+, I want to compare notes.

what I'm seeing

Every production agent I've put in front of a real user eventually has to be told "you have N dollars / M tokens / K minutes — spend them well." The way we express that today is some combination of context-window math, a recursion-depth cap, and a wall-clock timeout. None of those is the thing the user actually cares about. Users care about was this worth the spend, and no current primitive lets the agent reason about that mid-loop.

what I've tried

Injecting a "remaining budget" line into the system prompt at every turn. Letting the agent call a check_budget() tool. Wrapping the whole agent in a controller that kills the loop at a hard limit. The first is ignored under load. The second is honored but rarely used wisely. The third works but feels like driving with the parking brake on.

what I don't know

Whether "budget" wants to be a first-class concept the model is trained on, a runtime contract the platform enforces, or a separate planning loop that runs outside the executor. I suspect the right answer is all three at different layers, and that nobody is yet shipping the layered version.

→ if your team has shipped a real cost-aware agent in production, I'd love to compare architectures.

what I'm seeing

Every regulated deployment ships with an audit trail. At t=0, the trail is full and detailed and nobody reads it. At t=6 months, the regulator wants to know why a specific decision was made on a specific day, and the trail has either rotated out, been compressed past the point of usefulness, or refers to a model version that no longer exists. Either way it can't answer the question.

what I've tried

Snapshotting prompts and model versions alongside the decision. Storing structured "reasoning summaries" rather than raw chain-of-thought. Building a small replay harness that re-runs the original prompt against the original model on demand. The first two are cheap and partially useful. The third is the only one that actually answers regulator questions, and it doesn't scale.

what I don't know

Whether the labs will offer model-version permanence as a paid product (so replay actually works long-term), or whether enterprises have to accept that the audit trail is a snapshot of intent, not of behavior. The two answers imply very different compliance regimes, and the industry is sleepwalking past the choice.

→ if you work in MLOps for a regulated industry, I want to know what your replay story looks like.