https://www.jonam.io/categories/inference-engineering/Recent content in Inference-Engineering on jonam'LogHugo -- gohugo.ioen© 2026 Manoj. All Rights Reserved.Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/docvault/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/docvault/Compute any document’s context once, serve it to every user forever.https://www.jonam.io/journal/inference-engineering/research-topics/position-invariant-document-kv-cache/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/position-invariant-document-kv-cache/Can document KV states be cached independent of prompt position and reused across RAG queries?https://www.jonam.io/journal/inference-engineering/product-ideas/prefillx/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/prefillx/Cut TTFT for long-context document applications by precomputing and repairing reusable KV states.https://www.jonam.io/journal/inference-engineering/research-topics/temporal-turboquant-kv-tiering/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/temporal-turboquant-kv-tiering/Recent tokens stay high precision, older tokens degrade to INT4 or INT2, and TurboQuant makes the low-bit tiers less painful.https://www.jonam.io/journal/inference-engineering/product-ideas/infergrid/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/infergrid/Measure why your GPU bill is high, then tune batching, speculation, and quantization automatically.https://www.jonam.io/journal/inference-engineering/research-topics/roofline-adaptive-inference-scheduler/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/roofline-adaptive-inference-scheduler/Move from static max_num_seqs to a feedback loop that chases the hardware ridge point.https://www.jonam.io/journal/inference-engineering/product-ideas/draftos/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/draftos/Use idle CPU cores on GPU instances to draft tokens while the GPU verifies.https://www.jonam.io/journal/inference-engineering/research-topics/speculative-prefill/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/speculative-prefill/Speculative decoding is common; this asks whether speculation can reduce long-prompt prefill latency.https://www.jonam.io/journal/inference-engineering/research-topics/quantization-divergence-hallucination-signal/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/quantization-divergence-hallucination-signal/If FP8/INT4 and FP16 disagree sharply, the model may be in a fragile region.https://www.jonam.io/journal/inference-engineering/product-ideas/sloguard/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/sloguard/Protect enterprise P99 latency without buying more GPUs.https://www.jonam.io/journal/inference-engineering/product-ideas/haloscoreai/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/haloscoreai/A low-latency uncertainty signal for regulated AI applications.https://www.jonam.io/journal/inference-engineering/research-topics/online-eagle-draft-learning/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/online-eagle-draft-learning/Speculative decoding throws away a useful supervision signal: which draft tokens were accepted.https://www.jonam.io/journal/inference-engineering/product-ideas/distillaudit/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/distillaudit/Detect hidden preference transfer from teacher models to students.https://www.jonam.io/journal/inference-engineering/research-topics/slo-aware-kv-cache-tiering/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/slo-aware-kv-cache-tiering/Premium users get hot KV blocks; batch users spill to cheaper memory tiers.https://www.jonam.io/journal/inference-engineering/research-topics/attention-head-similarity-pruning/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/attention-head-similarity-pruning/Measure cross-head similarity on a prompt and skip heads that are redundant for that input.https://www.jonam.io/journal/inference-engineering/product-ideas/convocache/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/convocache/Store and rehydrate the conversation state that actually mattered.https://www.jonam.io/journal/inference-engineering/product-ideas/specdraft-cloud/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/specdraft-cloud/A draft model service that learns from accepted and rejected tokens.https://www.jonam.io/journal/inference-engineering/research-topics/unlearning-layer-in-attention/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/unlearning-layer-in-attention/Can we attenuate undesirable token associations inside attention without full retraining?https://www.jonam.io/journal/inference-engineering/research-topics/hardware-aware-inference-cpu-ideas/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/hardware-aware-inference-cpu-ideas/The software layer that becomes valuable if inference hardware shifts from GPU-first to AI CPU and custom accelerator designs.https://www.jonam.io/journal/inference-engineering/product-ideas/neuraledge/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/neuraledge/Schedule inference around thermal limits and split reflexes on-device from planning in the cloud.https://www.jonam.io/journal/inference-engineering/research-topics/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/research-topics/Novel and practical research directions around KV cache compression, scheduling, speculation, quantization, and hardware-aware serving.https://www.jonam.io/journal/inference-engineering/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/A living notebook for inference engineering research topics and product ideas.https://www.jonam.io/journal/inference-engineering/product-ideas/Mon, 18 May 2026 00:00:00 +0000https://www.jonam.io/journal/inference-engineering/product-ideas/Ten product directions built from KV cache reuse, roofline scheduling, speculative decoding, and inference observability.