Research Topics

The workshop discussion kept circling one bottleneck: inference is becoming a memory, scheduling, and reuse problem as much as a modeling problem.

Below is the organized research map. The individual pages go deeper on the highest-leverage ideas. The broader list is kept here so the whole brainstorm remains searchable.

Priority Research Tracks

#

Full Idea Inventory

#

Workshop-Derived Ideas

#

1. Temporal / distance-aware dynamic quantization of KV cache - keep recent tokens in high precision, compress old tokens more aggressively.
2. TurboQuant for pre-softmax attention scores - test whether rotation-based quantization reduces score-matrix outliers.
3. TurboQuant with LoRA fine-tuning - study whether task adapters can compensate for inference-time KV quantization error.
4. TurboQuant plus temporal compression - combine recency tiers with rotation-aware quantization.
5. Input-adaptive attention head pruning - prune heads that become redundant on a specific prompt.
6. Unlearning layer inside MHA - use an inference-time or lightly trained mask to weaken token associations.
7. KV sharing via prefix hashing - already partly deployed as prefix caching, but position-awareness still limits reuse.
8. KV compression for video models - natural fit because frame time maps to token distance.
9. Parallel transformer blocks vs sequential depth - useful but crowded architecture search territory.

Book-Derived Ideas

#

1. Roofline-aware adaptive batching.
2. Cross-layer KV cache aliasing by cosine similarity.
3. Speculative prefill with a draft model.
4. SLO-differentiated KV cache tiering.
5. Attention-sink sliding windows with RoPE re-anchoring.
6. Different quantization for prefill and decode GPU pools.
7. CUDA graph capture for padding-free packed variable-length batches.
8. Subliminal preference transfer auditing for distilled models.
9. CPU draft plus GPU verify speculative decoding.
10. Position-invariant prefix caching via RoPE-agnostic keys.
11. Continuous batching with SLO priority preemption.
12. FlashAttention-style tiling for Mamba / SSM selective scan.
13. Ridge-chasing speculative decoding window size.
14. Multi-turn KV persistence with forgetting curves.
15. Quantization error as an uncertainty / hallucination signal.
16. MoE-style attention head routing.
17. Thermal-budget-aware edge inference.
18. Shared document KV cache for RAG.
19. Online learning for EAGLE draft heads.
20. Disaggregated world models for embodied AI.

Mentor / Hardware Ideas

#

The mentor material and Bjarke Roune document push the same theme down to silicon:

• AI CPUs with systolic arrays and large SRAM.
• Compiler backend studios for new accelerators.
• HBM minimization planners.
• SSD-backed long memory.
• DMA compression engines that combine lossy quantization with lossless entropy coding.
• 1:2 and 2:4 sparsity toolkits.
• Memory hierarchy explorers.
• MoE token-router and network-topology co-design.
• Tiled software pipeline libraries.
• Tokens-per-dollar observability.
• AI chip co-design search.

Background Reading

#

• TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
• Efficient Streaming Language Models with Attention Sinks
• FlashAttention
• vLLM automatic prefix caching
• PagedAttention / vLLM paper
• Cache-Craft: Managing Chunk-Caches for Efficient RAG
• RAGCache
• TurboRAG
• Orca: iteration-level scheduling
• EAGLE speculative decoding
• LoRA
• FlexGen
• Mamba