Daily bookmarks: relevant news, facts and discoveries.
Articles 1219
Engineering ·
A September 29, 2026 report says BF16 rounding in FlashAttention may break a softmax-gradient symmetry and cause gradients to explode near the end of training. The authors propose GProj as a correction.
Engineering ·
The article examines positional encodings for Transformers, notes RoPE's bias toward nearby tokens, and organizes evaluations of alternatives across distinct configurations.
Engineering ·
A September 29, 2026 preview presents a benchmark for comparing information-flow methods and assessing whether deeper Transformers use their layers effectively.
Engineering ·
Published on September 29, 2026, the summary describes a study on transferring knowledge from pretrained Transformers to Mamba models and notes memory and throughput advantages for SSMs.
Engineering ·
A September 29, 2026 entry describes KITE, a scaling paradigm combining scaling with upcycling and expanding decoder components that reuse a KV cache generated by a smaller prefiller.
Engineering ·
Long context can extend exposure. Separate what an agent needs to remember from the identity of people mentioned in a conversation.
Engineering ·
Published on September 29, 2026, the MiMo RL Environment Explorer is described as an interactive space to inspect MiMo-V2.6 reinforcement-learning environments and run rollouts.
Engineering ·
A publication dated September 29, 2026 describes a 7B model that continuously updated its weights using prediction-market data and ran continuous backtests over 390 days. It says the model outperformed several larger models in the same test environment.
Engineering ·
The approach distills linear attention models in long contexts and, according to the report, fully recovers full-attention retrieval while reaching 83–93% of mathematical reasoning performance with 3 billion tokens.
Engineering ·
A paper published on September 29, 2026 tests on-policy distillation with one query across mathematics, coding, instruction following, and agentic tool use.
Engineering ·
Published on September 29, 2026, PTXBench evaluates model-generated matrix multiplication and attention kernels in architecture-specific PTX for H100 and B200 GPUs.
Engineering ·
The September 29, 2026 entry describes RESA, a residual-estimation proposal for recovering context missed when sparse attention selects only part of the KV cache. The post claims the technique adds no computational cost.
Engineering ·
Announced on September 29, 2026, TIRx Harness is an open compiler harness for agents to explore, debug, and optimize GPU kernels, with compiler analysis, a catalog, hardware specifications, and remote evaluation.
Engineering ·
Published on September 29, 2026, the bookmark describes Vx's use of the `mlir!` macro to lower MLIR directly into leading-zero count instructions, avoiding manually maintained routines for different hardware.
Engineering ·
An isolated runtime controls what an agent can do. Privacy also requires controlling the content it sends to AI.
Engineering ·
Published on September 28, 2026, the post reports that kernel design agents generated and optimized Kimi Delta Attention kernels, reaching up to 2.96× the performance of FlashKDA on B300 with one-tenth the state error.
Engineering ·
Published on September 28, 2026, the article describes a hierarchical block-selection method for sparse attention in long contexts, reducing selection cost from O(N²) to O(N log N).
Engineering ·
Turning speech into text changes the format, not the sensitivity. Evaluate every stage before introducing audio into a workflow.
Engineering ·
Published on September 28, 2026, the collection presents 200 environments for studying reinforcement learning in challenging, open-ended, long-horizon tasks.
Engineering ·
A September 28, 2026 announcement presents open-source tools for pretraining, attention, checkpoint conversion, and language-model serving.
Looking for something else?DocumentationContact
