Daily bookmarks: relevant news, facts and discoveries.
Articles 1219
Engineering ·
Radar: LiFT trains an image generator with flow matching and a shared transformer core, where each loop progressively refines the initial velocity estimate. According to the author, a model trained with 2 loops improves with up to 16 loops at inference.
Engineering ·
A post reports a single-voice, single-language text-to-speech model of about 9MB of weights, distilled from Kokoro-82M, which has 8M parameters. The author claims it runs anywhere and is very fast, without independent verification.
Engineering ·
A post reports removing up to 10 layers from Qwen3-8B-Base with LaCo. With 6 layers removed, the model is 16.7% smaller, keeping 78% to 94% of reasoning scores with a perplexity ratio of 1.68x, according to the author.
Engineering ·
An arXiv paper, dated October 7, 2026, proposing methods to reduce peak memory in distributed Mixture-of-Experts training, focusing on the all-to-all dispatcher.
Engineering ·
Radar: the post describes MOPD, a method in which a student model learns from its own rollouts and receives token-level supervision from domain-specialist teachers, aiming to combine the abilities of separately trained specialists.
Engineering ·
Radar: a post claims EmbeddingGemma 2, from Google, maps text, code, images, video and audio into a single vector space and uses an Apache 2.0 license. The source gives no metrics or technical documentation.
Engineering ·
Radar: the post presents JEPA-Anything, a domain-agnostic framework for world modeling based on orthogonal predictive factorization, which explores predictive representation learning across seven domains, from biology to climate.
Engineering ·
Radar item: LiFT repeatedly applies a shared Diffusion Transformer core and, according to the publication, outperforms a dense DiT on ImageNet 256×256 with about 60% fewer parameters.
Engineering ·
Radar: a report that 10 million EmbeddingGemma 2 vectors, quantized to 1 bit and reduced to 256 dimensions with MRL, dropped from about 30 GB to 0.4 GB of RAM, with a declared quality loss of about 5%.
Engineering ·
Briefing on a paper presenting Native Hybrid Attention, an approach that seeks to balance the speed of Transformers in long contexts with the retention of precise details from linear attention.
Engineering ·
Radar notes that Astral GPU Indexes now support PyTorch 2.13 and 2.14 and Python 3.15, with precompiled GPU-enabled wheels for FlashAttention, FlashAttention 3, causal-conv1d and DeepSpeed.
Engineering ·
Summary of a paper that investigates attention sinks and residual sinks in large language models and proposes that these outliers, along with softmax attention and RMSNorm, rescale other components of the architecture.
Engineering ·
Radar: the publication describes Dust, a zeroth-order optimization method that uses parallel perturbations of activations as a virtual population to pretrain transformers, with scale experiments and comparisons with backpropagation and evolution strategies.
Engineering ·
Radar: an article on sharing memory across recursions in recurrent Transformers, reporting a 76–79% reduction in context memory for models from 150M to 1B parameters, compared with standard Transformers.
Engineering ·
Report of scaling with nearly linear scalability and 35.3% MFU, using small-scale tests to narrow the search space before adding more GPUs.
Engineering ·
Radar: the SwiLA paper, from COLM 2026, proposes switching between multiple linear maps and discusses the trade-off between attention expressiveness and KV cache memory cost.
Engineering ·
Briefing on a section of the Machine Learning Engineering Open Book that describes symmetric memory in NCCL and PyTorch for small and medium payloads and for overlapping communication and computation.
Engineering ·
Radar: according to a note dated October 4, 2026, Google released a 4B Gemma model, fine-tuned to correct speaker labels and reduce word-level diarization errors on four benchmarks.
Engineering ·
A reported 450-million-parameter transformer trained with BF16 FlashAttention-3 saw its gradient norm rise 1,000× after 25 billion tokens, ending with higher loss than FP32 attention and no NaNs.
Engineering ·
Paper proposes rewriting expert demonstrations as trajectories that are correct for the task and likely under the target model, before using them in standard SFT.
Looking for something else?DocumentationContact
