Thinking Machines Lab 分析 LoRA 在哪些条件下可媲美全量微调

Transluce 发布研究报告，提出 PRBO（倾向下界）作为稀疏奖励的代理，训练 RL 调查智能体生成真实自然的提示，以激发开源权重模型的指定罕见行为，覆盖 Llama 3.1/4、Qwen 2.5 和 DeepSeek-V3。

Meta 与卡内基梅隆的论文提出用强化学习训练小型开源模型做工具选择，把请求拆成步骤、用关键词搜索并逐步选出工具。在 ToolBench 上，训练后的 Qwen3-8B 精确选中正确工具的比例为 41.5%，高于 Llama-4-Maverick（402B）最佳提示词设置的 30.6%；无需重新训练即在 4 个新工具基准中的 3 个上超过 Maverick，且换更强检索器后表现继续提升。

Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves duri…

Multi-token prediction (MTP) can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass. However, the best speculation horizon changes with context: longer drafts offer greater amorti…

As Large Language Models (LLMs) continue to scale, their energy consumption during inference has become a critical concern. While Autoregressive (AR) models are the industry standard, Diffusion Language Models (DLLMs) have emerged as a pro…

LLMs absorb vast knowledge from training data, yet their inherently probabilistic nature offers no guarantee that this knowledge will surface in their outputs. To bridge this gap, Knowledge Extraction Enhancement (KEE) techniques such as a…

Post-training pruning enables efficient deployment, but conventional methods do not explicitly preserve internal-gradient alignment between sparse and dense models. We study novel connections between downstream performance of compressed mo…

清华 NLP（OpenBMB 成员）联合中科院大学、东北大学、UIUC 和约翰霍普金斯大学提出 One-Shot OPD，将 OPD 训练集压缩到一条查询：数学任务上从 59.1 提升至 68.5（300 步），达到全量数据 OPD 69.8 的 87% 增益，并在代码、指令跟随和智能体工具使用上跨 Qwen、Llama、OLMo 成立。

Sliding-window KV inference refers to processing a sequence incrementally while retaining only a fixed-size cache of recent key and value states. It can be applied to pretrained causal transformers at inference time without additional trai…

研究在 Llama3 与 Qwen2.5 系列上做强弱蒸馏实验，独立改变 rollout 策略、token 级 KL 方向和学习率，发现 rollout 策略并非核心因素：前向 KL 对 rollout 策略极不敏感且表现稳定，反向 KL 则明显更敏感、偏好学生模型自生成的 rollout，而学习率主导遗忘与更新稀疏性。