每日 arXiv 论文简报
今日arXiv论文呈现自回归与扩散模型双轨并进的格局,两者合计35篇,占据绝对主导地位。值得注意的是,两类模型在图像生成与3D Avatar领域出现明显交叉——如Concept Guidance、Marionette、Omni-LiveAvatar、WAM-Diff2等论文同时出现在两个分类中,体现了AR与Diffusion技术融合的趋势。
整体趋势来看:
- 控制与可解释性成为焦点:Concept Guidance实现训练无关的精确生成控制,PROVE探索Prompt恢复的可验证性,The Linear Geometry研究Diffusion模型的可解释token与安全攻防
- 多模态融合深化:从RGBX-Next的G-Buffer渲染到Marionette的“预测状态-渲染几何-绘制外观”三位一体,模型正在打通视觉、语言、几何的边界
- 垂直领域深化:自动驾驶(InfiniVerse、WAM-Diff2)、音乐生成(MusicLayout)、生物信息学(Program-space Diffusion)、通信(MIMO检测)等专业方向涌现定制化模型
- 效率优化持续:WAM-Diff2通过AR到Diffusion蒸馏实现高效VLA,MagnifiQ探索渐进式超分辨率
最值得关注的研究:
- Concept Guidance:首个真正实现训练无关、精确 latent 控制的文本到图像方法,兼容多种Diffusion架构
- Marionette:统一预测世界状态、渲染几何、绘制外观的Avatar框架,代表多模态生成的新范式
- WAM-Diff2:AR-to-Diffusion蒸馏的层次化设计,为自动驾驶VLA提供高效方案
- The Linear Geometry of Interpretable Tokens:从几何角度揭示Diffusion模型的jailbreak机制,为模型安全提供理论基础
- Retrieval Grounding Latent Reasoning:将检索引入推理过程,为Dense Retrieval提供新思路
今日 Autoregressive 类别论文概述:
今日自回归相关论文主要聚焦于生成式模型的训练效率提升与多模态应用拓展。整体趋势呈现三大方向:1) 蒸馏压缩,如 WAM-Diff2 将自回归模型知识迁移至扩散模型以提升自动驾驶效率;2) 因果推理增强,如 ForgeWM 通过渐进式因果训练改进视频世界模型;3) 多模态实时生成,如 Omni-LiveAvatar 实现分钟级实时音视觉化身。亮点在于自回归范式正从传统语言建模扩展至推荐系统、音乐生成、场景补全等新领域,且与扩散模型、Transformer 的融合趋势明显。
重点论文推荐:
- WAM-Diff2: Hierarchical AR-to-Diffusion Distillation — 首次提出从自回归到扩散的层级蒸馏,为高效自动驾驶 VLA 模型提供新范式,值得关注其蒸馏机制设计。
- EchoRec: Multi-Item Prediction-Empowered Generative Recommendation — 将多项目预测引入生成式推荐,通过偏好对齐提升推荐质量,是推荐系统与自回归结合的创新尝试。
- Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Visual Avatar Generation — 实现分钟级实时音视觉化身生成,对实时互动应用有重要价值。
- MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation — 引入显式结构规划增强音乐生成的可控性,对音乐 AI 创作有重要意义。
- Marionette: Predicting World States, Rendering Geometry, Painting Appearance — 统一框架预测世界状态、渲染几何与外观,展现自回归在具身智能中的潜力。
Retrieval Grounding Latent Reasoning for Dense Retrieval
中文标题:面向密集检索的检索接地潜在推理
作者:Gang Zhou, Xiongxi Yu, Hu Tian, Yang Wei, Lu Pan, Ke Zeng, Shibiao Xu, Xiaolong Zheng
Reasoning-intensive retrieval requires text representations to capture not only semantic similarity, but also the reasoning needed to determine relevance under a given retrieval instruction. Existing reasoning-enhanced embedding models improve retrieval by incorporating reasoning information into dense representations, yet their supervision is typically dominated by the final retrieval objective. As a result, latent reasoning trajectories may learn shortcut reasoning patterns that preserve retrieval performance without producing meaningful incremental retrieval gains. We propose Retrieval Grounding Latent Reasoning (RGLT), a latent reasoning framework for dense retrieval that explicitly connects intermediate latent transitions with retrieval improvements. RGLT performs non-autoregressive reasoning in hidden space through an instruction-conditioned latent reasoning trajectory constructed from silent tokens. It combines process-supervised explicit-to-implicit distillation with retrieval-grounded supervision, using stage-wise CoT reconstruction to shape intermediate latent states and retrieval-effect credit to optimize incremental retrieval gains across the latent reasoning trajectories. Experiments on reasoning-intensive retrieval benchmarks show that RGLT consistently outperforms strong baselines while preserving efficient embedding inference.
推理密集型检索要求文本表示不仅能够捕捉语义相似性,还需捕捉在给定检索指令下判断相关性所需的推理能力。现有的推理增强嵌入模型通过将推理信息融入密集表示来改进检索,但其监督通常以最终检索目标为主。因此,潜在推理轨迹可能学习到保留检索性能的捷径推理模式,却无法产生有意义的增量检索增益。我们提出检索接地潜在推理(RGLT),这是一种将中间潜在转换与检索改进明确关联的密集检索潜在推理框架。RGLT通过由静默令牌构建的指令条件潜在推理轨迹,在隐藏空间中进行非自回归推理。它结合了过程监督的显式到隐式蒸馏与检索接地监督,使用阶段式思维链重建来塑造中间潜在状态,并利用检索效果信用来优化潜在推理轨迹上的增量检索增益。在推理密集型检索基准数据集上的实验表明,RGLT在保持高效嵌入推理的同时持续优于强基线方法。
EchoRec: Multi-Item Prediction-Empowered Generative Recommendation via Cycle-Consistent Preference Alignment
中文标题:EchoRec:通过循环一致性偏好对齐实现的多项目预测赋能生成式推荐
作者:Haokai Ma, Aoqi Hu, Yueao Xing, Ruobing Xie, Yonghui Yang, Teng Tu, Lei Meng, Tat-Seng Chua
Generative recommendation autoregressively generates the semantic IDs of the target item, unifying preference modeling and index retrieval within the shared token space. Recent attempts have introduced Multi-Token Prediction (MTP) into this field, yet they primarily inherit its efficiency merit, leaving its potential as dense supervision unexplored. Unlocking this potential hinges on whether future behaviors qualify as informative supervision. Our analysis reveals that future behaviors carry a semantic echo of the current one far above that of random pairs, which nevertheless decays along horizons under intent transitions, making them informative yet order-dependent signals. Motivated by this, we propose EchoRec, which empowers MTP with cycle-consistent holistic preference alignment across multi-horizon for generative recommendation. It comprises two synergistic modules. Horizon-aware Preference Generation (HPG) sequentially chains lightweight auxiliary branches upon the base recommender, where each branch conditions on its predecessor to respect preference evolution. Verifiable Holistic-Preference Alignment (VHA) further consolidates them into the holistic preference and echoes it back through cycle-consistent projectors to suppress spurious alignment, with theoretical guarantees that exclude the rank-collapse form of spurious alignment under an invertible transport, enabling the holistic preference to be retained in the decoding representation. All auxiliary components serve as disposable scaffolding discarded at inference, introducing negligible online serving overhead. Extensive experiments on three datasets demonstrate the superiority of our EchoRec, together with its naturally acquired multi-item generation ability. Our code and datasets will be available upon acceptance.
生成式推荐通过自回归生成目标项目的语义ID,将偏好建模和索引检索统一在共享的token空间中。近期的尝试将多token预测(MTP)引入该领域,但主要继承了其效率优势,而未充分挖掘其作为密集监督信号的潜力。释放这一潜力的关键在于未来行为是否能构成有效的监督信息。我们的分析表明,未来行为承载着远超随机配对的当前行为语义回声,然而在意图转换过程中,这种回声随时间 horizon 衰减,使其成为信息丰富但顺序依赖的信号。受此启发,我们提出了EchoRec,通过跨多horizon的循环一致性全量偏好对齐来增强MTP以实现生成式推荐。EchoRec包含两个协同模块。Horizon感知偏好生成(HPG)将轻量级辅助分支顺序链接到基础推荐器上,每个分支以其前驱为条件以尊重偏好演化。可验证全量偏好对齐(VHA)进一步将它们整合为全量偏好,并通过循环一致性投影器将其回传以抑制虚假对齐,理论上保证了在可逆传输下排除秩崩溃形式的虚假对齐,使全量偏好得以保留在解码表示中。所有辅助组件作为可丢弃的脚手架在推理时被剔除,引入几乎可以忽略的在线服务开销。在三个数据集上的广泛实验证明了我们EchoRec的优越性,以及其天然获得的多项目生成能力。代码和数据集将在论文录用后公开。
ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Models
中文标题:ForgeWM:面向少步动作条件视频世界模型的渐进式因果训练
作者:Xinye Li, Lingshuai Lin, Lei Wang, Liuzhou Zhang, Jialin Cui, Qingshan Li, Guanchu Wang, Qingbin Liu, Xi Chen, Jiang Bian, Wai Lam
Action-conditioned video world models require low-latency causal generation and reliable responses to game-native controls. Although causal distillation enables one- or few-step video synthesis, extending it to interactive world models remains challenging, as discrete keyboard states and continuous mouse motion must remain aligned with temporally compressed latent chunks during causal training and autoregressive rollout. We introduce ForgeWM, a progressive framework that transforms a bidirectional action-conditioned video generator into efficient few-step world models through domain adaptation, teacher-forced causal training, causal consistency distillation, and on-policy distribution matching with a bidirectional teacher. The resulting budget-specialized students operate at steady-state denoising budgets of 1, 2, and 4 steps. ForgeWM further supports a dual-path deployment protocol combining latency-critical interaction with optional replay-time refinement, where the one-step student re-noises and refines its saved draft. On paired Minecraft trajectories, ForgeWM leads the evaluated systems in Imaging Quality, reference-aligned motion-profile agreement, action-sign accuracy, and mouse-control accuracy, while achieving the lowest reference LPIPS; the same four-stage recipe transfers to gamepad-controlled FPS gameplay. Replay-time refinement matches four-step reference quality while remaining roughly three times closer to the experienced trajectory than regeneration from noise. These results demonstrate ForgeWM's effectiveness for controllable few-step video generation.
动作条件视频世界模型需要低延迟因果生成以及对游戏原生控制的可靠响应。虽然因果蒸馏能够实现一步或少步视频合成,但将其扩展到交互式世界模型仍然具有挑战性,因为在因果训练和自回归生成过程中,离散键盘状态和连续鼠标运动必须与时间压缩的潜在块保持对齐。我们提出了 ForgeWM,一个渐进式框架,通过领域适应、教师强制因果训练、因果一致性蒸馏以及与双向教师的在线策略分布匹配,将双向动作条件视频生成器转化为高效的少步世界模型。生成的预算专用学生模型在1、2和4步的稳态去噪预算下运行。ForgeWM 还支持双路径部署协议,将延迟关键的交互与可选的重放时精细化相结合,其中一步学生模型对其保存的草稿进行重新噪声处理和精细化。在配对的 Minecraft 轨迹上,ForgeWM 在图像质量、参考对齐的运动轮廓一致性、动作准确率和鼠标控制准确率方面领先于评估系统,同时实现了最低的参考 LPIPS;同样的四阶段方案也可迁移到手柄控制的 FPS 游戏玩法。重放时精细化达到了四步参考质量,同时与体验到的轨迹距离约为从噪声再生成的三分之一。这些结果证明了 ForgeWM 在可控少步视频生成方面的有效性。
Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation
中文标题:概念引导:文本到图像生成的精确、无需训练潜在控制
作者:Nikolai R\"ohrich, Isabell Hans, Felix Krause, Bj\"orn Ommer
Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance
文本到图像扩散模型存在两个主要缺陷,严重限制了其实际应用:(1)标准模型缺乏内置的连续概念特定引导机制(例如,用于精确控制图像的美学质量),以及(2)它们在需要高局部一致性的任务(例如生成文本或人类手部)上缺乏可靠性。为了解决这些问题,我们引入了概念层面互信息的新概念,并发现各层之间存在显著的、依赖于概念的差异,这表明特定结构的生成定位于网络的不同部分。我们利用这一洞察,通过强化概念相关层的影响来开发概念引导(Concept Guidance,CoG),这是一种精确的、目标特定的引导方法,可直接应用于现成模型,无需额外训练、外部模型、梯度或提示工程。CoG首先量化每层的概念特定影响,然后使用跳过概念相关层生成的预测的加权组合来引导去噪。我们在各种目标以及PixArt-alpha、SD3、SD3.5和FLUX.1-dev等流行模型上展示了性能提升。代码可访问 https://github.com/CompVis/concept_guidance。
RecipeNet: A Hierarchical Transformer for Recipe Data
中文标题:RecipeNet:用于配方数据的层次化Transformer模型
作者:Pin-Yen Huang, Sachin Chhabra, Prasanth Sai Gouripeddi, Abhinav Kumar, Baoxin Li
Recipe data arises in domains such as materials synthesis, pharmaceutical formulation, and industrial manufacturing, where procedures are represented as ordered sequences of steps containing heterogeneous structured fields. Existing tabular learning methods typically flatten this structure into fixed-schema representations, limiting their ability to capture hierarchical field interactions and procedural dependencies. We propose RecipeNet, a hierarchical Transformer architecture that encodes field-level interactions within each step and sequential dependencies across steps through stacked Transformer encoders. Experiments on multiple recipe datasets and tasks demonstrate that RecipeNet consistently outperforms existing tabular models, highlighting the value of hierarchical and sequential modeling for recipe representation learning.
配方数据出现在材料合成、药物配方和工业制造等领域,其过程被表示为包含异构结构字段的有序步骤序列。现有的表格学习方法通常将这种结构扁平化为固定模式表示,无法有效捕捉层次化字段交互和过程依赖。本文提出RecipeNet,一种层次化Transformer架构,通过堆叠的Transformer编码器对每个步骤内的字段级交互和步骤间的序列依赖进行编码。在多个配方数据集和任务上的实验表明,RecipeNet始终优于现有表格模型,验证了层次化和序列建模对配方表示学习的重要价值。
Learning-to-Transition for Large-scale and High-Order MIMO Detection
中文标题:大规模高阶MIMO检测的学习转迁方法
作者:Yubo Zhang, Yiyao Liu, Xiaodong Wang
High-order multiple-input multiple-output (MIMO) detection requires efficient search over a large discrete symbol space while producing reliable soft information for channel decoding. This paper develops a learning-to-transition (L2T) framework that formulates MIMO detection as a stochastic sequence of complete-vector transitions. At each transition, a channel-coupled Transformer updates both the instance embedding and the sampling policy, while a blockwise autoregressive factorization captures inter-stream dependence with moderate sequential complexity. For hard-output detection, a transition network is applied recursively and trained through a residual-to-BER curriculum, which first learns the MIMO search geometry from the exact residual metric and then aligns the policy with transmitted-bit accuracy. For soft-output reception, the well-trained hard policy is cloned at the parameter level into every layer of an untied soft-input soft-output iterative detection and decoding (IDD) receiver. This tied-to-untied transfer preserves the learned zero-prior search dynamics while enabling layer- and round-specific specialization under decoder feedback. Within each IDD round, decoder priors tilt candidate generation according to Bayes' rule, and likelihood-weighted terminal hypotheses produce posterior and extrinsic log-likelihood ratios for LDPC decoding. A multi-stage training strategy further stabilizes the hard-to-soft transfer by progressively exposing the receiver to synthetic and in-loop decoder-generated priors.
高阶多输入多输出(MIMO)检测需要在庞大的离散符号空间中进行高效搜索,同时产生用于信道解码的可靠软信息。本文提出了一种学习转迁(L2T)框架,将MIMO检测表述为完整向量转迁的随机序列。在每次转迁中,通道耦合Transformer同时更新实例嵌入和采样策略,而分块自回归因子化以适度的顺序复杂度捕获流间依赖性。对于硬输出检测,转迁网络被递归应用,并通过残差到误码率的课程学习进行训练,首先从精确残差度量学习MIMO搜索几何,然后将策略与传输比特精度对齐。对于软输出接收,经过良好训练的硬策略在参数层面被克隆到非耦合的软输入软输出迭代检测解码(IDD)接收机的每一层。这种耦合到非耦合的转迁保留了学习到的零先验搜索动态,同时在解码器反馈下实现了层特定和轮次特定的专细化。在每个IDD轮次中,解码器先验根据贝叶斯规则倾斜候选生成,似然加权的终止假设产生用于LDPC解码的后验和对数似然比外信息。多阶段训练策略通过逐步向接收机呈现合成和环内解码器生成的先验,进一步稳定了硬到软的转迁过程。
Marionette: Predicting World States, Rendering Geometry, Painting Appearance
中文标题:Marionette:预测世界状态、渲染几何、绘制外观
作者:Zian Meng, Zhen Li, Chuanhao Li, Qiang Li, Kaipeng Zhang
Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.
交互式游戏世界模型通常直接在像素或潜在空间中对视觉观察进行自回归,将姿态、几何和遮挡等结构化属性隐式地由同一生成序列维护。在长时域下,这些潜在世界属性的误差会累积,导致一致性和可控性变得脆弱。我们显式地建模不断演化的世界状态,将精确的几何计算委托给一个固定的零参数渲染器,仅让神经网络模型负责合成外观。我们将这一思想实例化为Marionette,一个用于包含关节角色的交互式游戏的世界模型。首先,两阶段自回归动力学模型预测一个显式且可解释的276维3D世界状态,包含多实体关节骨骼、度量根轨迹和旋转。其次,零参数图形桥将预测的状态转换为姿态控制视频,以闭式形式计算世界空间几何和遮挡。第三,控制条件视频扩散观察模型从生成的结构化控制中合成逼真的RGB观察。我们的实验验证了Marionette的两个特性。首先,预测的世界状态是直接可控的。强制执行不匹配的动作流会使根对齐关节误差在48个保留段上改变31%。其次,长时域行为由状态决定,并可以在状态层面修复。在自由情况下,两个生成的角色会漂移至21.2米(录制会话保持在5米左右),且三分之一的帧显示地面穿透。在显式状态上施加两条规则——地形碰撞体和分离上限——将穿透减少66%并保持角色交互,而无需改变观察模型。通过预测状态传递外观不会造成可检测的保真度损失,在FVD指标上为831,而录制姿态为799。
WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA
中文标题:WAM-Diff2: 用于高效自动驾驶VLA的分层自回归到扩散蒸馏方法
作者:Zhihao Zhu, Hanlin Shang, Mingwang Xu, Feipeng Cai, Zhuolin He, Yaoyi Li, Jianhua Han, Hang Xu, Siyu Zhu
Vision-Language-Action (VLA) models have emerged as a prominent paradigm for end-to-end autonomous driving; however, their efficient deployment is severely constrained by high computational latency and exposure bias arising from sequential autoregressive decoding. Conversely, while specialized diffusion policies enable low-latency, parallel execution, training them from scratch typically yields narrow, single-task architectures that lack holistic visual-linguistic reasoning. Successfully transforming pre-trained autoregressive generalists into parallel diffusion models could combine multi-task cognitive intelligence with execution efficiency, yet this transition presents a formidable architectural challenge due to mismatched attention patterns (causal versus bidirectional) and divergent optimization objectives. To bridge this divide, we introduce WAM-Diff2, a multi-task discrete diffusion VLA framework powered by a three-stage hierarchical distillation strategy. By structuring the architectural shift through progressive block-wise adaptation, block-wise distillation, and model-wise cross-scale distillation, WAM-Diff2 preserves the underlying semantic foundations of the base model while accelerating inference. Extensive evaluations across driving understanding, perception, and planning benchmarks demonstrate that WAM-Diff2 effectively mitigates exposure bias and achieves performance parity with autoregressive baselines. Crucially, the autoregressive-to-diffusion transition yields a 2.8x decoding speedup, which scales to an ultimate 15.1x acceleration when combined with system-level optimizations including FlashInfer and CUDA Graphs.
视觉语言动作(VLA)模型已成为端到端自动驾驶的主流范式;然而,其高效部署受到高计算延迟和顺序自回归解码带来的曝光偏差的严重制约。相反,虽然专门的扩散策略能够实现低延迟并行执行,但从零开始训练通常会产生狭窄的单任务架构,缺乏全面的视觉语言推理能力。将预训练的自回归通用模型成功转换为并行扩散模型可以结合多任务认知智能与执行效率,但由于注意力模式(因果注意力与双向注意力)不匹配以及优化目标存在差异,这一转变面临着严峻的架构挑战。为弥补这一差距,我们提出了WAM-Diff2,一个由三阶段分层蒸馏策略驱动的多任务离散扩散VLA框架。通过渐进式逐块适配、块式蒸馏和模型级跨尺度蒸馏来组织架构转变,WAM-Diff2在加速推理的同时保留了基础模型的底层语义基础。在驾驶理解、感知和规划基准测试中的广泛评估表明,WAM-Diff2有效缓解了曝光偏差,并实现了与自回归基线相当的性能。关键的是,自回归到扩散的转变带来了2.8倍的解码加速,当结合FlashInfer和CUDA Graphs等系统级优化时,最终可扩展至15.1倍的加速。
MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation
中文标题:MusicLayout:用于可控文本到音乐生成的显式结构规划
作者:Shuyu Li, Kejun Zhang, Jiahe Lei, Shulei Ji, Zihao Wang, Jiaxing Yu, Wanying Wu, Lei Wang
Text-to-music generation has advanced rapidly, but current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. To address this issue, we introduce MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music. We integrate MusicLayout into a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control. We evaluate MusicLayout through layout-conditioned generation, layout manipulation experiments, and matched-data ablations, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control. We have released the implementation as open source on GitHub at https://github.com/XaryLee/MusicLayout.
文本到音乐生成技术发展迅速,但当前系统仍主要依赖全局文本提示,导致生成音乐的结构组织隐含难以前置检查、控制或在音频生成前修改。为解决这一问题,我们提出MusicLayout,一种用于在文本到音乐生成中控制音乐结构的显式中间表示。MusicLayout将音乐作品描述为段落、织体、重复、变奏及乐器级编排的时间对齐布局,作为文本意图与生成音乐之间的可解释规划层。我们将MusicLayout整合到一个基于统一自回归公式的文本到音乐框架中,模型首先生成MusicLayout表示,然后在该表示的条件下预测同一序列中的音频标记。生成的MusicLayout可在音频生成前进行检查和修改,提供布局级结构控制机制。我们通过布局条件生成、布局操控实验和匹配数据消融实验对MusicLayout进行评估,证明了显式布局规划能够改善长程结构组织并支持布局级控制。我们已将实现代码开源发布于GitHub:https://github.com/XaryLee/MusicLayout
PROVE: Training-Free Prompt Recovery using Verifiable Evidence
中文标题:PROVE:使用可验证证据的免训练提示词恢复
作者:Rupayan Mallick, Mahsa Khoshnoodi, Sarah Adel Bargal
Modern text-to-image models can generate highly realistic images from natural-language prompts, while recent advances in prompt inversion have made it increasingly feasible to recover those prompts from generated outputs, raising new concerns for copyright protection and content ownership. As prompt marketplaces emerge, recovered prompts can enable both the unauthorized reproduction and redistribution of copyrighted creative works, and the exposure of the prompts that encode an artist's creative recipe in AI-generated content. Existing prompt inversion methods rely on gradient-based optimization, autoregressive captioning, or reinforcement learning. However, optimization-based methods often produce unreadable prompts, captioning methods hallucinate unverified details, and RL-based approaches frequently overfit to specific generators while introducing evaluation circularity. We introduce PROVE (Prompt Recovery with Verified Evidence), a training-free, black-box prompt inversion attack that reconstructs prompts by composing verifiable scene descriptions rather than optimizing token sequences, targeting both original copyrighted works and AI-generated content. The resulting prompts are fully auditable, with every recovered claim grounded in explicit image evidence, and are formalized through a precision-constrained recall maximization objective. Across MS-COCO, Flickr30K, and Lexica, using state-of-the-art text-to-image generators, PROVE consistently outperforms optimization, captioning, and RL-based baselines on image similarity (DINO, LPIPS) and text-image alignment (CLIP), without any training, generator access, or fine-tuning, demonstrating a stronger and more practical prompt inversion attack.
现代文本到图像模型可以根据自然语言提示词生成高度逼真的图像,而近期提示词反演技术的进展使得从生成图像中恢复这些提示词越来越可行,这引发了关于版权保护和内容所有权的新担忧。随着提示词市场的出现,恢复的提示词不仅可能导致受版权保护创意作品的未经授权复制和再分发,还可能暴露AI生成内容中编码艺术家创作配方的提示词。现有的提示词反演方法依赖于基于梯度的优化、自回归描述生成或强化学习。然而,基于优化的方法通常产生难以理解的提示词,描述方法会虚构未经验证的细节,而强化学习方法常过度拟合特定生成器并引入评估循环性。我们提出PROVE(可验证证据提示词恢复),这是一种无需训练的黑盒提示词反演攻击,通过组合可验证的场景描述而非优化token序列来重构提示词,针对原始版权作品和AI生成内容。生成的提示词完全可审计,每项恢复声明都基于明确的图像证据,并通过精度约束的召回最大化目标进行形式化。在MS-COCO、Flickr30K和Lexica数据集上,使用最先进的文本到图像生成器,PROVE在图像相似度(DINO、LPIPS)和文本-图像对齐(CLIP)方面持续优于基于优化、描述和强化的基线方法,且无需任何训练、生成器访问或微调,展示了更强大且更实用的提示词反演攻击能力。
Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Visual Avatar Generation
中文标题:Omni-LiveAvatar: 分钟级实时流式联合音视频虚拟形象生成
作者:Lunjie Zhu, Xingtong Ge, Fangyu Lin, Yi Zhang, Zhening Liu, Mengfei Li, Yumeng Zhang, Guanglu Song, Yu Liu, Jun Zhang
Joint audio-video generative models serve as foundation for immersive and interactive digital-human generation. Nevertheless, most existing models rely on bidirectional attention and multi-step denoising and can generate only short clips, making them unsuitable for real-time interaction over extended durations. We present Omni-LiveAvatar, the first framework for minute-level, real-time streaming joint audio-video avatar generation. Specifically, we propose (1) a progressive autoregressive distillation pipeline that transfers a large bidirectional joint audio-video diffusion model into a few-step autoregressive generator without auxiliary stabilization mechanisms; (2) a synchronized audio-video long-short-term memory that preserves global consistency under a bounded memory budget; and (3) a hierarchical rolling prompt planning strategy that enables coherent semantic evolution and seamless prompt transitions. Extensive experiments show that Omni-LiveAvatar generates high-quality, synchronized minute-level avatars in real time. In terms of speed, it achieves a 33$\times$ generation speedup over its teacher, LTX-2, on a single NVIDIA H200 GPU; in terms of generation quality, it outperforms accelerated baselines across visual quality, audio quality, cross-modal synchronization, and human fidelity. Our code is available at https://github.com/Aoko955/Omni-LiveAvatar.
联合音视频生成模型是实现沉浸式和交互式数字人生成的基础。然而,现有大多数模型依赖双向注意力和多步去噪机制,仅能生成短视频片段,因而不适用于长时间实时交互场景。本文提出Omni-LiveAvatar,这是首个支持分钟级实时流式联合音视频虚拟形象生成的框架。具体而言,我们提出(1)一个渐进式自回归蒸馏管道,能够将大型双向联合音视频扩散模型迁移为少步自回归生成器,且无需辅助稳定机制;(2)一个同步音视频长短期记忆模块,可在有限内存预算下保持全局一致性;(3)一个分层滚动提示规划策略,实现连贯的语义演化和无缝的提示转换。大量实验表明,Omni-LiveAvatar能够实时生成高质量、同步的分钟级虚拟形象。在速度方面,相比其教师模型LTX-2在单块NVIDIA H200 GPU上实现了33倍的生成加速;在生成质量方面,其在视觉质量、音频质量、跨模态同步和人类保真度等指标上均优于加速基线模型。代码已开源于https://github.com/Aoko955/Omni-LiveAvatar。
InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Driving
中文标题:InfiniVerse:面向自动驾驶的占用引导无界场景生成
作者:Xiaoyu Ye, Leheng Li, Xinyu Ji, Yingjie Cai, Hongda He, Xu Yan, Guanyi Zhao, Ying-Cong Chen, Bingbing Liu, Shuguang Cui, Zhen Li
Generating realistic, controllable, and temporally coherent urban environments is a critical yet unresolved challenge in the autonomous driving community. In this paper, we introduce InfiniVerse, a unified pipeline for long-range, 2D-3D-aligned, and controllable synthesis of dynamic urban scenes from a single frame. In practice, our approach first reconstructs a 3D occupancy representation from the input multi-view frame. This representation serves as a foundation for autoregressive scene extension along arbitrary trajectories. Subsequently, a video diffusion model translates the coarse occupancy grid into realistic, spatiotemporally consistent video sequences. Moreover, we propose a hierarchical sketch-and-refine paradigm, in which the generated videos are re-projected as image-conditioned feedback to enhance the 3D occupancy representation, establishing cross-modal alignment and mutual enhancement between the visual and spatial domains. Extensive evaluations on the Waymo Open Dataset and nuScenes demonstrate that InfiniVerse achieves state-of-the-art performance, with a FID of 6.4 and FVD of 67.97, significantly outperforming existing benchmarks in both duration and stability.
生成真实、可控且时间一致的城市环境是自动驾驶领域的一项关键而未解决的挑战。本论文提出了 InfiniVerse,一个统一的 pipeline,能够从单帧图像实现动态城市场景的长距离、2D-3D对齐且可控的合成。具体而言,我们的方法首先从输入的多视角帧图像中重建 3D 占用表示,以此作为沿任意轨迹进行自回归场景扩展的基础。随后,视频扩散模型将粗糙的占用网格转换为真实且时空一致的视频序列。此外,我们提出了层次化草图-精化范式,将生成的视频重投影为图像条件反馈,以增强 3D 占用表示,从而建立视觉与空间域之间的跨模态对齐和相互增强。在 Waymo Open Dataset 和 nuScenes 数据集上的广泛评估表明,InfiniVerse 取得了领先性能,FID 达到 6.4,FVD 达到 67.97,在持续时间和稳定性方面均显著优于现有基准方法。
今日 Diffusion 领域的研究呈现多元化发展态势,主要集中在模型控制与优化、应用场景拓展和安全对齐三个方向。在模型控制方面,免训练 latent 控制和 Adapter 组合技术成为热点,显示出精细化控制生成的需求增长。强化学习与扩散模型的结合继续深入,步级奖励机制和偏好对齐方法为模型微调提供了新思路。应用层面涵盖图像修复、超分辨率、虚拟形象生成、自动驾驶场景生成等多领域,体现扩散模型在视觉内容创作和物理模拟中的广泛应用。同时,安全相关研究聚焦于概念遗忘和越狱攻击防御,表明学界对模型安全性的重视。推理效率优化(如缓存机制、蒸馏技术)也是值得关注的技术趋势。
- Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation - 提出了免训练的精确潜在控制方法,为文生图提供了一种灵活且无需额外训练的控制机制,对于实际应用具有重要价值。
- CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning - 创新性地引入步级奖励机制,无需额外训练即可实现基于强化学习的扩散模型微调,为模型优化提供了高效新范式。
- BAG: Budget-Aware Gating for Diffusion Caching - 针对扩散模型推理开销大的问题,提出了预算感知的门控缓存机制,在计算资源受限场景下具有重要的实用意义。
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View - 从统一路径空间视角设计强化学习方法,为扩散模型的 RL 训练提供了理论框架和系统指导。
- Concept Unlearning by Modeling Key Steps of Diffusion Process - 通过建模扩散过程关键步骤实现概念遗忘,为模型安全性和可控性提供了新的技术路径。
SDO: Subspace Deconflicting Operator for Multi-Adapter Composition
中文标题:SDO:用于多适配器组合的子空间解冲突算子
作者:Zhongsheng Wang, Zhedong Lin, Qian Liu, Xinyu Zhang, Jiamou Liu
Composing independently trained adapters within a shared diffusion backbone provides a modular approach to multi-character generation, but naive joint deployment often causes identity mixing, cross-character attribute leakage, and unstable scene composition. We study this interference from a parameter-space perspective and hypothesize that it arises partly from conflicts between overlapping dominant subspaces in shared layers. To address this issue, we propose \textbf{SDO}, a \textbf{S}ubspace \textbf{D}econflicting \textbf{O}perator for multi-adapter composition. SDO reconstructs layer-wise low-rank updates from the selected adapters, extracts compact subspace signatures, measures pairwise conflict through output-subspace overlap, and applies a permutation-equivariant transformation that suppresses harmful shared directions while retaining identity-specific characteristics. The resulting representations are mapped back to standard adapter updates and can be directly incorporated into existing diffusion inference pipelines. Experiments demonstrate that SDO consistently improves identity fidelity and compositional stability, with particularly clear gains as the number of jointly composed adapters increases.
在共享扩散骨干网络中组合独立训练的适配器为多角色生成提供了一种模块化方法,但朴素联合部署往往导致身份混合、跨角色属性泄露和场景组合不稳定。我们从参数空间角度研究这种干扰,并假设其部分源于共享层中重叠主导子空间之间的冲突。为解决此问题,我们提出SDO(子空间解冲突算子)。SDO从选定适配器重建逐层低秩更新,提取紧凑子空间签名,通过输出子空间重叠度量成对冲突,并应用置换等变变换来抑制有害的共享方向,同时保留身份特定特征。所得表示映射回标准适配器更新,可直接集成到现有扩散推理流程中。实验表明,SDO始终如一地提升身份保真度和组合稳定性,且随着联合组合适配器数量的增加,改进效果尤为显著。
Program-space Diffusion for Morphology-to-Transcriptomics Prediction
中文标题:用于形态学-转录组学预测的程序空间扩散模型
作者:Ruyter Swann, Dorent Reuben, Racoceanu Daniel
Spatial transcriptomics (ST) enables genome-wide gene expression profiling while preserving tissue architecture, but its cost and limited scalability remain major bottlenecks. This has motivated models that predict spatial expression directly from routine histology. Despite promising results, most existing approaches operate at the gene level without leveraging established transcriptomic modeling practices and rely on heterogeneous gene selection strategies, which complicates fair comparison across methods. We propose to reformulate morphology-to-transcriptomics prediction as conditional generation in transcriptional program space, thereby exploiting coordinated transcriptional variation instead of predicting genes independently. Using consensus non-negative matrix factorization (cNMF), we extract a low-dimensional set of transcriptional programs capturing coordinated expression variation in the training data, and train a conditional diffusion model to generate program activations from histology. This formulation exploits coordinated transcriptional variation and substantially lowers the dimensionality of the conditional generative task.
空间转录组学能够在保留组织结构的同时实现全基因组基因表达谱分析,但其成本高昂和可扩展性有限仍然是主要瓶颈。这促使学界开发能够直接从常规组织学预测空间表达的模型。尽管现有方法取得了令人鼓舞的成果,但大多数方法在基因水平上运作,未能利用已建立的转录组学建模方法,且依赖于异质性的基因选择策略,这使得不同方法之间的公平比较变得复杂。我们提出将形态学-转录组学预测重新表述为转录程序空间中的条件生成任务,从而利用协调的转录变异而非独立预测基因。我们使用一致性非负矩阵分解(cNMF)从训练数据中提取低维转录程序集,以捕捉协调表达变异,并训练条件扩散模型从组织学图像生成程序激活值。该方法充分利用了协调转录变异,并显著降低了条件生成任务的维度。
Secret-Stego Dissimilarity as a Design Axis: Invertible Coverless Image Steganography with Diffusion Models
中文标题:秘密-隐写图像不相似性作为设计轴:基于扩散模型的可逆无载体图像隐写术
作者:Hongxin Xu, Jianping Mei, Can Wang, Defang Chen
Coverless image steganography (CIS) synthesizes a stego image rather than modifying an existing cover image, enabling authorized recipients to reconstruct the original secret image from the stego. Existing diffusion-based CIS methods can generate natural-looking stego images but preserve substantial visual similarity to the secret image. This resemblance risks exposing structural and semantic cues, giving rise to security vulnerabilities that cannot be evaluated solely via recovery fidelity. Achieving substantial visual dissimilarity between the secret and stego images without compromising stego quality and recovery fidelity remains challenging. To address this issue, we propose InvCISD, an invertible diffusion framework that couples the latent representations of the secret and an irrelevant reference image with an invertible network called LIMNet. We first train LIMNet in diffusion latent space, followed by end-to-end fine-tuning of the entire network, i.e., LIMNet integrated diffusion inversion and generation modules. Experiments demonstrate that the proposed method substantially reduces secret-stego visual similarity, improves stego quality, and retains satisfactory secret reconstruction quality. Our further investigation shows that all evaluated methods are highly detectable by the CIS-oriented steganalysis model, indicating that resistance against targeted steganalysis constitutes a critical direction for future CIS research.
无载体图像隐写术(CIS)生成隐写图像而非修改现有载体图像,使授权接收者能够从隐写图像中重建原始秘密图像。现有基于扩散的CIS方法可以生成自然逼真的隐写图像,但保留了与秘密图像的大量视觉相似性。这种相似性存在暴露结构和语义线索的风险,带来仅凭恢复保真度无法评估的安全漏洞。在不损害隐写质量和恢复保真度的前提下,实现秘密图像与隐写图像之间的大幅视觉不相似性仍具挑战性。为解决这一问题,我们提出InvCISD,一种可逆扩散框架,使用名为LIMNet的可逆网络将秘密图像和无关参考图像的潜在表示耦合起来。我们首先在扩散潜空间中训练LIMNet,随后对整个网络进行端到端微调,即LIMNet集成扩散反演和生成模块。实验表明,所提方法大幅降低了秘密-隐写图像的视觉相似性,提高了隐写质量,并保持了令人满意的秘密重建质量。进一步研究表明,所有评估方法均可被面向CIS的隐写分析模型高度检测,表明抵抗针对性隐写分析是未来CIS研究的关键方向。
Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation
中文标题:概念引导:文本到图像生成的精确、无需训练潜在控制
作者:Nikolai R\"ohrich, Isabell Hans, Felix Krause, Bj\"orn Ommer
Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance
文本到图像扩散模型存在两个主要缺陷,严重限制了其实际应用:(1)标准模型缺乏内置的连续概念特定引导机制(例如,用于精确控制图像的美学质量),以及(2)它们在需要高局部一致性的任务(例如生成文本或人类手部)上缺乏可靠性。为了解决这些问题,我们引入了概念层面互信息的新概念,并发现各层之间存在显著的、依赖于概念的差异,这表明特定结构的生成定位于网络的不同部分。我们利用这一洞察,通过强化概念相关层的影响来开发概念引导(Concept Guidance,CoG),这是一种精确的、目标特定的引导方法,可直接应用于现成模型,无需额外训练、外部模型、梯度或提示工程。CoG首先量化每层的概念特定影响,然后使用跳过概念相关层生成的预测的加权组合来引导去噪。我们在各种目标以及PixArt-alpha、SD3、SD3.5和FLUX.1-dev等流行模型上展示了性能提升。代码可访问 https://github.com/CompVis/concept_guidance。
Marionette: Predicting World States, Rendering Geometry, Painting Appearance
中文标题:Marionette:预测世界状态、渲染几何、绘制外观
作者:Zian Meng, Zhen Li, Chuanhao Li, Qiang Li, Kaipeng Zhang
Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.
交互式游戏世界模型通常直接在像素或潜在空间中对视觉观察进行自回归,将姿态、几何和遮挡等结构化属性隐式地由同一生成序列维护。在长时域下,这些潜在世界属性的误差会累积,导致一致性和可控性变得脆弱。我们显式地建模不断演化的世界状态,将精确的几何计算委托给一个固定的零参数渲染器,仅让神经网络模型负责合成外观。我们将这一思想实例化为Marionette,一个用于包含关节角色的交互式游戏的世界模型。首先,两阶段自回归动力学模型预测一个显式且可解释的276维3D世界状态,包含多实体关节骨骼、度量根轨迹和旋转。其次,零参数图形桥将预测的状态转换为姿态控制视频,以闭式形式计算世界空间几何和遮挡。第三,控制条件视频扩散观察模型从生成的结构化控制中合成逼真的RGB观察。我们的实验验证了Marionette的两个特性。首先,预测的世界状态是直接可控的。强制执行不匹配的动作流会使根对齐关节误差在48个保留段上改变31%。其次,长时域行为由状态决定,并可以在状态层面修复。在自由情况下,两个生成的角色会漂移至21.2米(录制会话保持在5米左右),且三分之一的帧显示地面穿透。在显式状态上施加两条规则——地形碰撞体和分离上限——将穿透减少66%并保持角色交互,而无需改变观察模型。通过预测状态传递外观不会造成可检测的保真度损失,在FVD指标上为831,而录制姿态为799。
An InSAR Phase Unwrapping Framework for Large-scale and Complex Events
中文标题:面向大尺度复杂事件的InSAR相位解缠框架
作者:Yijia Song, Juliet Biggs, Alin Achim, Robert Popescu, Simon Orrego, Nantheera Anantrasirichai
Phase unwrapping remains a critical and challenging problem in InSAR processing, particularly in scenarios involving complex deformation patterns. In earthquake-related deformation, shallow sources can generate surface-breaking faults and abrupt displacement discontinuities, which severely disrupt phase continuity and often cause conventional unwrapping algorithms to fail. Another limitation of existing learning-based unwrapping methods is their reliance on fixed and relatively small input sizes, while real InSAR interferograms are typically large-scale and spatially heterogeneous. This mismatch restricts the applicability of many neural network approaches to real-world data. In this work, we present a phase unwrapping framework based on a diffusion model, developed to process large-scale interferograms and to address phase discontinuities caused by deformation. By leveraging a diffusion model architecture, the proposed method can recover physically consistent unwrapped phase fields even in the presence of fault-related phase jumps. Experimental results on both synthetic and real datasets demonstrate that the method effectively addresses discontinuities associated with near-surface deformation and scales well to large InSAR images, offering a practical alternative to manual unwrapping in challenging scenarios.
相位解缠在InSAR处理中仍然是一个关键且具有挑战性的问题,特别是在涉及复杂形变模式的场景中。在地震相关形变中,浅源可以产生地表破裂断层和急剧位移不连续性,这严重破坏了相位连续性,往往导致传统解缠算法失效。现有的基于学习的解缠方法的另一个局限性在于其依赖于固定且相对较小的输入尺寸,而真实的InSAR干涉图通常是大尺度和空间异质的。这种不匹配限制了许多神经网络方法在真实数据中的适用性。本研究提出了一种基于扩散模型的相位解缠框架,旨在处理大尺度干涉图并解决由形变引起的相位不连续问题。通过利用扩散模型架构,即使存在断层相关的相位跳变,所提出的方法也能恢复物理上一致的解缠相位场。在合成数据集和真实数据集上的实验结果表明,该方法有效地解决了与浅地表形变相关的不连续性问题,并能够很好地扩展到大尺寸InSAR图像,在具有挑战性的场景中为人工解缠提供了一种实用的替代方案。
WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA
中文标题:WAM-Diff2: 用于高效自动驾驶VLA的分层自回归到扩散蒸馏方法
作者:Zhihao Zhu, Hanlin Shang, Mingwang Xu, Feipeng Cai, Zhuolin He, Yaoyi Li, Jianhua Han, Hang Xu, Siyu Zhu
Vision-Language-Action (VLA) models have emerged as a prominent paradigm for end-to-end autonomous driving; however, their efficient deployment is severely constrained by high computational latency and exposure bias arising from sequential autoregressive decoding. Conversely, while specialized diffusion policies enable low-latency, parallel execution, training them from scratch typically yields narrow, single-task architectures that lack holistic visual-linguistic reasoning. Successfully transforming pre-trained autoregressive generalists into parallel diffusion models could combine multi-task cognitive intelligence with execution efficiency, yet this transition presents a formidable architectural challenge due to mismatched attention patterns (causal versus bidirectional) and divergent optimization objectives. To bridge this divide, we introduce WAM-Diff2, a multi-task discrete diffusion VLA framework powered by a three-stage hierarchical distillation strategy. By structuring the architectural shift through progressive block-wise adaptation, block-wise distillation, and model-wise cross-scale distillation, WAM-Diff2 preserves the underlying semantic foundations of the base model while accelerating inference. Extensive evaluations across driving understanding, perception, and planning benchmarks demonstrate that WAM-Diff2 effectively mitigates exposure bias and achieves performance parity with autoregressive baselines. Crucially, the autoregressive-to-diffusion transition yields a 2.8x decoding speedup, which scales to an ultimate 15.1x acceleration when combined with system-level optimizations including FlashInfer and CUDA Graphs.
视觉语言动作(VLA)模型已成为端到端自动驾驶的主流范式;然而,其高效部署受到高计算延迟和顺序自回归解码带来的曝光偏差的严重制约。相反,虽然专门的扩散策略能够实现低延迟并行执行,但从零开始训练通常会产生狭窄的单任务架构,缺乏全面的视觉语言推理能力。将预训练的自回归通用模型成功转换为并行扩散模型可以结合多任务认知智能与执行效率,但由于注意力模式(因果注意力与双向注意力)不匹配以及优化目标存在差异,这一转变面临着严峻的架构挑战。为弥补这一差距,我们提出了WAM-Diff2,一个由三阶段分层蒸馏策略驱动的多任务离散扩散VLA框架。通过渐进式逐块适配、块式蒸馏和模型级跨尺度蒸馏来组织架构转变,WAM-Diff2在加速推理的同时保留了基础模型的底层语义基础。在驾驶理解、感知和规划基准测试中的广泛评估表明,WAM-Diff2有效缓解了曝光偏差,并实现了与自回归基线相当的性能。关键的是,自回归到扩散的转变带来了2.8倍的解码加速,当结合FlashInfer和CUDA Graphs等系统级优化时,最终可扩展至15.1倍的加速。
Multiphase-Diff: Diffusion-Based Generative Modeling for High-Contrast Multiphase Physical Systems with Sharp Interfaces
中文标题:Multiphase-Diff:基于扩散的高对比度锐利界面多相物理系统生成式建模
作者:Yining Huang, Zhenyu Liang
Physics-constrained diffusion for high-contrast, sharp-interface multiphase fields faces three coupled difficulties. At coefficient jumps, expanded pointwise strong-form PDE residuals contain singular gradient terms that can penalize physical interfaces. Under extreme contrast, low-magnitude phases may fall below the diffusion noise floor and be erased, misscaled, or generated with negative coefficients, while a global likelihood scale allows high-magnitude phases to dominate supervision. We therefore propose Multiphase-Diff, which makes three corresponding contributions: (i) a conservative flux residual that avoids differentiating discontinuous coefficients and enforces discrete conservation; (ii) an analytic bijective representation that maps low-amplitude signals to order-one latent scales and guarantees coefficient positivity through exponential decoding; and (iii) a Jacobi-preconditioned likelihood that normalizes local residual scales for balanced supervision. Experiments on three complementary multiphase benchmarks demonstrate the superiority of Multiphase-Diff over seven baselines in both physical and distributional fidelity and its robustness across phase contrasts and compositions, establishing its effectiveness for scientific sample generation in this challenging regime.
物理约束扩散方法在高对比度、锐利界面的多相场建模中面临三个相互耦合的难题。在系数跳变点,展开的点态强形式PDE残差包含可能惩罚物理界面的奇异梯度项。在极端对比度下,低幅值相可能降至扩散噪声底以下而被擦除、错误缩放或生成负系数,而全局似然尺度使得高幅值相主导监督过程。因此,本文提出Multiphase-Diff,做出三方面贡献:(i) 守恒通量残差,避免对不连续系数求导并强制离散守恒;(ii) 解析双射表示,将低幅值信号映射到阶一潜在尺度,并通过指数解码保证系数正性;(iii) 雅可比预条件似然,对局部残差尺度进行归一化以实现平衡监督。在三个互补多相基准数据集上的实验表明,Multiphase-Diff在物理和分布保真度方面显著优于七种基线方法,并在不同的相对比度和组成下表现出强大的鲁棒性,验证了其在这种挑战性场景下进行科学样本生成的有效性。
Limitations of Synthetic Data Generation in Specialized Data-Scarce Domains
中文标题:合成数据生成在专业数据稀缺领域的局限性
作者:Edward Zhang, Marcel Hussing, Tanay Tandon, Shenbagaraj Kannapiran, Jason Hughes, Youkang Wang, Joshua Caswell, Agelos Kratimenos, Yi Fan Li, Milan Manoj, Ethan Sanchez, Sumukh Shrote, Camillo Jose Taylor, Daniel A. Hashimoto, Eric Eaton
Advances in diffusion-based generative models have motivated the use of synthetic image generation to alleviate data scarcity in vision tasks. While this strategy has shown promise in natural image benchmarks such as ImageNet, its effectiveness in sparse, high-variance real-world domains remains unclear. In this work, we focus on domains where images differ substantially from common image datasets and additional data are expensive to obtain. Against non-generative data augmentation baselines, we evaluate the downstream classifier performance improvements yielded by two schools of generative sparse data extension: distribution modeling and sample perturbation. Across five trauma classification tasks using subject-wise train--validation splits, no generative approach consistently outperforms a strong non-generative baseline. Feature-space analysis reveals recurring failure modes: memorization or collapse, distributional drift, and generation of visually plausible but simplified canonical instances that are easier to classify than real data.
扩散生成模型的进展促使人们利用合成图像生成来缓解视觉任务中的数据稀缺问题。虽然该策略在ImageNet等自然图像基准测试中表现出前景,但其在稀疏、高方差的现实世界领域中的有效性仍不清楚。在本工作中,我们聚焦于图像与常见图像数据集差异显著且获取额外数据成本高昂的领域。针对非生成式数据增强基线,我们评估了两类生成式稀疏数据扩展方法——分布建模和样本扰动——所带来的下游分类器性能提升。使用受试者级别的训练-验证划分对五个创伤分类任务进行评估,结果显示没有任何生成式方法能持续超越一个强大的非生成式基线。特征空间分析揭示了反复出现的失败模式:记忆化或坍缩、分布漂移,以及生成在视觉上可信但比真实数据更容易分类的简化典型实例。
RGBX-Next: Towards Realistic Generative Rendering from G-Buffers
中文标题:RGBX-Next: 基于G-Buffer的逼真生成式渲染研究
作者:Zheng Zeng, Marco Salvi, Lifan Wu, Jan Nov\'ak, Daqi Lin, Saeed Hadadan, Yichen Sheng, Robert Pottorff, Shiqiu Liu, Ravi Ramamoorthi, Ling-Qi Yan, Milo\v{s} Ha\v{s}an
Diffusion models have achieved impressive results in image, video, and streaming generation. However, compared to traditional 3D rendering, they still lack precise control over the generated output. We believe a viable path forward is to use generative models as learned renderers conditioned on traditionally rendered G-buffers. We introduce RGBX-Next, a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Our key contribution is a general recipe for finetuning diffusion transformer (DiT) models into generative forward and inverse renderers. We show that the resulting models achieve high quality in both realistic generative rendering and intrinsic decomposition. We will make all our models publicly available. We believe that the design principles presented in this paper will benefit future research on controllable generative forward and inverse rendering.
扩散模型在图像、视频和流媒体生成方面取得了令人瞩目的成果。然而,与传统三维渲染相比,它们仍然缺乏对生成输出的精确控制。我们认为一个可行的研究方向是将生成模型用作基于传统渲染G-Buffer进行条件学习的渲染器。我们提出了RGBX-Next,这是一个统一的生成式正向和逆向渲染框架,能够从图像、视频和流媒体中估计G-Buffer,并从G-Buffer渲染出逼真的图像、视频和流媒体。我们的关键贡献是提出了一种通用的方法,用于将扩散变换器(DiT)模型微调为生成式正向和逆向渲染器。实验表明,所得到的模型在逼真生成渲染和内在分解任务中均达到了高质量水平。我们将公开发布所有模型。我们相信本文提出的设计原则将有助于未来可控生成式正向和逆向渲染的研究。
Source-Agnostic Image Translation Based on Latent Aware Adaptive Masking
中文标题:基于潜在感知自适应掩码的源无关图像翻译
作者:Tomislav Dobri\v{c}ki, Byung-Woo Hong
In this work, we propose a source-agnostic framework that dynamically refines a binary mask throughout the reverse diffusion process by computing the discrepancies of a pretrained diffusion model's prediction for each latent time step. Rather than relying on a fixed threshold, our method introduces a time-dependent statistical thresholding scheme derived from the empirical mean and standard deviation of prediction discrepancies across the latent noisy images from the target distribution. This allows the mask to adapt to the model&x27;s varying predictive confidence at different noise levels, effectively isolating domain-specific regions while preserving global structural coherence. Experimental results on the AFHQ and Celeba-HQ datasets demonstrate that our approach outperforms state-of-the-art unsupervised Image-to-Image methods in both realism (FID, KID) and faithfulness (SSIM, LPIPS). By requiring only a pretrained model of the target domain, our approach enables precise, automated localization and seamless translation across diverse source distributions without any specialized training. The project source code is available at: https://github.com/dtoma95/PM-Edit
在本工作中,我们提出了一个源无关框架,该框架通过计算预训练扩散模型对每个潜在时间步的预测差异,在逆向扩散过程中动态优化二值掩码。我们的方法引入了一种依赖于时间的统计阈值方案,该方案源自目标分布潜在噪声图像预测差异的经验均值和标准差。这使得掩码能够适应模型在不同噪声水平下的预测置信度变化,有效隔离领域特定区域,同时保留全局结构一致性。在AFHQ和Celeba-HQ数据集上的实验结果表明,我们的方法在逼真度(FID、KID)和保真度(SSIM、LPIPS)方面均优于最先进的无监督图像到图像翻译方法。由于仅需要目标域的预训练模型,我们的方法能够实现精确的自动化定位,并在无需任何专门训练的情况下跨不同源分布进行无缝翻译。项目源代码可见于:https://github.com/dtoma95/PM-Edit
MagnifiQ: Patch-aware Text Guided Progressive Upscaling for High-Resolution Image Restoration
中文标题:MagnifiQ: 面向高分辨率图像修复的块感知文本引导渐进式放大方法
作者:Mahesh Reddy, Yashesh Savani, Antoine Mercier, Hong Cai, Fatih Porikli, Guillaume Berger
High-resolution image restoration from degraded inputs is challenging because it must preserve global structural consistency while recovering fine-grained local details, especially at 4K resolution where direct diffusion-based restoration is computationally expensive and prone to repeated or inconsistent textures. In this work, we introduce MagnifiQ, an image restoration framework that progressively upscales and restores images across resolutions, e.g., from 1024x1024 to 4096x4096. Our approach leverages a pre-trained text-to-image diffusion model such as SDXL and adapts it for more scalable high-resolution inference by replacing its original self-attention layers with convolutional operations whose computational cost grows linearly with image resolution. We further propose a progressive upscaling strategy that iteratively restores images over multiple resolution stages, refining each intermediate output rather than directly hallucinating the final 4K image, thereby improving global coherence and reducing high-resolution artifacts. To enhance local details while controlling content drift, MagnifiQ uses patch-specific text prompts that provide spatially localized semantic guidance during restoration. Extensive experiments on synthetic and real-world degraded images show that MagnifiQ outperforms prior diffusion-based restoration methods in perceptual quality and human preference, producing sharper textures and more coherent 4K results while offering practical speed--quality trade-offs through its scalable backbone and progressive design.
从降质输入进行高分辨率图像修复具有挑战性,因为需要在恢复细粒度局部细节的同时保持全局结构一致性,尤其是在4K分辨率下,直接基于扩散的修复方法计算量大且容易产生重复或不一致的纹理。本工作中,我们提出了MagnifiQ,一个跨分辨率渐进式放大和修复图像的框架,例如从1024×1024到4096×4096。我们的方法利用预训练的文本到图像扩散模型(如SDXL),通过用卷积操作替换其原始自注意力层来适配更具可扩展性的高分辨率推理,卷积操作的计算成本随图像分辨率线性增长。我们进一步提出了一种迭代多分辨率阶段渐进式放大策略来修复图像,精炼每个中间输出而非直接生成最终4K图像,从而提高全局一致性并减少高分辨率伪影。为增强局部细节同时控制内容漂移,MagnifiQ使用块特定文本提示在修复过程中提供空间局部语义指导。在合成和真实世界降质图像上的广泛实验表明,MagnifiQ在感知质量和人类偏好方面优于先前的基于扩散的修复方法,能够生成更清晰的纹理和更连贯的4K结果,同时通过其可扩展骨干网络和渐进式设计提供实用的速度-质量权衡。
Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Visual Avatar Generation
中文标题:Omni-LiveAvatar: 分钟级实时流式联合音视频虚拟形象生成
作者:Lunjie Zhu, Xingtong Ge, Fangyu Lin, Yi Zhang, Zhening Liu, Mengfei Li, Yumeng Zhang, Guanglu Song, Yu Liu, Jun Zhang
Joint audio-video generative models serve as foundation for immersive and interactive digital-human generation. Nevertheless, most existing models rely on bidirectional attention and multi-step denoising and can generate only short clips, making them unsuitable for real-time interaction over extended durations. We present Omni-LiveAvatar, the first framework for minute-level, real-time streaming joint audio-video avatar generation. Specifically, we propose (1) a progressive autoregressive distillation pipeline that transfers a large bidirectional joint audio-video diffusion model into a few-step autoregressive generator without auxiliary stabilization mechanisms; (2) a synchronized audio-video long-short-term memory that preserves global consistency under a bounded memory budget; and (3) a hierarchical rolling prompt planning strategy that enables coherent semantic evolution and seamless prompt transitions. Extensive experiments show that Omni-LiveAvatar generates high-quality, synchronized minute-level avatars in real time. In terms of speed, it achieves a 33$\times$ generation speedup over its teacher, LTX-2, on a single NVIDIA H200 GPU; in terms of generation quality, it outperforms accelerated baselines across visual quality, audio quality, cross-modal synchronization, and human fidelity. Our code is available at https://github.com/Aoko955/Omni-LiveAvatar.
联合音视频生成模型是实现沉浸式和交互式数字人生成的基础。然而,现有大多数模型依赖双向注意力和多步去噪机制,仅能生成短视频片段,因而不适用于长时间实时交互场景。本文提出Omni-LiveAvatar,这是首个支持分钟级实时流式联合音视频虚拟形象生成的框架。具体而言,我们提出(1)一个渐进式自回归蒸馏管道,能够将大型双向联合音视频扩散模型迁移为少步自回归生成器,且无需辅助稳定机制;(2)一个同步音视频长短期记忆模块,可在有限内存预算下保持全局一致性;(3)一个分层滚动提示规划策略,实现连贯的语义演化和无缝的提示转换。大量实验表明,Omni-LiveAvatar能够实时生成高质量、同步的分钟级虚拟形象。在速度方面,相比其教师模型LTX-2在单块NVIDIA H200 GPU上实现了33倍的生成加速;在生成质量方面,其在视觉质量、音频质量、跨模态同步和人类保真度等指标上均优于加速基线模型。代码已开源于https://github.com/Aoko955/Omni-LiveAvatar。
Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View
中文标题:扩散模型的强化学习设计:统一路径空间视角
作者:Yixian Xu, Yuanrui Zhang, Shengjie Luo, Liwei Wang, Di He
Reinforcement learning (RL) post-training provides a direct way to align diffusion models with human preferences and task-specific rewards. However, current RL algorithms for diffusion models remain fragmented: reverse-trajectory methods rely on discretized likelihood ratios, whereas forward-matching methods train on reward-labeled noising versions of the rollout samples. This paper shows that these seemingly different losses arise from a single path-space principle. Starting from the regularized diffusion-RL objective, we use importance sampling between sampling SDEs to obtain an explicit policy-gradient estimator on trajectory space. The estimator contains the stochastic It\^o integral underlying Flow-GRPO-type updates; we derive an equivalent variance-reduced value-gradient form that recovers the forward-matching structure of AWM and DiffusionNFT. This identifies the empirical gap between these method families as a variance-reduction effect rather than a difference in RL principle. The derivation yields a unified design space organized by value-gradient estimation, weight functions, and sampling choices. Within this space, we propose a multi-sample KDE value-gradient estimator that reuses rollout groups, together with scale-bounded weight families that retain stable existing recipes while excluding singular ones. Experiments on SD3.5-M and Qwen-Image models validate the variance-reduction explanation and show that the resulting recipe improves over prior diffusion-RL baselines.
强化学习后训练为扩散模型与人类偏好及任务特定奖励的对齐提供了一种直接方式。然而,目前针对扩散模型的强化学习算法仍然分散:逆向轨迹方法依赖于离散化的似然比,而前向匹配方法则对rollout样本的奖励标记加噪版本进行训练。本文表明这些看似不同的损失函数源自同一个路径空间原理。从正则化扩散-RL目标出发,我们利用采样随机微分方程之间的重要性采样,在轨迹空间上获得了一个显式的策略梯度估计器。该估计器包含了Flow-GRPO型更新背后的随机Itô积分;我们推导了一种等价的方差缩减值梯度形式,它恢复了AWM和 DiffusionNFT的前向匹配结构。这将这些方法族之间的经验差距识别为方差缩减效应而非强化学习原理的差异。该推导产生了一个由值梯度估计、权重函数和采样选择统一组织的设计空间。在此空间内,我们提出了一种复用rollout组的多样本KDE值梯度估计器,以及有界尺度权重族,它在保留现有稳定配方的同时排除奇异的权重形式。在SD3.5-M和 Qwen-Image模型上的实验验证了方差缩减解释,并表明所得配方优于先前的扩散-RL基准。
The Linear Geometry of Interpretable Tokens: Jailbreaking Attacks and Defenses for Unlearned Diffusion Models
中文标题:可解释Token的线性几何:针对反学习扩散模型的越狱攻击与防御
作者:Siyi Chen, Yimeng Zhang, Sijia Liu, Qing Qu
Diffusion models excel at generating high-quality images but can memorize and reproduce harmful concepts when prompted. Although fine-tuning methods have been proposed to unlearn a target concept, they struggle to fully erase it while maintaining generation quality on other concepts, leaving models vulnerable to jailbreak attacks. Existing jailbreak methods demonstrate this vulnerability but offer limited insight into how unlearned models retain harmful concepts, limiting progress on effective defenses. In this work, we show that the erased concept persists as a coherent, interpretable linear subspace of the token embedding space, and that both an attack and a defense follow directly from this structure. We introduce SubAttack, a novel jailbreaking attack that reads out this subspace by learning an orthogonal set of attack token embeddings, each being a linear combination of human-interpretable textual elements, revealing that unlearned models still retain the target concept through related textual components. Furthermore, our attack is also more powerful and transferable across text prompts, initial noises, and unlearned models than prior attacks. Conversely, projecting out the same subspace yields SubDefense, a lightweight plug-and-play defense mechanism that suppresses the residual concept in unlearned models. SubDefense provides stronger robustness than existing defenses while better preserving safe generation quality. Extensive experiments across multiple unlearning methods, concepts, and attack types demonstrate that our approach advances both understanding and mitigation of vulnerabilities in diffusion unlearning.
扩散模型在生成高质量图像方面表现出色,但在提示下可能记忆并重现有害概念。尽管已有微调方法被提出用于消除目标概念,但这些方法难以在保持其他概念生成质量的同时完全擦除目标概念,导致模型易受越狱攻击。现有的越狱方法展示了这一漏洞,但对反学习模型如何保留有害概念的机制提供了有限的理解,限制了有效防御的研究进展。在本工作中,我们发现被消除的概念作为一致的、可解释的线性子空间存在于token嵌入空间中,且攻击和防御方法均可直接从该结构推导得出。我们提出了SubAttack,这是一种新型越狱攻击,通过学习一组正交的攻击token嵌入来读取该子空间,每个嵌入都是人类可解释的文本元素的线性组合,揭示了反学习模型仍通过相关文本组件保留目标概念。此外,我们的攻击相比先前方法更强大,且在文本提示、初始噪声和反学习模型之间具有更强的可转移性。相反,投影消除同一子空间得到了SubDefense,这是一种轻量级的即插即用防御机制,可抑制反学习模型中的残留概念。SubDefense提供了比现有防御更强的鲁棒性,同时更好地保持了安全生成质量。我们在多种反学习方法、概念和攻击类型上进行了广泛实验,证明我们的方法推进了对扩散模型反学习漏洞的理解和缓解。
CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
中文标题:CoCA:基于强化学习的文本到图像扩散模型微调中的步骤级奖励
作者:Xinyao Liao, Wei Wei, Xiaoye Qu, Qiyuan He, Angela Yao, Yu Cheng
Recent advances in text-to-image (T2I) diffusion model fine-tuning leverage reinforcement learning (RL) to align generated images with learnable reward functions. The existing approaches reformulate denoising as a Markov decision process for RL-driven optimization. However, they suffer from reward sparsity, receiving only a single delayed reward per generated trajectory. This flaw hinders precise step-level attribution of denoising actions, undermines training efficiency. To address this, we propose a simple yet effective credit assignment framework that dynamically distributes dense rewards across denoising steps. Specifically, we track changes in cosine similarity between intermediate and final images to quantify each step's contribution on progressively reducing the distance to the final image. Our approach avoids additional auxiliary neural networks for step-level preference modeling and instead uses reward shaping to highlight denoising phases that have a greater impact on image quality. Our method achieves 1.25 to 2 times higher sample efficiency and better generalization across four human preference reward functions, without compromising the original optimal policy.
近期文本到图像(T2I)扩散模型微调利用强化学习(RL)将生成图像与可学习奖励函数对齐。现有方法将去噪重新表述为马尔可夫决策过程以进行强化学习驱动优化,但存在奖励稀疏问题,每个生成轨迹仅获得单一延迟奖励。这一缺陷阻碍了对去噪动作的精确步骤级归因,降低了训练效率。为解决这一问题,我们提出了一个简单而有效的信用分配框架,能够在去噪步骤中动态分配密集奖励。具体而言,我们追踪中间图像与最终图像之间余弦相似度的变化,以量化每个步骤对逐步缩小与最终图像距离的贡献。我们的方法无需使用额外的辅助神经网络进行步骤级偏好建模,而是通过奖励塑造来突出对图像质量影响更大的去噪阶段。我们的方法在四个人类偏好奖励函数上实现了1.25至2倍的样本效率提升和更好的泛化性能,且不影响原始最优策略。
Concept Unlearning by Modeling Key Steps of Diffusion Process
中文标题:通过建模扩散过程关键步骤实现概念解除学习
作者:Chaoshuo Zhang, Chenhao Lin, Zhengyu Zhao, Le Yang, Qian Wang, Chao Shen
Text-to-image diffusion models remain susceptible to generating undesirable or harmful content. Although concept unlearning mitigates this risk, existing methods struggle with a critical optimization dilemma: thorough semantic erasure frequently induces the catastrophic forgetting of unrelated generative capabilities. To overcome this challenge, we propose Key Step Concept Unlearning (KSCU). Serving as an integrated methodological refinement deeply motivated by information theory, KSCU explores the profound impact of step scheduling order and reveals that traditional randomized timestep sampling severely disrupts trajectory dependency. We demonstrate that indiscriminately targeting the entire diffusion process is inefficient, as the optimal step range for unlearning inherently varies across different concepts. Rather than globally fine-tuning all timesteps, KSCU explicitly integrates a sequential-scheduling-based Key Step Table, CFG-aware leakage compensation, and prompt augmentation to dynamically isolate optimization to a concept-specific active region. This localized strategy successfully eradicates the target concept while preventing the structural collapse caused by early-step over-optimization. Consequently, KSCU significantly reduces computational overhead and establishes a state-of-the-art trade-off between concept erasure and utility retention. Comprehensive evaluations demonstrate that KSCU consistently delivers superior performance across diverse unlearning tasks, including nudity, style, object classes, and mass instance concepts. For example, in nudity removal, KSCU yields a 96.5% unlearning accuracy alongside a state-of-the-art FID of 14.1.
文生图扩散模型在生成不良或有害内容方面仍存在风险。尽管概念解除学习可以缓解这一风险,但现有方法面临关键优化困境:彻底擦除语义常常导致不相关的生成能力发生灾难性遗忘。为克服这一挑战,我们提出关键步骤概念解除学习(KSCU)。KSCU作为一种深度基于信息理论的方法,探讨了步骤调度顺序的深远影响,并揭示传统随机时间步采样严重破坏轨迹依赖性。我们证明,无差别地针对整个扩散过程进行优化是低效的,因为不同概念的最优步骤范围存在固有差异。KSCU摒弃全局微调所有时间步的方式,而是明确整合基于顺序调度的关键步骤表、CFG感知泄漏补偿和提示增强技术,将优化动态隔离至概念特定的活跃区域。该局部化策略成功消除了目标概念,同时防止了早期步骤过度优化导致的结构崩溃。因此,KSCU显著降低了计算开销,并在概念擦除与效用保留之间实现了最先进的权衡。全面评估表明,KSCU在多种解除学习任务中持续提供卓越性能,包括裸体内容、风格、物体类别和大规模实例概念。例如,在裸体内容移除任务中,KSCU实现了96.5%的解除学习准确率,同时达到FID为14.1的最先进水平。
RefGC-SR$^2$: Reference-guided Super-Resolution and Refinement of AI Generated Content
中文标题:RefGC-SR²: AI生成内容的参考引导超分辨率与细化
作者:Jeahun Sung, Dahyeon Kye, Soo Ye Kim, Jihyong Oh
Reference-guided generation (e.g., object compositing, customization) has progressed rapidly, yet current pipelines share a fundamental limitation: the object-centric high-resolution reference image (HRRI) provided by users is downsampled to a fixed low-resolution (LR) before being fed into the model, so the fine-grained details are discarded before the output is even produced. In addition, the generation step then introduces its own artifacts (e.g., identity distortion) on top of this loss. Existing reference-guided generated content refinement (RefGCR) methods can correct some of these artifacts but still operate in the LR domain; reference-guided super-resolution (RefSR) methods recover resolution but assume natural-image degradations and ignore the artifact distribution of generative pipelines. To address both gaps in a single formulation, we introduce a new task: reference-guided generated content super-resolution-refinement (RefGC-SR$^2$), where the original HRRI is reused at the post-processing stage to recover lost details, refine generative artifacts, and upscale the output simultaneously. We construct the first real-world triplet data generation pipeline for this RefGC-SR$^2$ task, training a diptych-conditioned generator to synthesize paired low-quality anchors that public pretrained models cannot provide. We further present a frequency-aware diffusion transformer model for RefGC-SR$^2$ that selectively injects fine details from the HRRI while removing generative artifacts. Extensive experiments demonstrate that our RefGC-SR$^2$ model successfully (i) refines the object identity faithfully with respect to the reference, and (ii) recovers high-resolution details, so that the final result is significantly higher quality and practically more usable compared to existing RefGCR and RefSR baselines.
参考引导生成(如目标合成、定制化)进展迅速,但当前流程存在一个根本性限制:用户提供的以对象为中心的高分辨率参考图像(HRRI)在输入模型前会被下采样到固定的低分辨率(LR),导致细粒度细节在输出产生前就已经丢失。此外,生成步骤本身还会在此基础上引入额外的伪影(如身份畸变)。现有的参考引导生成内容细化(RefGCR)方法可以纠正部分伪影,但仍在低分辨率域进行处理;参考引导超分辨率(RefSR)方法虽然能恢复分辨率,但假设的是自然图像退化,忽略了生成管道的伪影分布。为在一个统一的框架中解决这两个问题,我们引入了一个新任务:参考引导生成内容超分辨率-细化(RefGC-SR²),在后处理阶段重用原始HRRI以同时恢复丢失的细节、细化生成伪影并提升输出分辨率。我们为该RefGC-SR²任务构建了首个真实世界三元组数据生成管道,训练了一个双联条件生成器来合成公开预训练模型无法提供的成对低质量锚点。我们进一步提出了一种频率感知扩散Transformer模型用于RefGC-SR²,该模型能从HRRI中选择性注入细粒度细节,同时去除生成伪影。大量实验表明,我们的RefGC-SR²模型成功实现了(i)相对于参考图像忠实地细化对象身份,(ii)恢复高分辨率细节,使得最终结果与现有的RefGCR和RefSR基线相比质量显著更高且更实用。
$\mu$Flow: Leveraging Average Images for Improving Generalisation of Deepfake Faces Detectors
中文标题:μFlow:利用平均图像改进深度伪造人脸检测器的泛化性能
作者:Orazio Pontorno, Mattia Litrico, Luca Guarnera, Mario Valerio Giuffrida, Sebastiano Battiato
Current generative models, including GANs and diffusion models, have reached an outstanding level of photorealism, posing significant risks to privacy and security. To ensure real-world applicability, deepfake detectors must generalise effectively to unseen generators. However, most existing approaches rely on supervised training with both real and fake images, which limits their generalisation especially across generators categories (e.g. GANs vs DMs). In this work, we introduce $\mu$Flow, a one-class deepfake detector trained only on real images without relying on pseudo-deepfakes or synthetic artifacts. Our approach builds on the observation that averaging multiple images amplifies consistent generative traces, producing highly discriminative feature representations. We leverage this property by modelling the distribution of features extracted from averaged images and training a normalizing flow to align the feature space of individual images with this distribution. This alignment yields a likelihood-based criterion that separates real and fake samples while promoting strong generalisation. We evaluate $\mu$Flow on a fully out-of-distribution setting, where both real and fake datasets are unseen during training. Experimental results show that our method significantly outperforms SOTA detectors. Project page: https://opontorno.github.io/MuFlow.
当前的生成模型(包括GANs和扩散模型)已达到照片级真实感水平,对隐私和安全构成重大风险。为确保实际应用,深度伪造检测器必须能够有效泛化至未见过的生成器。然而,现有大多数方法依赖于真实图像和伪造图像的监督训练,这限制了其泛化能力,尤其是在不同生成器类别(如GANs与扩散模型之间)。在本工作中,我们提出了μFlow,这是一种仅基于真实图像训练的单类深度伪造检测器,不依赖于伪深度伪造或合成伪影。我们的方法基于以下观察:平均多幅图像会放大一致的生成痕迹,产生高度判别的特征表示。我们利用这一特性,对从平均图像提取的特征分布进行建模,并训练一个归一化流将单个图像的特征空间与该分布对齐。这种对齐产生了一种基于似然的判别准则,能够分离真实和伪造样本,同时促进强泛化。我们在完全分布外设置下评估μFlow,即训练时使用未见过的真实和伪造数据集。实验结果表明,我们的方法显著优于最先进的检测器。项目页面:https://opontorno.github.io/MuFlow
InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Driving
中文标题:InfiniVerse:面向自动驾驶的占用引导无界场景生成
作者:Xiaoyu Ye, Leheng Li, Xinyu Ji, Yingjie Cai, Hongda He, Xu Yan, Guanyi Zhao, Ying-Cong Chen, Bingbing Liu, Shuguang Cui, Zhen Li
Generating realistic, controllable, and temporally coherent urban environments is a critical yet unresolved challenge in the autonomous driving community. In this paper, we introduce InfiniVerse, a unified pipeline for long-range, 2D-3D-aligned, and controllable synthesis of dynamic urban scenes from a single frame. In practice, our approach first reconstructs a 3D occupancy representation from the input multi-view frame. This representation serves as a foundation for autoregressive scene extension along arbitrary trajectories. Subsequently, a video diffusion model translates the coarse occupancy grid into realistic, spatiotemporally consistent video sequences. Moreover, we propose a hierarchical sketch-and-refine paradigm, in which the generated videos are re-projected as image-conditioned feedback to enhance the 3D occupancy representation, establishing cross-modal alignment and mutual enhancement between the visual and spatial domains. Extensive evaluations on the Waymo Open Dataset and nuScenes demonstrate that InfiniVerse achieves state-of-the-art performance, with a FID of 6.4 and FVD of 67.97, significantly outperforming existing benchmarks in both duration and stability.
生成真实、可控且时间一致的城市环境是自动驾驶领域的一项关键而未解决的挑战。本论文提出了 InfiniVerse,一个统一的 pipeline,能够从单帧图像实现动态城市场景的长距离、2D-3D对齐且可控的合成。具体而言,我们的方法首先从输入的多视角帧图像中重建 3D 占用表示,以此作为沿任意轨迹进行自回归场景扩展的基础。随后,视频扩散模型将粗糙的占用网格转换为真实且时空一致的视频序列。此外,我们提出了层次化草图-精化范式,将生成的视频重投影为图像条件反馈,以增强 3D 占用表示,从而建立视觉与空间域之间的跨模态对齐和相互增强。在 Waymo Open Dataset 和 nuScenes 数据集上的广泛评估表明,InfiniVerse 取得了领先性能,FID 达到 6.4,FVD 达到 67.97,在持续时间和稳定性方面均显著优于现有基准方法。
BAG: Budget-Aware Gating for Diffusion Caching
中文标题:BAG: 面向扩散缓存的预算感知门控
作者:Tong Zhao, Mingkun Lei, Yucheng Han, Chi Zhang
Diffusion caching is a lightweight strategy that accelerates Diffusion Transformers (DiTs) by reusing intermediate features across denoising steps, but existing paradigms face a fundamental trade-off: online heuristics lack global budget awareness, whereas static schedules lack instance adaptivity and fail to flexibly adapt to varying runtime budget constraints. To bridge this gap, we present BAG (Budget-Aware Gating), a novel caching policy that unifies global budget pacing with dynamic, instance-adaptive feature reuse. Rather than relying on hand-crafted rules, BAG employs a lightweight gating network that dynamically decides whether to execute a full computation or reuse cached features at each step by jointly conditioning on the budget state and local trajectory feedback. We train this policy via offline-to-online schedule distillation, transferring the decision-making of offline-searched schedules into a compact online gate. Extensive experiments on FLUX.1-dev, Wan2.1, and Qwen-Image-2512 demonstrate that BAG consistently outperforms state-of-the-art caching methods across various speedup tiers while remaining robust across different resolutions, seeds, and guidance scales. Code will be released.
扩散缓存是一种轻量级策略,通过在去噪步骤中重用中间特征来加速扩散变压器(Diffusion Transformers, DiTs),但现有范式面临一个根本性的权衡:在线启发式方法缺乏全局预算感知,而静态调度缺乏实例适应性,无法灵活适应不同的运行时预算约束。为弥补这一差距,我们提出了BAG(Budget-Aware Gating,预算感知门控),一种将全局预算节奏与动态实例自适应特征重用统一起来的新型缓存策略。BAG不依赖手工规则,而是采用一个轻量级门控网络,通过同时对预算状态和局部轨迹反馈进行条件化,动态决定在每个步骤是执行完整计算还是重用缓存特征。我们通过离线到在线调度蒸馏来训练这一策略,将离线搜索到的调度决策转移到一个紧凑的在线门控器中。在FLUX.1-dev、Wan2.1和Qwen-Image-2512上的大量实验表明,BAG在各种加速层级上始终优于最先进的缓存方法,同时在不同分辨率、随机种子和引导尺度下均保持鲁棒性。代码将公开发布。
Spatially-Grounded Text-to-Video Generation via Inference-Time Gradient-Free Optimization
中文标题:基于空间锚定的文本到视频生成方法推理时无梯度优化
作者:Guillaume Jeanneret, Mathis Koroglu, Hugo Caselles-Dupr\'e, Arnaud Dapogny, Matthieu Cord
Diffusion Transformer Text-to-Video models have achieved remarkable synthesis quality, yet fine-grained spatial controllability remains a significant challenge. While existing training-free methods produce solid overall results in spatially grounded generation, \ie, placing a specific object in a designated location, they rely on gradient-based optimization techniques that incur prohibitive computational overhead, a bottleneck amplified in modern large-scale architectures. To address this limitation, we present Gradient-free Analytical Trajectory Optimization Video Generation (GATO-Vid), a novel training-free and gradient-free approach for precise spatial guidance. Rather than relying on costly backward passes, we introduce an alternative cross-attention score and solve it analytically to obtain an exact, closed-form solution. To use our analytical solution, we propose an on-the-fly injection mechanism tailored to the topological manifold of the transformer's latent space. Our experiments demonstrate that GATO-Vid significantly outperforms existing baselines in localization accuracy while introducing minimal computational overhead.
扩散变换器文本到视频模型已取得优异的合成质量,然而细粒度空间可控性仍是一项重大挑战。现有的无训练方法在空间锚定生成(即在指定位置放置特定物体)方面能够产生良好的整体效果,但它们依赖于基于梯度的优化技术,产生高昂的计算开销,在现代大规模架构中这一瓶颈更为明显。为解决这一局限性,我们提出了无梯度解析轨迹优化视频生成(GATO-Vid),一种用于精确空间引导的新型无训练、无梯度方法。我们不依赖代价高昂的反向传播,而是引入一种替代交叉注意力分数并通过解析求解获得精确的闭式解。为使用我们的解析解,我们提出了一种针对变换器潜在空间拓扑流形量身定制的即时注入机制。我们的实验表明,GATO-Vid在定位精度方面显著优于现有基线方法,同时引入了极小的计算开销。
Latent Reward Registers for Diffusion Preference Alignment
中文标题:扩散偏好对齐的潜在奖励寄存器
作者:Yuanshen Guan, Zipeng Feng, Chengru Song, Zhiwei Xiong, Peiqin Sun
Aligning diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, which creates a severe temporal credit-assignment problem across the denoising process. We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents. Learnable, position-free register tokens are appended as an auxiliary read path to a frozen Diffusion Transformer (DiT), extracting preference signals without altering the generator's hidden states or velocity field. The resulting dense, differentiable reward field spans the full denoising trajectory and supports two alignment strategies. For training, Reward-Gradient On-Policy Distillation (RG-OPD) converts this dense reward field into per-step targets at states visited by the current generator, replacing rollout-intensive policy gradients with direct on-policy distillation. For inference, Reward-Guided Sampling (RGS) steers trajectories with magnitude-matched reward-gradient corrections and no parameter updates. Empirically, at high noise levels (t=0.8) the registers reach the highest pairwise accuracy among the evaluated latent reward models. RG-OPD outperforms online reinforcement learning baselines while reducing GPU hours by up to 33x. RGS achieves significant reward improvement with a favorable reward-quality balance against training-free baselines. Code and weights are to be available at https://github.com/Guanys-dar/latent-reward-register
使扩散模型与人类偏好对齐通常依赖于对最终生成样本评估的稀疏终端奖励,这在整个去噪过程中造成了严重的时间信用分配问题。我们提出潜在奖励寄存器,这是一种从中间噪声潜变量直接估计终端偏好的机制。可学习的、无位置依赖的寄存器令牌作为辅助读取路径附加到冻结的扩散Transformer(DiT)上,在不改变生成器的隐藏状态或速度场的情况下提取偏好信号。由此产生的密集可微奖励场跨越完整的去噪轨迹,并支持两种对齐策略。对于训练,奖励梯度在线策略蒸馏(RG-OPD)将密集奖励场转换为当前生成器访问状态下的每步目标,用直接在线策略蒸馏替代 rollout 密集的策略梯度。对于推理,奖励引导采样(RGS)使用幅度匹配的奖励梯度修正来引导轨迹,且无需参数更新。在高噪声水平(t=0.8)下,寄存器在评估的潜在奖励模型中达到了最高的成对准确率。RG-OPD 优于在线强化学习基线,同时将 GPU 小时数减少多达 33 倍。RGS 实现了显著的奖励提升,并在与无训练基线的比较中取得了有利的奖励-质量平衡。代码和权重可从 https://github.com/Guanys-dar/latent-reward-register 获取。
今日未找到该分类的匹配论文。
今日未找到该分类的匹配论文。
今日未找到该分类的匹配论文。
今日未找到该分类的匹配论文。
今日未找到该分类的匹配论文。
今日未找到该分类的匹配论文。