ESC
输入关键词搜索文章
目录

每日 arXiv 论文简报

2026-08-22 · 30 篇论文 · 按研究方向分组
自动追踪 · LLM 总览 · 研究雷达
30Total Papers
7Autoregressive
20Diffusion
3Image Compression
01D Visual Tokenizer
0Diffusion Visual Encoder
Daily Radar
每日总览

今日 arXiv 的研究呈现出一个明显的核心趋势:生成式 AI 正在从 2D 向 4D(时空)扩展,并同时追求更高的效率和物理可解释性。在扩散模型领域,20 篇论文中有近三分之一聚焦于视频、3D 和 4D 生成任务(如 Stream4D、Block3D、4DAnyone、AvatarDynamicer),这反映了生成内容从静态图像向动态、可交互场景的演进。自回归模型则与扩散方法出现深度交叉——如 RoMAN-Flow 将自回归归一化流用于机器人强化学习,Stream4D 将自回归架构与扩散视频模型结合,体现了不同生成范式之间的融合。

另一条主线是效率优化与硬件协同设计:KV-Cache 的所有权感知管理、推测解码、块式扩散等方法共同指向推理成本的控制;尤其值得注意的是关于模拟计算内存(Analog Compute-in-Memory)非理想特性对扩散 Transformer 影响的研究,这代表了对物理硬件与模型深度 co-design 的前沿探索。

跨领域应用持续拓展:分割掩膜作为模拟到真实的迁移桥梁(Mask2Real-WM)、Flow Matching 用于 PET 图像重建、Gramian Chebyshev 神经算子用于无线信道压缩等,表明生成模型正深入医疗、通信、机器人等实体领域。

  • Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion — 通过块式扩散策略显著提升文本到 3D 生成效率,为消费级设备上的 3D 内容创作提供可行性。
  • 4DAnyone: Create Anyone in 4D from a Casual Monocular Video — 只需单目视频即可重建人物的 4D 表示,是数字人/虚拟形象领域的重要突破。
  • Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge for Controllable Dexterous World Models — 提出用分割掩膜作为模拟到真实的桥梁,为机器人灵巧操作的世界模型训练提供新范式。
  • When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities — 首个系统研究模拟计算内存非理想性对大规模扩散 Transformer 影响的论文,对下一代 AI 芯片设计有重要参考价值。
  • Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy — 提出所有权感知的 KV-Cache 管理,在对话系统中实现更精准的推理相关性,兼具工程价值与理论创新。
autoregressive
Autoregressive
7 篇论文

Autoregressive 类别今日总览

今日 Autoregressive 相关论文聚焦于推理效率优化跨模态生成两大主题。在效率提升方面,Fractional Decay KV-Cache 和 Approximate Speculative Decoding 分别从内存管理和推测解码角度优化自回归模型的推理速度;SparSTAR 通过稀疏注意力机制降低视频合成的计算开销。在生成任务上,Stream4D 和 Block3D 各自探索了视频与 3D 生成的新范式,分别关注 4D 一致性和分块扩散策略。RoMAN-Flow 则创新性地将自回归归一化流应用于机器人操作的离线强化学习,拓展了自回归方法的应用边界。整体趋势表明,研究重点正从模型架构转向推理部署效率和实际应用场景的探索。

重点论文推荐:

  • RoMAN-Flow:首次将自回归归一化流应用于机器人离线强化学习,为操作技能学习提供了新的生成式建模思路。
  • SparSTAR:提出时空稀疏注意力机制,有效解决了自回归视频合成中计算成本过高的问题。
  • Approximate Speculative Decoding:通过近似推测解码技术,可在保持输出质量的前提下显著加速自回归模型的推理过程。
  • Stream4D:创新性地引入 4D 一致性约束,解决了流式视频生成中的时序连贯性问题。
  • Block3D:提出分块扩散策略,大幅提升文本到 3D 内容的生成效率与质量。

Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems

2026-08-21T04:00:00autoregressive, cs.AI, cs.CL2608.18098

中文标题:分数衰减KV缓存:面向对话系统中推理相关性的所有权感知内存管理

作者:Sukanta Ganguly

摘要:

Key-value (KV) caching is essential for efficient autoregressive inference in transformer based dialog systems, yet existing strategies treat all cached entries uniformly or apply coarse eviction heuristics that fail to adapt as dialog topics evolve. We propose Fractional Decay KV-Cache (FD-KVC), a novel algorithm that maintains a dual-channel scoring mechanism for each cached KV pair: a cumulative attention channel that tracks aggregate importance (akin to H2O), and a recency-weighted relevance channel governed by temporal decay and reinforcement-inspired updates. The combination enables FD-KVC to both preserve historically important tokens and rapidly adapt when dialog topics shift. An adaptive learning rate driven by an ownership loss function ensures convergence without oscillation. FD-KVC operates entirely on CPU with negligible overhead. Across five diverse multi-turn dialog scenarios with 600 dialogs each, FD-KVC outperforms H2O, the state-of-the-art heavy-hitter baseline, by +6.7% on composite late-turn alignment, with improvements of +127% on topic-shift, +87% on gradual evolution, and +30% on mixed-topic dialogs. FD-KVC adapts to new topics 3.6X faster than H2O and achieves the highest topic diversity (80.6%) across all methods. Ablation studies confirm the contribution of each component.

摘要中文:

键值(KV)缓存对于基于Transformer的对话系统中的高效自回归推理至关重要,然而现有策略对所有缓存条目一视同仁,或采用粗粒度的淘汰启发式方法,无法适应对话主题的演变。我们提出分数衰减KV缓存(FD-KVC),这是一种创新算法,为每个缓存的KV对维护双通道评分机制:累积注意力通道用于跟踪整体重要性(类似于H2O),以及由时间衰减和强化学习启发更新控制的近因加权相关性通道。这种组合使FD-KVC既能保留历史上重要的标记,又能在对话主题转变时迅速适应。由所有权损失函数驱动的自适应学习率确保收敛过程无振荡。FD-KVC完全在CPU上运行,开销可忽略不计。在五个不同的多轮对话场景(每个场景600个对话)中,FD-KVC在复合后期对齐指标上优于H2O(当前最先进的重点击基线)达+6.7%,在主题转换任务上提升+127%,在渐进演变任务上提升+87%,在混合主题对话任务上提升+30%。FD-KVC适应新主题的速度比H2O快3.6倍,并在所有方法中实现最高的主题多样性(80.6%)。消融实验验证了各组件的贡献。

Approximate Speculative Decoding

2026-08-21T04:00:00autoregressive, cs.AI, cs.LG2608.03447

中文标题:近似投机解码

作者:Yuannuo Feng, Zegang Peng, Yuxin Xie, Yubing Ye, Yizhe Chen, Wenshuai Yao, Wenyong Zhou, Wang Kang

摘要:

Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel. Under standard greedy verification, decoding stops at the first draft token that differs from the target argmax, discarding the remaining target-scored suffix. Although accepting such a mismatch changes the decoding trajectory, it can make a contiguous suffix reusable when its tokens remain target-greedy under the realized prefix. In this paper, we introduce \textbf{Approximate Speculative Decoding (ASD)}, a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection. ASD accepts selected mismatches subject to a local target-logit regret gate, a per-block exception cap, and a persistent request-level regret budget, then reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes. ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero. Experiments show that ASD improves fixed-workload throughput by $3.05\%$--$15.26\%$ over matched strict verification and averages a $7.78\%$ gain across seven Qwen3-14B + DSpark-14B tasks. On DeepSeek-V4-Flash (284B) with DSpark it also raises verifier-side acceptance by roughly $10\%$--$16\%$ on GSM8K and MATH-500 in an FP4-to-FP8 compatibility setting. The source code is publicly available at: https://github.com/Kissmetothemoon/ASD

摘要中文:

投机解码通过使用目标模型并行验证draft块来加速自回归生成。在标准贪婪验证下,解码在第一个与目标argmax不同的draft token处停止,并丢弃剩余的目标评分后缀。尽管接受这种不匹配会改变解码轨迹,但当其token在实现的前缀下保持目标贪婪时,可以使连续后缀变得可复用。本文引入了 extbf{近似投机解码(Approximate Speculative Decoding, ASD)},这是一种无需训练的验证器,用预算最长前缀选择替代二进制首次不匹配截断。ASD在局部目标logit遗憾门、每块异常上限和持续请求级遗憾预算的约束下接受选中的不匹配,然后在无需额外近似决策或目标模型前向传播的情况下复用连续的目标贪婪后缀。ASD既不需要新的draft模型,也不需要微调,当预算为零时完全等价于标准贪婪验证。实验表明,ASD在固定工作负载吞吐量上比匹配的严格验证提升3.05%至15.26%,并在七个Qwen3-14B + DSpark-14B任务中平均获得7.78%的收益。在DeepSeek-V4-Flash(284B)配合DSpark使用时,在FP4到FP8兼容性设置下,GSM8K和MATH-500上的验证器侧接受率也提高了约10%至16%。源代码公开可访问:https://github.com/Kissmetothemoon/ASD

Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models

2026-08-21T04:00:00autoregressive, cs.AI, cs.CV, diffusion2608.19556

中文标题:Stream4D:流式自回归扩散视频模型的4D一致性

作者:Yuanhao Ban, Jiaqi Feng, Hengguang Zhou, Xiaohuan Pei, Justin Cui, Cho-Jui Hsieh

摘要:

Streaming autoregressive diffusion models enable real-time, long-horizon video generation, but their training objectives optimize local frame prediction rather than the geometry and dynamics of a coherent world: long rollouts accumulate geometric drift and degrade into static or unnatural motion. Recent bidirectional approaches address this problem using rewards signals built upon 3D Gaussian-Splatting reconstruction. However, a single rigid 3d reconstruction cannot model a dynamic scene, so this critic penalizes genuine object motion as reconstruction error and is maximized by freezing the video. This shortcut is especially detrimental in the AR setting, where each chunk can propagate an already-static configuration. In this work, we propose Stream4D, which replaces the static critic with a feed-forward 4D reconstruction reward that explicitly models scene dynamics, allowing coherent motion to receive high consistency rewards. To further guide motion magnitude and quality, we add a motion prior that rewards natural scene-flow magnitude while penalizing jitter and non-rigid artifacts. Our final recipe combines these two terms with a lightweight perceptual anchor. Across various autoregressive video backbones and various generation horizons, Stream4D improves 4D reconstruction quality, preserves motion more effectively, and achieves higher human-aligned preference. Project page: https://banyuanhao.github.io/Stream4D/

摘要中文:

流式自回归扩散模型能够实现实时、长时域视频生成,但其训练目标优化的是局部帧预测而非连贯世界的几何与动态:长时程推演会积累几何漂移,并退化为静止或不自然的运动。最近的双向方法利用基于3D Gaussian-Splatting重建的奖励信号来解决这一问题。然而,单一的刚性3D重建无法建模动态场景,因此该判别器将真实物体运动惩罚为重建误差,且通过冻结视频可获得最高奖励。这种捷径在自回归设置中尤其有害,因为每个片段都可能传播已静止的配置。本研究提出Stream4D,用前馈4D重建奖励取代静态判别器,该奖励显式建模场景动态,使连贯运动能够获得高一致性奖励。为了进一步引导运动幅度和质量,我们添加了一个运动先验,奖励自然的场景流幅度,同时惩罚抖动和非刚性伪影。我们的最终方案将这两个 term 与轻量级感知锚点相结合。在各种自回归视频骨干网络和不同生成时域下,Stream4D提升了4D重建质量,更有效地保留运动,并获得更高的类人偏好。项目主页:https://banyuanhao.github.io/Stream4D/

Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion

2026-08-21T04:00:00autoregressive, cs.CV, diffusion2608.19567

中文标题:Block3D:通过分块扩散实现高效的文本到3D生成

作者:Bowen Cui, Weijie Wang, Zeyu Zhang, Yefei He, Mingda Lin, Haoyu Zhao, Yuanyu He, Donny Y. Chen, Feng Chen, Bohan Zhuang

摘要:

While text-to-3D generation has advanced rapidly, achieving high geometric fidelity at low inference cost remains challenging. Existing text-to-3D methods either decode discrete shape tokens autoregressively or iteratively refine global 3D representations with diffusion or flow models. However, autoregressive decoding is sequential and cannot revise errors, whereas diffusion and flow-matching models repeatedly process the full representation, making high-quality generation increasingly expensive. In this paper, we propose Block3D, a block-wise diffusion framework that partitions the discrete shape-token sequence into contiguous blocks, generates the blocks autoregressively, and jointly denoises all tokens within the current block. To alleviate error accumulation, we introduce confidence-guided intra-block correction, which revises low-confidence tokens before each block is finalized. On a held-out set from TRELLIS-500K, Block3D reduces mean end-to-end generation time from 25.71 seconds to 4.99 seconds, achieving a $5.15\times$ speedup over the fine-tuned autoregressive baseline without sacrificing geometric fidelity.

摘要中文:

虽然文本到3D生成发展迅速,但在低推理成本下实现高几何保真度仍具有挑战性。现有的文本到3D方法要么自回归地解码离散形状token,要么使用扩散或流模型迭代优化全局3D表示。然而,自回归解码是顺序执行的且无法修正错误,而扩散和流匹配模型则需要反复处理完整表示,导致高质量生成的成本不断增加。本文提出Block3D,一个分块扩散框架,该框架将离散形状token序列划分为连续块,自回归地生成这些块,并在当前块内对所有token进行联合去噪。为了减轻误差累积,我们引入了置信度引导的块内修正机制,在每个块最终确定前对低置信度token进行修正。在TRELLIS-500K的保留测试集上,Block3D将平均端到端生成时间从25.71秒减少到4.99秒,相比微调后的自回归基线实现了5.15倍的加速,且未牺牲几何保真度。

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

2026-08-21T04:00:00autoregressive, cs.CV, diffusion2608.20208

中文标题:RoMAN-Flow: 驯服自回归归一化流用于机器人操作的离线强化学习

作者:Shaoxuan Wang, Guangting Zheng, Rui Huang, Zhipeng Tang, Sha Zhang, Jiajun Deng, Yanyong Zhang

摘要:

Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.

摘要中文:

离线强化学习利用先前收集的数据来改进机器人策略,无需进一步的环境交互。然而,现有的扩散模型和流匹配机器人策略缺乏可计算的似然值,限制了其在基于似然的离线强化学习后训练中的应用。自回归归一化流(AR-NFs)既提供了表达性的动作建模能力,又支持精确的似然估计,但在策略优化和部署过程中,其顺序采样会产生大量的采样开销。我们提出了RoMAN-Flow(基于自回归归一化流的机器人操作),这是一个离线强化学习框架,通过在两个阶段解决采样瓶颈问题,使AR-NF策略能够真正应用于机器人操作。在策略优化阶段,RoMAN-Flow采用了一种无采样的优势加权似然目标函数,该函数可以在不从自回归策略采样的情况下,为离线数据集中的高优势动作分配更高的似然值。为了实现高效部署,RoMAN-Flow将优化后的自回归策略蒸馏为一个单步动作生成器,从而实现低延迟的动作预测。在多个模拟操作基准测试和真实机器人平台上的实验表明,RoMAN-Flow在实现具有竞争力的策略性能的同时,大幅降低了推理延迟。代码可访问 https://github.com/konnyaku28/RoMAN-Flow。

SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis

2026-08-21T04:00:00autoregressive, cs.CV, diffusion2608.10519

中文标题:SparSTAR: 时空自回归视频合成的稀疏注意力方法

作者:Jongbeom Lee, Hyunwoo Yu, Jincheol Yang, Jaemin Choi, Suk-Ju Kang

摘要:

InfinityStar extends visual autoregressive generation to video through a sequence of image and clip pyramids. Its changing scale and cross-clip context, however, leave late-scale attention costly and make sparse patterns reused from diffusion or image VAR models unreliable. We introduce SparSTAR, a training-free block-sparse attention method tailored to this setting. At each expensive scale and attention head, SparSTAR scores contiguous key blocks from the current query and key activations, retains required conditioning context, and executes the selected blocks through a forward-only sparse path. We analyze cross-scale consistency within a clip, pattern persistence across clip boundaries, and quality degradation as reuse spans increasingly distant scales. Across these analyses, important key blocks shift, showing that recomputing block selection at each target scale is more reliable than reusing a transferred mask. On 720p text-to-video and image-to-video generation, SparSTAR preserves every token and refinement scale while providing about a 1.6x end-to-end speedup and maintaining VBench and paired-output reconstruction fidelity close to dense InfinityStar.

摘要中文:

InfinityStar通过图像和视频片段的金字塔序列将视觉自回归生成扩展到视频领域。然而,其不断变化的尺度和跨视频片段的上下文使得后期尺度的注意力计算代价高昂,且从扩散模型或图像VAR模型中复用的稀疏模式并不牢靠。我们提出SparSTAR,这是一种针对该场景设计的免训练块稀疏注意力方法。在每个高代价尺度和注意力头上,SparSTAR对来自当前查询和键激活的连续关键块进行评分,保留必要的条件上下文,并通过纯前向的稀疏路径执行所选块。我们分析了视频片段内的跨尺度一致性、跨片段边界的模式持续性,以及复用跨度延伸到更远尺度时的质量退化情况。通过这些分析发现,重要的关键块会发生变化,表明在每个目标尺度上重新计算块选择比复用迁移的掩码更为可靠。在720p文本到视频和图像到视频生成任务中,SparSTAR保留了每个token和精炼尺度,同时实现了约1.6倍的端到端加速,并在VBench和成对输出重建保真度方面保持与密集InfinityStar相当的性能。

Neural Prior Estimation: Learning Class Priors from Latent Representations

2026-08-21T04:00:00autoregressive, cs.CV, cs.LG2602.17853

中文标题:神经先验估计:从潜在表示学习类别先验

作者:Masoud Yavari, Payman Moallem

摘要:

Logit adjustment corrects class imbalance using the empirical class prior. We study whether a comparable class-frequency signal can instead be learned from the network representation, without explicitly supplying class counts to the correction rule. We introduce the Neural Prior Estimator (NPE), which attaches one or more lightweight Prior Estimation Modules (PEMs) to the latent representation. Each PEM is trained with a one-way logistic objective on the ground-truth coordinate. The resulting frequency-dependent outputs are combined into an NPE estimate and used as a learned logit correction, forming NPE-LA. In a simplified scalar model, the optimum of the PEM objective is monotone in the class count and grows asymptotically as $\log N_c$, up to a slower $\log \log N_c$ term. Experiments on long-tailed CIFAR-10 and CIFAR-100 show that NPE-LA is competitive with standard logit adjustment and improves minority-class performance over CE and cRT in the reported settings. Experiments on STARE and ADE20K further show that the same idea can be used as a lightweight recalibration mechanism for dense prediction.

摘要中文:

逻辑调整利用经验类别先验来纠正类别不平衡问题。研究探讨是否可以从网络表示中学习到类似的类别频率信号,而无需显式地向纠正规则提供类别计数。研究提出了神经先验估计器(NPE),该方法将一个或多个轻量级先验估计模块(PEM)连接到潜在表示上。每个PEM使用基于真实坐标的单向逻辑目标进行训练。得到的频率依赖输出被合并为NPE估计值,并用作学习的逻辑调整,形成NPE-LA。在简化的标量模型中,PEM目标的最优值随类别计数单调增加,并渐近增长为log N_c(包含一个较慢的log log N_c项)。在长尾CIFAR-10和CIFAR-100上的实验表明,NPE-LA与标准逻辑调整具有相当的竞争力,并在所报告的设置中改善了少数类性能,优于CE和cRT。STARE和ADE20K上的实验进一步表明,同样的思路可以用作密集预测的轻量级重校准机制。

diffusion
Diffusion
20 篇论文

今日 Diffusion 相关论文呈现多领域爆发的态势,主要集中在 3D/4D 生成、视频时空一致性、医学影像重建等前沿方向。

整体趋势:

  • 3D/4D 生成爆发:多篇论文聚焦从单目视频或文本创建动态 3D/4D 内容,包括 AvatarDynamizer(动态纹理生成动态Avatar)、4DAnyone(单目视频创建4D人物)、DreamHand(从视频扩散模型恢复3D手部动作)等
  • 时空一致性突破:Stream4D 提出流式自回归扩散实现 4D 一致性;HarmoHOI 解决多视角手-物交互的 appearance-motion 和谐问题
  • 条件控制与可解释性:Mask2Real-WM 用分割掩码作为 Sim-to-Real 桥接控制可操作世界模型;Plug-in Interpretation 探讨条件注入机制
  • 医学影像应用:Flow Matching 用于 PET 图像重建,拓展了扩散/流匹配范式的医学应用场景

重点推荐论文:

  • Block3D (2608.19567) — 块式扩散显著提升文本到3D生成效率,解决高分辨率3D内容生成的计算瓶颈问题
  • 4DAnyone (2608.20335) — 从任意单目视频创建4D人物Avatar,推动动态数字人技术的实用化
  • Stream4D (2608.19556) — 首个流式自回归扩散视频模型,实现4D时空一致性,为长视频生成提供新思路
  • Flow Matching-Based PET (2608.20112) — 开创性将流匹配应用于PET图像重建,拓展扩散范式在医学影像的真实价值
  • Decoupling High/Low Frequencies (2509.05441) — 频率解耦方法提升图像生成保真度,为高质量细节重建提供新范式

Discretizing Continuous Time Series for Imputation with Masked Diffusion Training

2026-08-21T04:00:00cs.AI, cs.LG, diffusion2608.19119

中文标题:基于掩码扩散训练的连续时间序列离散化插补方法

作者:Dongbin Kim, Seungyun Lee, Geonwoo Shin, Jaewook Lee

摘要:

Time series imputation is a crucial area for reliable time series analysis, yet it remains challenging due to the complex temporal dynamics and noise of real-world data. Existing approaches, however, exhibit two limitations: missing and observed values are embedded within the same representation space without explicit structural separation, and continuous diffusion-based methods are trained to predict added noise rather than the original signal. To address these, we propose the Masked Diffusion Time-series Imputation Model (MDTIM), which leverages the training paradigm of masked diffusion model for imputation tasks. The MASK token is structurally orthogonal to valid observations, and the model directly predicts the original values, naturally aligning both the representation and the learning objective with the imputation task. To bridge the gap between discrete masked diffusion and the continuous, ordinal nature of time series, we further introduce Stochastic Discretization, which maps continuous values to ordinal-aware tokens while preserving continuous dynamics. Our experiments on diverse benchmarks confirm that MDTIM achieves superior robustness and scalability, consistently outperforming state-of-the-art deterministic and generative baselines across various missing scenarios.

摘要中文:

时间序列插补是实现可靠时间序列分析的关键领域,但由于现实数据的复杂时间动态和噪声,该任务仍具有挑战性。然而,现有方法存在两个局限性:缺失值和观测值在同一表示空间中进行嵌入,缺乏明确的知识结构分离;连续扩散方法被训练用于预测添加的噪声而非原始信号。针对这些问题,我们提出了掩码扩散时间序列插补模型(MDTIM),该模型利用掩码扩散模型的训练范式完成插补任务。MASK标记在结构上与有效观测正交,模型直接预测原始值,自然地将表示空间和学习目标与插补任务对齐。为弥合离散掩码扩散与时间序列连续、有序性质之间的差距,我们进一步提出了随机离散化方法,该方法在保留连续动态的同时将连续值映射为序数感知标记。我们在多种基准数据集上的实验证实,MDTIM具有优异的鲁棒性和可扩展性,在各种缺失场景下始终优于最先进的确定性和生成式基线方法。

Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge for Controllable Dexterous World Models

2026-08-21T04:00:00cs.AI, cs.CV, cs.LG, cs.RO, diffusion2607.04546

中文标题:Mask2Real-WM:分割掩码作为可控灵巧世界模型的模拟到真实迁移桥梁

作者:Riccardo O. Feingold, Davide Liconti, Chenyu Yang, Robert K. Katzschmann

摘要:

Action-conditioned world models allow robots to predict the future consequences of candidate actions without additional physical interaction, supporting policy evaluation, planning, and data augmentation. We present Mask2Real-WM, a two-stage action-conditioned world model for dexterous manipulation that decouples pixel prediction into a dynamics model and a rendering model. The dynamics model predicts future segmentation masks from past masks and 23-DoF action sequences. The rendering model maps the predicted masks to photorealistic RGB using a ControlNet-augmented Stable Video Diffusion backbone. The smaller sim-to-real gap in segmentation space enables the dynamics model to benefit from large-scale pretraining on over 50 h of synthetic simulation data, followed by fine-tuning on fewer than 2.5 h of real demonstrations. Experiments on a dexterous pick-and-place benchmark show that mask conditioning and simulation pretraining are both required for per-DoF action controllability across all 23 degrees of freedom. In contrast, monolithic baselines capture broad hand and end-effector trajectories but do not reliably reflect fine-grained, per-joint action effects.

摘要中文:

动作条件世界模型使机器人能够在不进行额外物理交互的情况下预测候选动作的未来后果,支持策略评估、规划和数据增强。我们提出Mask2Real-WM,这是一个用于灵巧操作的两阶段动作条件世界模型,将像素预测解耦为动力学模型和渲染模型。动力学模型根据历史分割掩码和23自由度动作序列预测未来分割掩码。渲染模型使用ControlNet增强的Stable Video Diffusion主干网络将预测掩码映射为逼真的RGB图像。分割空间中较小的模拟到真实差距使动力学模型能够受益于超过50小时合成模拟数据的大规模预训练,随后在不到2.5小时的真实演示数据上进行微调。在灵巧拾取放置基准测试上的实验表明,掩码条件化和模拟预训练对于所有23个自由度的逐自由度动作可控性都是必需的。相比之下,整体基线方法能够捕捉手部和末端执行器的整体轨迹,但无法可靠地反映细粒度的逐关节动作效果。

Measuring the Dependency Gap: Diagnosing Inter-Column Fidelity in Tabular Generative Models

2026-08-21T04:00:00cs.AI, cs.LG, diffusion2607.21636

中文标题:度量依赖差距:诊断表格生成模型中的列间保真度

作者:Jie Zhang

摘要:

Synthetic tabular data are valued for preserving not just column-wise marginals but inter-column dependency. Yet the most commonly reported certification score, a linear (logistic-regression) classifier two-sample test (C2ST), is largely blind to it: a fully-factorized baseline that destroys all inter-column dependency still appears nearly real, a known weakness we confirm on four benchmarks, while pairwise Trend penalizes the same baseline only mildly. We therefore apply a stronger, gradient-boosted C2ST and decompose its score into marginal, dependency, and numerical-categorical cross terms, each read against a zero-dependency reference and a real-data oracle. Applied to flow-matching (TabbyFlow) and diffusion (TabDiff) generators, it exposes a persistent dependency gap of the same order in both. Destroying dependency outright with every marginal intact collapses minority-class F1 by 0.38-0.61, though the generators' much smaller residual gaps do not track the shortfalls that remain. The gap is neither a structural limitation of mean-field objectives nor an artifact of sampling discretization, and a 16x capacity increase does not close it. Shrinking capacity eightfold, however, doubles it, so the measurement does respond to capacity; what remains points to the absence of direct dependency supervision.

摘要中文:

合成表格数据的价值不仅在于保持列边缘分布,更在于保持列间依赖性。然而,最常被报告的认证分数——线性(逻辑回归)分类器双样本检验(C2ST)——在很大程度上无法检测到这一点:一个完全因式分解的基线模型虽然破坏了所有列间依赖性,却仍然看起来几乎像真实数据,这一已知弱点在四个基准数据集上得到确认,而成对趋势(pairwise Trend)对该基线的惩罚却非常轻微。因此,我们应用了一种更强的梯度提升C2ST,并将其分数分解为边缘项、依赖项和数值-分类交叉项,每一项都相对于零依赖参考和真实数据 oracle 进行评估。应用于流匹配(TabbyFlow)和扩散(TabDiff)生成器,揭示了两者之间存在相同量级的持续依赖差距。在保持每个边缘分布完整的情况下彻底破坏依赖性,会使少数类F1分数下降0.38-0.61,尽管生成器的小得多的残留差距并未追踪到剩余的不足。该差距既不是平均场目标的结构性限制,也不是采样离散化的artifact,而16倍的容量增加也无法消除它。然而,将容量减少八倍会使差距翻倍,因此该度量确实响应容量变化;剩余的差距指向缺乏直接依赖监督的问题。

A Plug-in Interpretation of Conditioning in Score-Based Diffusion Models

2026-08-21T04:00:00cs.CV, diffusion2608.19504

中文标题:分数生成扩散模型中条件化的插件式解释

作者:Libo Chen, Souvik Ghosh, Teo Deveney, Chris Budd, Vinay P. Namboodiri

摘要:

We propose a conditioning mechanism for diffusion models based on multi-speed joint diffusion of the target and the condition. The mechanism learns an unconditional joint score network and enforces conditioning at inference via a plug-in correction term. The plug-in term separates the conditioning contribution from the learned unconditional dynamics, offering a transparent view of how the condition steers generation of the target distribution. Building on this, we derive explicit conditional reverse-time SDEs and approximate probability-flow ODEs, enabling principled and directly comparable conditional samplers. To reduce the induced ODE--SDE discrepancy, we introduce a log-Fokker--Planck residual regularization that improves ODE sampling quality. Experiments on conditional image generation tasks demonstrate competitive performance and support the effectiveness of the plug-in conditioning view. Additional ODE--SDE comparison experiments show that the log-Fokker--Planck residual regularization improves deterministic ODE sampling.

摘要中文:

本文提出了一种基于目标与条件多速度联合扩散的扩散模型条件化机制。该机制学习一个无条件的联合分数网络,并在推理时通过一个插件校正项来强制执行条件化。插件项将条件贡献与学习到的无条件动力学分离开来,提供了关于条件如何引导目标分布生成的透明视角。在此基础上,本文推导出了显式的条件反向时间随机微分方程(SDE)和近似的概率流常微分方程(ODE),从而实现了原则性强且可直接比较的条件采样器。为了减少由此产生的ODE-SDE差异,本文引入了一种对数福克-普朗克残差正则化方法,以提高ODE采样质量。在条件图像生成任务上的实验表明,该方法具有竞争力的性能,并验证了插件式条件化视角的有效性。额外的ODE-SDE对比实验表明,对数福克-普朗克残差正则化改善了确定性ODE采样的质量。

Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models

2026-08-21T04:00:00autoregressive, cs.AI, cs.CV, diffusion2608.19556

中文标题:Stream4D:流式自回归扩散视频模型的4D一致性

作者:Yuanhao Ban, Jiaqi Feng, Hengguang Zhou, Xiaohuan Pei, Justin Cui, Cho-Jui Hsieh

摘要:

Streaming autoregressive diffusion models enable real-time, long-horizon video generation, but their training objectives optimize local frame prediction rather than the geometry and dynamics of a coherent world: long rollouts accumulate geometric drift and degrade into static or unnatural motion. Recent bidirectional approaches address this problem using rewards signals built upon 3D Gaussian-Splatting reconstruction. However, a single rigid 3d reconstruction cannot model a dynamic scene, so this critic penalizes genuine object motion as reconstruction error and is maximized by freezing the video. This shortcut is especially detrimental in the AR setting, where each chunk can propagate an already-static configuration. In this work, we propose Stream4D, which replaces the static critic with a feed-forward 4D reconstruction reward that explicitly models scene dynamics, allowing coherent motion to receive high consistency rewards. To further guide motion magnitude and quality, we add a motion prior that rewards natural scene-flow magnitude while penalizing jitter and non-rigid artifacts. Our final recipe combines these two terms with a lightweight perceptual anchor. Across various autoregressive video backbones and various generation horizons, Stream4D improves 4D reconstruction quality, preserves motion more effectively, and achieves higher human-aligned preference. Project page: https://banyuanhao.github.io/Stream4D/

摘要中文:

流式自回归扩散模型能够实现实时、长时域视频生成,但其训练目标优化的是局部帧预测而非连贯世界的几何与动态:长时程推演会积累几何漂移,并退化为静止或不自然的运动。最近的双向方法利用基于3D Gaussian-Splatting重建的奖励信号来解决这一问题。然而,单一的刚性3D重建无法建模动态场景,因此该判别器将真实物体运动惩罚为重建误差,且通过冻结视频可获得最高奖励。这种捷径在自回归设置中尤其有害,因为每个片段都可能传播已静止的配置。本研究提出Stream4D,用前馈4D重建奖励取代静态判别器,该奖励显式建模场景动态,使连贯运动能够获得高一致性奖励。为了进一步引导运动幅度和质量,我们添加了一个运动先验,奖励自然的场景流幅度,同时惩罚抖动和非刚性伪影。我们的最终方案将这两个 term 与轻量级感知锚点相结合。在各种自回归视频骨干网络和不同生成时域下,Stream4D提升了4D重建质量,更有效地保留运动,并获得更高的类人偏好。项目主页:https://banyuanhao.github.io/Stream4D/

Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion

2026-08-21T04:00:00autoregressive, cs.CV, diffusion2608.19567

中文标题:Block3D:通过分块扩散实现高效的文本到3D生成

作者:Bowen Cui, Weijie Wang, Zeyu Zhang, Yefei He, Mingda Lin, Haoyu Zhao, Yuanyu He, Donny Y. Chen, Feng Chen, Bohan Zhuang

摘要:

While text-to-3D generation has advanced rapidly, achieving high geometric fidelity at low inference cost remains challenging. Existing text-to-3D methods either decode discrete shape tokens autoregressively or iteratively refine global 3D representations with diffusion or flow models. However, autoregressive decoding is sequential and cannot revise errors, whereas diffusion and flow-matching models repeatedly process the full representation, making high-quality generation increasingly expensive. In this paper, we propose Block3D, a block-wise diffusion framework that partitions the discrete shape-token sequence into contiguous blocks, generates the blocks autoregressively, and jointly denoises all tokens within the current block. To alleviate error accumulation, we introduce confidence-guided intra-block correction, which revises low-confidence tokens before each block is finalized. On a held-out set from TRELLIS-500K, Block3D reduces mean end-to-end generation time from 25.71 seconds to 4.99 seconds, achieving a $5.15\times$ speedup over the fine-tuned autoregressive baseline without sacrificing geometric fidelity.

摘要中文:

虽然文本到3D生成发展迅速,但在低推理成本下实现高几何保真度仍具有挑战性。现有的文本到3D方法要么自回归地解码离散形状token,要么使用扩散或流模型迭代优化全局3D表示。然而,自回归解码是顺序执行的且无法修正错误,而扩散和流匹配模型则需要反复处理完整表示,导致高质量生成的成本不断增加。本文提出Block3D,一个分块扩散框架,该框架将离散形状token序列划分为连续块,自回归地生成这些块,并在当前块内对所有token进行联合去噪。为了减轻误差累积,我们引入了置信度引导的块内修正机制,在每个块最终确定前对低置信度token进行修正。在TRELLIS-500K的保留测试集上,Block3D将平均端到端生成时间从25.71秒减少到4.99秒,相比微调后的自回归基线实现了5.15倍的加速,且未牺牲几何保真度。

When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities

2026-08-21T04:00:00cs.CV, diffusion2608.19644

中文标题:当引导超出量程:模拟存算一体非理想性下的扩散变换器重新校准

作者:Wenshuai Yao, Wenyong Zhou

摘要:

Diffusion Transformers (DiTs) incur high memory traffic and energy costs because sampling repeatedly evaluates large denoisers dominated by linear operations. Analog compute-in-memory (CIM) can alleviate these costs by executing linear operations within weight-storing memory arrays. However, CIM nonidealities perturb effective weights, with errors accumulating along the state-dependent denoising trajectory; their interaction with classifier-free guidance (CFG) remains underexplored. In this paper, we characterize the impact of analog CIM nonidealities on DiT sampling. Although conditional and unconditional predictions can each remain close to their clean counterparts, their difference (the CFG residual) is disproportionately attenuated and rotated. Identifying this residual as a controllable failure channel, we propose a retraining-free, sampler-side recalibration that adjusts only the CFG scale for a given CIM condition. Trajectory-level analysis shows that moderate recalibration strengthens the target-oriented component preserved in the distorted residual, enabling earlier commitment to a prompt-consistent semantic region. In contrast, excessive guidance amplifies the full noisy residual and degrades quality, resulting in a finite, noise-dependent optimum. Extensive experiments on PixArt-Sigma, PixArt-alpha, and DiT-XL/2 show that the optimal guidance scale increases with CIM noise. Using 30,000 samples per condition, guidance recalibration consistently restores generation quality across simulated CIM mappings, closing at least 87% of the CIM-induced FID gap at a CIM noise level of 0.20. It reduces FID from 59.22 to 20.49 on PixArt-Sigma, 72.37 to 21.12 on PixArt-alpha, and 20.89 to 6.62 on DiT-XL/2.

摘要中文:

扩散变换器(DiT)因采样过程中需反复评估主要由线性操作构成的大型去噪器,产生较高的存储访问和能耗成本。模拟存算一体(CIM)可通过在存储权重值的内存阵列内执行线性操作来缓解这些成本。然而,CIM非理想性会扰动有效权重,且误差会沿着状态依赖的去噪轨迹累积;其与无分类器引导(CFG)的相互作用仍缺乏充分探索。本文表征了模拟CIM非理想性对DiT采样的影响。尽管条件预测和无条件预测各自可能仍接近其理想状态,但它们的差值(CFG残差)却被不成比例地衰减和旋转。通过将这一残差识别为一个可控的故障通道,我们提出了一种无需重新训练的采样器端重新校准方法,仅针对给定的CIM条件调整CFG尺度。轨迹级分析表明,适度的重新校准可增强扭曲残差中保留的目标导向分量,使模型能够更早地锁定与提示一致的语义区域。相反,过度的引导会放大整个噪声残差并降低质量,导致存在一个与噪声相关的有限最优值。在PixArt-Sigma、PixArt-alpha和DiT-XL/2上的大量实验表明,最优引导尺度随CIM噪声增加而增加。在每种条件下使用30,000个样本,引导重新校准在模拟CIM映射中一致地恢复生成质量,在CIM噪声水平为0.20时至少弥补了87%的CIM诱导FID差距。它将PixArt-Sigma的FID从59.22降至20.49,PixArt-alpha从72.37降至21.12,DiT-XL/2从20.89降至6.62。

AvatarDynamizer: From Static to Dynamic Human Avatars via Generative Dynamic Textures

2026-08-21T04:00:00cs.CV, diffusion2608.19900

中文标题:AvatarDynamizer:通过生成式动态纹理从静态人类化身到动态化身

作者:Guoxing Sun, Heming Zhu, Linjie Lyu, Pascal Fua, Christian Theobalt, Marc Habermann

摘要:

For full-body avatars, modeling surface dynamics is crucial for overcoming the uncanny valley and achieving perceptual realism. Person-agnostic methods recover static 3D avatars from monocular images, videos, or text prompts, but their skeleton-driven animations lack realistic surface dynamics such as clothing wrinkles. In contrast, person-specific methods achieve high-quality rendering and realistic dynamics, but require expensive multi-view captures for each individual. Recent generalizable dynamic avatar methods struggle to embed surface dynamics, leading to either limited multi-view consistency or dynamic expressiveness. To this end, we propose AvatarDynamizer, a generative method that transforms an off-the-shelf static 3D avatar into a controllable, realistic, and multi-view-consistent 4D avatar. We introduce a novel texture-space surface-dynamics embedding and formulate avatar dynamics modeling as conditional texture generation. Our encoder--decoder representation embeds pose-dependent dynamics into dynamic texture maps, enabling compatibility with pre-trained video diffusion models while decoding them into 3D Gaussians for multi-view consistent rendering. Since existing datasets are limited in scale, sequence length, or motion diversity, we collect a large-scale multi-view dataset with long sequences covering diverse skeletal motions and surface dynamics. Experiments show that our method effectively animates static avatars with faithful surface dynamics and outperforms competing generalizable methods in visual fidelity, especially under limited dynamic training data.

摘要中文:

对于全身化身而言,建模表面动态对于克服恐怖谷效应并实现感知真实感至关重要。与人物无关的方法能够从单目图像、视频或文本提示中恢复静态3D化身,但其骨骼驱动动画缺乏真实的表面动态,如衣物褶皱。相比之下,针对特定人物的方法实现了高质量渲染和真实动态,但需要为每个个体进行昂贵的多视角采集。近期出现的通用动态化身方法难以嵌入表面动态,导致多视角一致性或动态表达能力受限。为此,我们提出了AvatarDynamizer,一种生成式方法,可将现成的静态3D化身转换为可控、逼真且多视角一致的4D化身。我们引入了一种新颖的纹理空间表面动态嵌入方法,并将化身动态建模形式化为条件纹理生成。我们的编码器-解码器表示将姿态依赖的动态嵌入动态纹理图,使其能够与预训练视频扩散模型兼容,同时将其解码为3D高斯以实现多视角一致性渲染。由于现有数据集在规模、序列长度或运动多样性方面存在局限,我们收集了一个大规模多视角数据集,包含覆盖多样骨骼运动和表面动态的长序列。实验表明,我们的方法能够有效地为静态化身赋予忠实的表面动态,并在视觉质量方面优于现有的通用方法,尤其是在动态训练数据有限的条件下。

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

2026-08-21T04:00:00autoregressive, cs.CV, diffusion2608.20208

中文标题:RoMAN-Flow: 驯服自回归归一化流用于机器人操作的离线强化学习

作者:Shaoxuan Wang, Guangting Zheng, Rui Huang, Zhipeng Tang, Sha Zhang, Jiajun Deng, Yanyong Zhang

摘要:

Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.

摘要中文:

离线强化学习利用先前收集的数据来改进机器人策略,无需进一步的环境交互。然而,现有的扩散模型和流匹配机器人策略缺乏可计算的似然值,限制了其在基于似然的离线强化学习后训练中的应用。自回归归一化流(AR-NFs)既提供了表达性的动作建模能力,又支持精确的似然估计,但在策略优化和部署过程中,其顺序采样会产生大量的采样开销。我们提出了RoMAN-Flow(基于自回归归一化流的机器人操作),这是一个离线强化学习框架,通过在两个阶段解决采样瓶颈问题,使AR-NF策略能够真正应用于机器人操作。在策略优化阶段,RoMAN-Flow采用了一种无采样的优势加权似然目标函数,该函数可以在不从自回归策略采样的情况下,为离线数据集中的高优势动作分配更高的似然值。为了实现高效部署,RoMAN-Flow将优化后的自回归策略蒸馏为一个单步动作生成器,从而实现低延迟的动作预测。在多个模拟操作基准测试和真实机器人平台上的实验表明,RoMAN-Flow在实现具有竞争力的策略性能的同时,大幅降低了推理延迟。代码可访问 https://github.com/konnyaku28/RoMAN-Flow。

Unwarping the Lens: A Physics-Grounded Approach to Video Glasses Removal

2026-08-21T04:00:00cs.CV, diffusion2608.20212

中文标题:去除镜头畸变:一种基于物理的视频眼镜去除方法

作者:Radim Spetlik, David Futschik, Radek Danecek, Feitong Tan, Ziqian Bai, Rohit Pandey, Yinda Zhang

摘要:

High-fidelity removal of eyeglasses from video is a major challenge in facial attribute editing, as the underlying facial geometry is often obscured by complex refractive distortions and view-dependent specular reflections. While large-scale generative priors have shown promise in eye-glasses removal via static image inpainting, they often lack the structural constraints necessary to maintain identity, expression, and pose, leading to visible "identity drift" in both static images and dynamic sequences. In this paper, we propose a novel transfer framework that addresses the stochastic nature of generative priors. Our pipeline first extracts high-fidelity synthetic face images from a commercial-grade generative model (Nano Banana, Gemini 3 Pro Image), regularizes them via a three-stage structural filtering process to preserve identity, expression, and pose, and finally applies physically-based simulation of lens optics during training to provide diverse, paired data. This process transfers Nano Banana's photo-realistic, multi-view knowledge into a specialized restoration architecture, JFSnet (Joint Feature-Spatial network). JFSnet integrates DINOv2-based semantic features with a convolutional decoder for spatial reconstruction, leveraging translation equivariance constraints to improve temporal consistency and high-frequency detail preservation. Evaluations on the curated Flickr-Faces-HQ (FFHQ) subset (12,163 images) show that our approach achieves high fidelity and structural accuracy, while maintaining inference speed of 27.68 FPS. In perceptual studies on CelebV-Text video sequences, our results are consistently preferred over diffusion and GAN-based baselines for ocular consistency, temporal stability, and overall restoration quality.

摘要中文:

高保真去除视频中的眼镜是面部属性编辑领域的一项重大挑战,因为底层面部几何结构往往被复杂的折射畸变和视角相关的镜面反射所遮挡。虽然大规模生成先验已通过静态图像修复在眼镜去除任务中展现出潜力,但它们往往缺乏维持身份、表情和姿态所需的结构约束,导致静态图像和动态序列中出现明显的“身份漂移”问题。本文提出了一种针对生成先验随机性特征的新型迁移框架。我们的pipeline首先从商业级生成模型(Nano Banana、Gemini 3 Pro Image)中提取高保真合成人脸图像,通过三阶段结构过滤过程对其进行正则化以保留身份、表情和姿态信息,并在训练过程中应用基于物理的镜头光学模拟来提供多样化、成对的数据。该过程将Nano Banana的照片级真实、多视角知识迁移到专门的修复架构JFSnet(联合特征-空间网络)中。JFSnet将基于DINOv2的语义特征与卷积解码器相结合用于空间重建,利用平移等变约束来提升时间一致性和高频细节保留能力。在筛选后的Flickr-Faces-HQ(FFHQ)子集(12,163张图像)上的评估表明,我们的方法实现了高保真度和结构准确性,同时保持了27.68 FPS的推理速度。在CelebV-Text视频序列的感知研究中,我们的结果在眼部一致性、时间稳定性和整体修复质量方面始终优于扩散模型和GAN基线方法。

DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery

2026-08-21T04:00:00cs.CV, diffusion2608.20308

中文标题:DreamHand:面向遮挡鲁棒第一人称3D手部动作恢复的视频扩散模型重构

作者:Yufei Liu, Xixi Wang, Hao Li, Ganlong Zhao, Kaitong Cai, Chengkai Jin, Chunxiao Liu, Jianbo Liu, Siyuan Huang, Xingang Pan, Hongsheng Li

摘要:

Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-sight gaps. Existing single-frame and windowed temporal regressors fail when hand shortly leaves the frame, while recent video diffusion models (VDMs) rely on heavy, stochastic multi-step sampling as pixel-space renderers. We instead repurpose VDM into a deterministic geometry encoder. A single forward pass over the clean latent exposes scene content beyond current observations, including occluded and out-of-sight hands. We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent Encoder and decodes them with a Bidirectional Spatiotemporal Decoder. DreamHand recovers continuous bimanual trajectories with metric placement and no external detector, while a Ray-Based Camera Solver supports a second configuration that needs no test-time camera intrinsics. Across five egocentric benchmarks, DreamHand sets a new state of the art, cutting MPJPE-p by 30% on occlusion-heavy ARCTIC and 40% on HOT3D. These gains reach 46%-61% once out-of-sight hands are included in the evaluation, offering a scalable path from everyday human video to robot manipulation data.

摘要中文:

第一人称视频为具身AI提供了可扩展的操控数据,然而由于严重的目标遮挡和频繁的视野缺失,恢复度量3D手部轨迹仍具挑战性。现有的单帧和窗口时序回归器在手部短暂离开画面时失效,而近期的视频扩散模型(VDM)依赖于沉重的、随机的多步采样方式作为像素空间渲染器。本文另辟蹊径,将VDM重构为确定性几何编码器。单次前向传播穿越干净潜向量即可揭示当前观测之外的场景内容,包括被遮挡的和视野外的手部。本文提出DreamHand,一个离线片段级框架,通过确定性干净潜向量编码器提取特征,并使用双向时空解码器进行解码。DreamHand在无需外部检测器的情况下恢复具有度量放置的连续双手轨迹,同时基于射线的相机求解器支持无需测试时相机内参的第二种配置。在五个第一人称基准测试中,DreamHand创下新标杆,在遮挡严重的ARCTIC上使MPJPE-p降低30%,在HOT3D上降低40%。一旦将视野外的手部纳入评估,这些提升幅度可达46%-61%,为从日常人类视频获取机器人操控数据提供了一条可扩展的路径。

4DAnyone: Create Anyone in 4D from a Casual Monocular Video

2026-08-21T04:00:00cs.CV, diffusion2608.20335

中文标题:4DAnyone: 从随意单目视频中创建任意人的4D形象

作者:Yudong Jin, Tao Xie, Qihang Zhang, Zehong Shen, Zhen Xu, Yujun Shen, Hujun Bao, Xiaowei Zhou, Yinghao Xu

摘要:

We present 4DAnyone, a framework for reconstructing 4D humans from an uncalibrated monocular video by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (4DGS). Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction. We identify this failure as a bounded-attention-context problem: when target views exceed the capacity of a single DiT forward pass, they must be split into groups, exposing two coupled bottlenecks. On the reference-context side, conditioning on all previously generated views grows as $O(N)$, weakening cross-view appearance guidance. On the target-context side, disjoint groups cannot directly exchange information, causing global structural drift. 4DAnyone addresses both bottlenecks with two complementary designs: Reference Context Packing (RCP) compresses growing reference views into a fixed-length mixed-resolution context with $O(1)$ reference-context complexity, while Target Context Routing (TCR) rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps. We further build the MVGameHuman dataset using our in-house game engine and combine it with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization. See our project page for video results and source code: https://4danyone.github.io.

摘要中文:

我们提出4DAnyone,这是一个从未校准的单目视频中重建4D人体的框架,通过生成重建级别的多视角一致性视频,并将其提升至4D高斯溅射(4DGS)。现有的相机控制视频扩散模型能够合成看似合理的新视角视频,但在扩展到4DGS重建所需的数十个目标视角时,无法保持一致性。我们将这一失败归因于有限注意力上下文问题:当目标视角超过单次DiT前向传播的容量时,需要将它们分组处理,由此暴露了两个相互耦合的瓶颈。在参考上下文方面,以所有先前生成的视角为条件时复杂度增长为O(N),削弱了跨视角的外观引导能力;在目标上下文方面,离散的视角组之间无法直接交换信息,导致全局结构漂移。4DAnyone通过两种互补设计解决这两个瓶颈:参考上下文打包(RCP)将增长的参考视角压缩为固定长度的混合分辨率上下文,参考上下文复杂度为O(1);目标上下文路由(TCR)在去噪过程中轮换目标视角分组,使组间在高噪声步骤时共享上下文,并在低噪声步骤时稳定细节。我们进一步使用内部游戏引擎构建了MVGameHuman数据集,并结合光场舞台和野外视频数据集进行训练。在DNA-Rendering和DyMVHumans上的实验表明,4DAnyone在新视角视频质量和下游4DGS重建方面均优于先前方法,并具有鲁棒的野外泛化能力。

Continuous Adversarial MeanFlow Transfer

2026-08-21T04:00:00cs.CV, cs.LG, diffusion2608.19540

中文标题:连续对抗性MeanFlow迁移

作者:Yara Bahram, Zahra Dehghani, M\'elodie Desbos, Eric Granger, Pablo Piantanida, Mohammadhadi Shateri

摘要:

Training fast generators on new domains with limited data remains challenging for two reasons. First, adapting a pretrained diffusion or flow model to a new domain leaves its costly multi-step sampling unaddressed, and existing acceleration methods are tied to the source parameterization--$\epsilon$, $x$, $v$, or $u$--leaving heterogeneous pretrained models with no common acceleration target. Second, while adversarial refinement is proven effective for few-step quality, it is formulated only for instantaneous-velocity flows, not for the finite-interval average velocities that MeanFlow (MF) models predict. We address both problems. We propose MeanFlow-Transfer, which maps heterogeneous source outputs into a shared velocity representation, uses it to initialize an MF generator from the source weights, and optimizes an MF objective on the target domain. This unifies adaptation and acceleration in a single training loop across a broad range of pretrained models. We then introduce Continuous Adversarial MeanFlow, a post-training stage that extends continuous adversarial flow models from instantaneous velocities to MF's finite-interval average velocities. CAMF contrasts changes in a learned potential between real and predicted interval endpoints, recovering fine detail that MF regression averages away, and reduces to the instantaneous criterion in the vanishing-interval limit. Adapting four ImageNet-based source models--DiT ($\epsilon$), SiT ($v$), JiT ($x$), iMF ($u$)--to five target domains, MF-T with CAMF matches or exceeds the fine-tuned teacher in FID and FDD at up to $125\times$ fewer Neural Function Evaluations (NFEs), while CAMF improves MF-T&x27;s few-step FID by $29\%$ on average.

摘要中文:

在有限数据的新域上训练快速生成器仍具有挑战性,原因有二。首先,将预训练的扩散模型或流模型适应新域时,其昂贵的多步采样问题仍未得到解决,现有的加速方法与源参数化(ε、x、v或u)绑定,导致异构预训练模型缺乏共同的加速目标。其次,尽管对抗性细化已被证明对少步质量有效,但它仅针对瞬时速度流设计,而不适用于MeanFlow(MF)模型预测的有限区间平均速度。我们针对这两个问题提出了解决方案。我们提出了MeanFlow-Transfer,该方法将异构源输出映射到共享的速度表示,使用它从源权重初始化MF生成器,并在目标域上优化MF目标。这在广泛的预训练模型中统一了适应和加速。随后,我们引入了连续对抗性MeanFlow,这是一种后训练阶段,将连续对抗性流模型从瞬时速度扩展到MF的有限区间平均速度。CAMF对比学习势能在真实区间端点和预测区间端点之间的变化,恢复被MF回归平均掉的精细细节,并在区间消失极限下简化为瞬时准则。在将四个基于ImageNet的源模型——DiT(ε)、SiT(v)、JiT(x)、iMF(u)——适应到五个目标域时,MF-T结合CAMF在少至125倍更少的神经函数评估(NFE)次数下,在FID和FDD上达到或超过微调的教师模型,而CAMF平均将MF-T的少步FID提高了29%。

Flow Matching-Based PET Image Reconstruction

2026-08-21T04:00:00cs.CV, diffusion, eess.IV, physics.med-ph2608.20112

中文标题:基于流匹配的正电子发射断层成像图像重建

作者:Fumio Hashimoto, Ziqian Huang, Tatsuya Yokota, Kuang Gong

摘要:

Generative models have shown strong potential for positron emission tomography (PET) image reconstruction. Although diffusion model-based reconstruction methods have demonstrated promising performance, they often require many reverse sampling steps with data-consistency updates incorporated into the sampling process. Flow matching offers an attractive alternative because it can directly estimate clean images from intermediate states, allowing data-consistency refinement to be separated from flow propagation. In this work, we proposed flow matching-based PET image reconstruction methods. We first established PET-FlowDPS by incorporating Poisson likelihood guidance with an expectation-maximization (EM)-based preconditioner into the FlowDPS framework. We then proposed a model-based PET reconstruction method that used a pretrained flow matching model as a prior, in which the flow-based prior, PET data refinement, and stochastic propagation were interpreted within an approximate Bayesian framework. Experimental results using [$^{\text{18}}\text{F}$]FDG brain PET datasets showed that the proposed method achieved better bias-variance trade-offs across different dose levels compared with other reference methods. These results demonstrated the potential of flow matching as a generative prior for quantitative PET image reconstruction.

摘要中文:

生成式模型在正电子发射断层成像(PET)图像重建中展现出强大的潜力。尽管基于扩散模型的重建方法已取得良好性能,但此类方法通常需要大量反向采样步骤,并在采样过程中融入数据一致性约束。流匹配提供了一种有吸引力的替代方案,因为它可以直接从中间状态估计清晰图像,从而将数据一致性细化与流传播分离开来。在本工作中,我们提出了基于流匹配的PET图像重建方法。我们首先建立了PET-FlowDPS,将基于期望最大化(EM)的预处理器引入泊松似然引导的FlowDPS框架。随后,我们提出了一种基于模型的PET重建方法,该方法利用预训练的流匹配模型作为先验,并在近似贝叶斯框架内解释基于流的先验、PET数据细化和随机传播。使用[¹⁸F]FDG脑PET数据集的实验结果表明,与其他参考方法相比,所提出的方法在不同剂量水平下实现了更好的偏差-方差权衡。这些结果证明了流匹配作为定量PET图像重建生成式先验的潜力。

Regressor-Guided Image Editing Shifts Emotion and Disengagement Timing in Social Media

2026-08-21T04:00:00cs.AI, cs.CV, cs.HC, diffusion2501.12289

中文标题:回归器引导的图像编辑改变社交媒体中的情绪感知与脱离时机

作者:Christoph Gebhardt, Robin Willardt, Seyedmorteza Sadat, Chih-Wei Ning, Andreas Brombach, Jie Song, Otmar Hilliges, Christian Holz

摘要:

Internet overuse is a widespread phenomenon in today's digital society. Existing interventions, such as time limits or grayscaling, often rely on restrictive controls that provoke psychological reactance and are frequently circumvented. Building on prior work showing that emotional responses mediate the relationship between content consumption and online engagement, we investigate whether regulating the emotional impact of images can reduce online use in a non-coercive manner. We introduce and systematically analyze three regressor-guided image-editing approaches, spanning low-level attribute optimization, latent style-space optimization, and diffusion-based editing. While the first two modify low-level visual features (e.g., contrast, color), the diffusion-based method enables higher-level changes (e.g., adjusting clothing, facial features). A controlled image-rating study shows that only the diffusion-based approach shifts perceived emotion toward a neutral reference without reducing perceived quality. In a follow-up social media experiment, edited images were associated with earlier disengagement among users who left the feed.

摘要中文:

互联网过度使用是当今数字社会中的普遍现象。现有干预措施,如时间限制或灰度化,往往依赖限制性控制手段,容易引发心理抗拒且容易被规避。基于先前研究表明情绪反应是内容消费与在线参与之间关系的中介变量,本研究探讨是否能以非强制性的方式调节图像的情感影响力来减少在线使用。 本研究提出并系统分析了三种回归器引导的图像编辑方法,涵盖低层属性优化、潜在风格空间优化和基于扩散模型的编辑。前两种方法修改低层视觉特征(如对比度、颜色),而扩散模型方法能够实现更高层次的改变(如调整服装、面部特征)。 受控图像评分研究表明,只有基于扩散模型的方法能够在不降低感知质量的前提下,将感知情绪转向中性参考水平。在后续的社交媒体实验中,编辑后的图像与离开信息流的用户更早脱离相关联。

Decoupling High and Low Frequencies for Faithful Image Generation with Fine Details

2026-08-21T04:00:00cs.CV, cs.LG, diffusion2509.05441

中文标题:解耦高频与低频以实现具有精细细节的逼真图像生成

作者:Tejaswini Medi, Hsien-Yi Wang, Arianna Rampini, Margret Keuper

摘要:

Latent generative models compress images into learned embeddings prior to synthesis, and the generation quality critically depends on how faithfully these embeddings preserve visual detail. We observe that while such embeddings are effective at reconstructing low frequency structure, they struggle to recover sharp high frequency details that are essential for perceptual realism. Conventional reconstruction objectives implicitly prioritize coarse structural information over high frequency content, which can lead to overly smoothed outputs and degraded visual quality in textured regions. Motivated by this observation, we propose DeBaT, a Decoupled frequency Band Tokenizer that explicitly separates the learning of low and high frequency band embeddings. This decoupling enables accurate reconstruction of fine details while preserving global coherence. Integrated into a latent diffusion based generative model, DeBaT allows for sharper and more realistic samples than previous latent tokenizers, confirming that the explicit decoupling of high and low frequency bands eases the preservation of visual details in learned embedding spaces.

摘要中文:

潜空间生成模型在合成之前将图像压缩为学习到的嵌入表示,生成质量的关键取决于这些嵌入对视觉细节的忠实保留程度。我们观察到,虽然这类嵌入在重建低频结构方面效果良好,但在恢复对感知真实感至关重要的锐利高频细节方面存在困难。传统的重建目标隐含地优先考虑粗糙的结构信息而非高频内容,这可能导致输出过度平滑,并在纹理区域出现视觉质量下降。受此观察启发,我们提出DeBaT,一个解耦频带分词器,能够明确分离低频和高频频带嵌入的学习。这种解耦能够在保持全局一致性的同时精确重建精细细节。集成到基于潜空间扩散的生成模型中,DeBaT能够生成比以往潜空间分词器更锐利、更逼真的样本,证实了高频和低频频带的显式解耦有助于在学习到的嵌入空间中保留视觉细节。

HarmoHOI: Harmonizing Appearance and 3D Motion for Multi-view Hand-Object Interaction Synthesis

2026-08-21T04:00:00cs.CV, diffusion2607.17097

中文标题:HarmoHOI:面向多视角手-物体交互合成的外观与3D运动协调

作者:Lingwei Dang, Juntong Li, Zonghan Li, Hongwen Zhang, Liang An, Wei Min, Yebin Liu, Qingyao Wu

摘要:

Hand-Object Interaction (HOI) synthesis is a cornerstone for animation production and embodied AI. Despite the strong priors of video foundation models, multi-view consistent HOI synthesis remains challenging due to complex hand motions and occlusions. We present HarmoHOI, a unified diffusion framework that jointly and harmoniously generates synchronized multi-view HOI videos and globally aligned 3D point tracks. Our core insight is that robust multi-view consistency fundamentally requires globally aligned 3D geometry and motion. To this end, we propose a Mixture of Multi-view Diffusion Transformer that co-models RGB videos and 3D point tracks. By representing point tracks as pseudo-videos, we align 3D geometric signals with the 2D latent space of foundation models, thereby minimizing the domain gap and easing adaptation of priors. To further ensure geometry consistency, we introduce Global Motion Aligning Diffusion, which refines coarse point tracks into metric-scale, globally aligned 3D trajectories. HarmoHOI enables on-the-fly co-evolution of 2D appearance and 3D motion during denoising. To overcome the scarcity of multi-view HOI data, we employ a hybrid data curriculum learning strategy that successfully transfers generic priors from single-view data to synchronized multi-view generation. Experimental results show that HarmoHOI achieves state-of-the-art performance in visual quality, motion plausibility, and multi-view geometric consistency. Project page available at https://droliven.github.io/HarmoHOI_project.

摘要中文:

手-物体交互(HOI)合成是动画制作和具身AI的核心技术。尽管视频基础模型具有强大的先验知识,但由于复杂的手部动作和遮挡,实现多视角一致的HOI合成仍然具有挑战性。我们提出了HarmoHOI,一个统一的扩散框架,能够协同、和谐地生成同步多视角HOI视频和全局对齐的3D点轨迹。我们的核心见解是,稳健的多视角一致性从根本上需要全局对齐的几何和运动。为此,我们提出了多视角扩散Transformer混合模型(Mixture of Multi-view Diffusion Transformer),共同建模RGB视频和3D点轨迹。通过将点轨迹表示为伪视频,我们将3D几何信号与基础模型的2D潜在空间对齐,从而最小化域差距并便于先验知识的适配。为进一步确保几何一致性,我们引入了全局运动对齐扩散(Global Motion Aligning Diffusion),将粗糙的点轨迹精炼为度量尺度的全局对齐3D轨迹。HarmoHOI能够在去噪过程中实现2D外观和3D运动的实时协同演化。为克服多视角HOI数据的稀缺性,我们采用了混合数据课程学习策略,成功地将单视角数据的通用先验知识迁移到同步多视角生成任务中。实验结果表明,HarmoHOI在视觉质量、运动合理性和多视角几何一致性方面达到了最先进的性能。

SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis

2026-08-21T04:00:00autoregressive, cs.CV, diffusion2608.10519

中文标题:SparSTAR: 时空自回归视频合成的稀疏注意力方法

作者:Jongbeom Lee, Hyunwoo Yu, Jincheol Yang, Jaemin Choi, Suk-Ju Kang

摘要:

InfinityStar extends visual autoregressive generation to video through a sequence of image and clip pyramids. Its changing scale and cross-clip context, however, leave late-scale attention costly and make sparse patterns reused from diffusion or image VAR models unreliable. We introduce SparSTAR, a training-free block-sparse attention method tailored to this setting. At each expensive scale and attention head, SparSTAR scores contiguous key blocks from the current query and key activations, retains required conditioning context, and executes the selected blocks through a forward-only sparse path. We analyze cross-scale consistency within a clip, pattern persistence across clip boundaries, and quality degradation as reuse spans increasingly distant scales. Across these analyses, important key blocks shift, showing that recomputing block selection at each target scale is more reliable than reusing a transferred mask. On 720p text-to-video and image-to-video generation, SparSTAR preserves every token and refinement scale while providing about a 1.6x end-to-end speedup and maintaining VBench and paired-output reconstruction fidelity close to dense InfinityStar.

摘要中文:

InfinityStar通过图像和视频片段的金字塔序列将视觉自回归生成扩展到视频领域。然而,其不断变化的尺度和跨视频片段的上下文使得后期尺度的注意力计算代价高昂,且从扩散模型或图像VAR模型中复用的稀疏模式并不牢靠。我们提出SparSTAR,这是一种针对该场景设计的免训练块稀疏注意力方法。在每个高代价尺度和注意力头上,SparSTAR对来自当前查询和键激活的连续关键块进行评分,保留必要的条件上下文,并通过纯前向的稀疏路径执行所选块。我们分析了视频片段内的跨尺度一致性、跨片段边界的模式持续性,以及复用跨度延伸到更远尺度时的质量退化情况。通过这些分析发现,重要的关键块会发生变化,表明在每个目标尺度上重新计算块选择比复用迁移的掩码更为可靠。在720p文本到视频和图像到视频生成任务中,SparSTAR保留了每个token和精炼尺度,同时实现了约1.6倍的端到端加速,并在VBench和成对输出重建保真度方面保持与密集InfinityStar相当的性能。

Guided Diffusion by Optimized Loss Functions on Relaxed Parameters for Inverse Material Design

2026-08-21T04:00:00cs.CE, cs.CV, cs.LG, diffusion2602.15648

中文标题:基于松弛参数优化损失函数的引导扩散用于逆向材料设计

作者:Jens U. Kreber, Christian Wei{\ss}enfels, Joerg Stueckler

摘要:

Inverse design problems are common in engineering and materials science. The forward direction, i.e., computing output quantities from design parameters, typically requires running a numerical simulation, such as a FEM, as an intermediate step, which is an optimization problem by itself. In many scenarios, several design parameters can lead to the same or similar output values. For such cases, multi-modal probabilistic approaches are advantageous to obtain diverse solutions. A major difficulty in inverse design stems from the structure of the design space, since discrete parameters or further constraints disallow the direct use of gradient-based optimization. To tackle this problem, we propose a novel inverse design method based on diffusion models. Our approach relaxes the original design space into a continuous grid representation, where gradients can be computed by implicit differentiation in the forward simulation. A diffusion model is trained on this relaxed parameter space in order to serve as a prior for plausible relaxed designs. Parameters are sampled by guided diffusion using gradients that are propagated from an objective function specified at inference time through the differentiable simulation. A design sample is obtained by backprojection into the original parameter space. We develop our approach for a composite material design problem where the forward process is modeled as a linear FEM problem. We evaluate the performance of our approach in finding designs that match a specified bulk modulus. We demonstrate that our method can propose multiple diverse designs within 1% relative error margin from medium to high target bulk moduli in 2D and 3D settings. We also demonstrate that the material density of generated samples can be minimized simultaneously by using a multi-objective loss function.

摘要中文:

逆向设计问题在工程和材料科学中非常普遍。正向方向,即从设计参数计算输出量,通常需要运行数值模拟(如有限元方法)作为中间步骤,而其本身就是一个优化问题。在许多场景下,多个设计参数可以导致相同或相似的输出值。对于这种情况,多模态概率方法在获取多样化解决方案方面具有优势。逆向设计的一个主要困难来源于设计空间的结构,因为离散参数或额外约束不允许直接使用基于梯度的优化方法。为了解决这个问题,我们提出了一种基于扩散模型的新型逆向设计方法。我们的方法将原始设计空间松弛为连续网格表示,在正向模拟中可以通过隐式微分计算梯度。在该松弛参数空间上训练扩散模型,以作为合理松弛设计的先验。使用从推理时指定的目标函数通过可微分模拟传播的梯度来引导扩散进行参数采样。通过反投影获得原始参数空间的设计样本。我们针对复合材料设计问题开发了我们的方法,其中正向过程建模为线性有限元问题。我们评估了所提方法在寻找匹配指定体积模量的设计方面的性能。我们证明,在二维和三维设置下,该方法可以在1%相对误差范围内为中等到高目标体积模量提出多个不同的设计。我们还证明,通过使用多目标损失函数,可以同时最小化生成样本的材料密度。

Text-based Tactile Graphics Generation for the Visually Impaired

2026-08-21T04:00:00cs.CV, cs.GR, cs.HC, diffusion2607.22674

中文标题:面向视障人士的基于文本的触觉图形生成

作者:Ruihan Gao, Joonghyuk Shin, Ava Pun, Jaesik Park, Wenzhen Yuan, Jun-Yan Zhu

摘要:

Tactile graphics are a primary medium for blind and low-vision (BLV) individuals to access non-textual information. However, they are difficult to scale or personalize. While recent generative models have revolutionized visual content creation, they are optimized for screen-based visual realism and fail to satisfy the haptic perceptual and physical fabrication constraints required for touch. We present the first integrated generative system that produces fabrication-ready 2.5D tactile graphics directly from natural language prompts, jointly generating global base geometry, fine-grained tactile surface textures, and standard-compliant braille within a unified 3D-printable representation. Our approach introduces fabrication-aware techniques, including template-guided relief generation, a fast diffusion-based text-to-texture module for high-resolution tileable normal maps, and strict base flattening to ensure tactile readability and printability, while supporting both automatic generation and interactive texture control. Extensive evaluations, together with in-person user studies with BLV participants and blindfolded sighted participants using physically 3D-printed outputs, show that participants consistently prefer our results over baselines. By extending generative graphics beyond screens to touchable reliefs, our work broadens access to generative AI for the BLV community and beyond.

摘要中文:

触觉图形是盲人和低视力(BLV)群体获取非文本信息的主要媒介。然而,它们难以扩展或个性化定制。尽管近期生成式模型革新了视觉内容创作,但这些模型针对屏幕视觉真实性进行优化,无法满足触觉感知的物理制造约束要求。我们提出了首个集成式生成系统,能够直接从自然语言提示生成可直接用于制造的2.5D触觉图形,在统一的3D可打印表示中联合生成全局基准几何形状、精细触觉表面纹理和符合标准的盲文。我们的方法引入了制造感知技术,包括模板引导的浮雕生成、基于快速扩散的文本体纹理模块(用于生成高分辨率可平铺法线贴图),以及严格的基准平面化以确保触觉可读性和可打印性,同时支持自动生成和交互式纹理控制。广泛的评估以及与BLV参与者和蒙眼视力参与者使用实际3D打印输出进行的人体研究表明,参与者始终更偏好我们的结果。通过将生成式图形从屏幕扩展到可触摸的浮雕,我们的工作拓展了生成式AI在BLV社区及其他领域的可及性。

image_compression
Image Compression
3 篇论文

今日图像压缩领域概览(2025年8月)

今日图像压缩领域呈现多元化发展趋势,三篇论文分别从无线通信物理压缩、统一轻量级图像生成、DCT域质量评估三个不同角度切入。值得关注的是,神经算子方法开始应用于无线通道的物理层压缩,体现了深度学习与传统通信领域的交叉融合。紧凑统一的图像生成模型成为新热点,旨在兼顾压缩率与生成质量。同时,研究者们继续探索更精准的质量评估指标,试图建立MSE与SSIM之间的桥梁。

  • Swift-Image: Exploring the Performance Frontier of Compact Unified Image Generation Models - 该论文提出了紧凑统一的图像生成模型,探索了性能边界,对于实现高效图像压缩与生成的一体化具有重要参考价值。
  • Estimating SSIM from MSE for DCT-Based Compressed Images via Modeling Local Error Statistics - 该工作通过建模局部误差统计,实现了从MSE到SSIM的估计,为DCT压缩图像的质量评估提供了新的方法论。
  • GCNO: Gramian Chebyshev Neural Operator for Physics-Based Compression of Wireless Channels - 该论文创新性地将Gramian Chebyshev神经算子应用于无线通道的物理基础压缩,代表了AI与通信交叉领域的前沿探索。

GCNO: Gramian Chebyshev Neural Operator for Physics-Based Compression of Wireless Channels

2026-08-21T04:00:00cs.AI, cs.IT, cs.LG, image_compression, math.IT2608.18522

中文标题:GCNO:用于无线信道物理压缩的格拉姆-切比雪夫神经算子

作者:Rafid Umayer Murshed, Shahab Hamidi-Rad, Elahe Soltanaghai, Akshay Malhotra

摘要:

Large antenna arrays allow wireless systems to serve more users and achieve higher data rates, but they also make channel feedback expensive: the receiving device must repeatedly report a large complex-valued channel matrix to the base station. Most neural compressors treat this matrix like an image and replace it with a fixed-length code that only a matched neural decoder can interpret. The message therefore does not adapt to channel complexity, and changing the antenna count typically requires retraining. We ask whether a device can instead report only the few dominant propagation paths underlying each channel. We introduce the Gramian Chebyshev Neural Operator (GCNO), a physics-based, variable-rate compressor that identifies a sample-dependent set of path directions. GCNO uses receive-transmit channel structure to locate paths, a first-order Taylor correction to refine directions that fall between grid points, and least squares to recover their complex strengths. It is trained without path labels, and the base station reconstructs the channel analytically from the transmitted path tuples rather than through a learned decoder. Across three ray-traced environments, GCNO achieves better reconstruction accuracy at the same payload - or lower payload at the same accuracy - than neural feedback baselines, and transfers to unseen antenna counts without retraining.

摘要中文:

大规模天线阵列使无线系统能够服务更多用户并实现更高数据率,但也使得信道反馈成本高昂:接收设备必须重复向基站报告大型复值信道矩阵。大多数神经压缩器将此矩阵视为图像,并用固定长度编码替代,而只有匹配的神经解码器才能解释该编码。因此,消息无法适应信道复杂性变化,且改变天线数量通常需要重新训练。我们探讨设备是否可以仅报告每个底层信道的主导传播路径。我们提出了格拉姆-切比雪夫神经算子(GCNO),这是一种基于物理的变率压缩器,能够识别与样本相关的路径方向集合。GCNO利用收发信道结构来定位路径,使用一阶泰勒校正来细化落在网格点之间的方向,并通过最小二乘法恢复其复数增益。该模型在无路径标签的情况下进行训练,基站从传输的路径元组中解析重构信道,而非通过学习到的解码器。在三个射线追踪环境中,GCNO在相同有效载荷下实现更好的重构精度,或在相同精度下使用更低的有效载荷,优于神经反馈基线,并且能够迁移到未见过的天线数量而无需重新训练。

Swift-Image: Exploring the Performance Frontier of Compact Unified Image Generation Models

2026-08-21T04:00:00cs.CV, image_compression2608.20334

中文标题:Swift-Image:探索紧凑型统一图像生成模型的性能前沿

作者:Taihang Hu, Zhao Wang, Zuan Gao, Tao Liu, Hao Yan, Zhengze Xu, Yuhang Yu, Yongchao Du, Xingjian Wang, Jun Zheng, Qinye Zhou, Zhengrui Chen, Chao Lin, Yefeng Shen, Zhengtao Wu, Ge Wu, Xiaoli Xu, Denghui Yang, Huayu Zhang, Mingzhou Zhang, Mengting Chen

摘要:

We present Swift-Image, a compact unified model for text-to-image generation, single-image editing, and multi-image editing. Our goal is to explore how far a relatively small visual generator can be pushed through systematic training engineering under a constrained computational budget. Swift-Image adopts an efficient 6B single-stream DiT and a progressive training pipeline that evolves from broad semantic coverage to higher resolution, stronger visual quality, and unified generation-editing supervision. For post-training, we employ parallel expert reinforcement learning followed by multi-teacher on-policy distillation to alleviate interference among heterogeneous objectives. We further decouple high-level reasoning from pixel-level rendering with a Prompt Enhancer that translates user requests into generator-aligned visual specifications. For efficient deployment, structural pruning and few-step distillation produce 3B and accelerated variants. Swift-Image achieves leading aggregate performance among evaluated open-source models with only 6B parameters and 243K GPU training hours; the compressed 3B model incurs nearly no loss, while few-step distillation further improves aggregate editing performance with substantially fewer sampling steps. Our study also summarizes practical lessons for architecture, data curriculum, post-training, prompt enhancement, and model compression.

摘要中文:

我们提出了Swift-Image,一个用于文本到图像生成、单图像编辑和多图像编辑的紧凑统一模型。我们的目标是探索在受限计算预算下,通过系统性训练工程能将相对较小的视觉生成器推进到何种程度。Swift-Image采用高效的6B单流DiT和渐进式训练管道,从广泛语义覆盖逐步演进到更高分辨率、更强视觉质量和统一生成-编辑监督。在后训练阶段,我们采用并行专家强化学习 followed by multi-teacher on-policy distillation,以缓解异构目标之间的相互干扰。我们进一步通过Prompt Enhancer将高层推理与像素级渲染解耦,将用户请求转换为生成器对齐的视觉规范。在高效部署方面,结构化剪枝和少步蒸馏产生了3B参数和加速变体。Swift-Image仅用6B参数和24.3万GPU训练小时即在评估的开源模型中取得了领先的聚合性能;压缩后的3B模型几乎无性能损失,而少步蒸馏进一步提升了聚合编辑性能,同时显著减少了采样步数。我们的研究还总结了架构、数据课程、后训练、提示增强和模型压缩方面的实践经验。

Estimating SSIM from MSE for DCT-Based Compressed Images via Modeling Local Error Statistics

2026-08-21T04:00:00cs.CV, cs.MM, image_compression2608.02549

中文标题:基于局部误差统计建模的DCT压缩图像SSIM从MSE估计

作者:Luc Trudeau, Maria G. Martini

摘要:

Efficient and perceptually meaningful quality assessment is a fundamental requirement for image and video processing, compression, and streaming systems. This article shows that, in the context of Discrete Cosine Transform ( DCT)-based compressed images, Structural Similarity Index ( SSIM ) can be approximated from global Peak Signal to Noise Ratio (PSNR) or Mean Square Error ( MSE) using local statistics derived only from the reference image. While prior work assumes access to local MSE, we propose two approaches to approximate local MSE by redistributing the global MSE using variance or standard-deviation-based weighting. Experiments on the Kodak and Xiph Subset1 datasets across a range of JPEG quality levels demonstrate that both approaches provide accurate and robust SSIM approximations, substantially outperforming the global MSE baseline. The proposed framework is designed to extend naturally to video, where reference-derived statistics can be amortized across multiple encodes of the same content.

摘要中文:

高效且符合人类感知的质量评估是图像视频处理、压缩和流媒体系统的基本要求。本文研究表明,在基于离散余弦变换(DCT)的压缩图像中,结构相似性指数(SSIM)可以通过仅从参考图像导出的局部统计量,从全局峰值信噪比(PSNR)或均方误差(MSE)进行近似估计。既往工作假设需要获取局部MSE,而本文提出两种方法,通过基于方差或标准差的加权重新分配全局MSE来近似局部MSE。在Kodak和Xiph Subset1数据集上针对多种JPEG质量级别的实验表明,两种方法均能提供准确且稳健的SSIM近似性能,显著优于全局MSE基线。所提框架可自然扩展到视频场景,其中从参考图像导出的统计量可在同一内容的多次编码中分摊使用。

visual_tokenizer_1d
1D Visual Tokenizer
0 篇论文

今日未找到该分类的匹配论文。

今日未找到该分类的匹配论文。

diffusion_visual_encoder
Diffusion Visual Encoder
0 篇论文

今日未找到该分类的匹配论文。

今日未找到该分类的匹配论文。