每日 arXiv 论文简报

Convolutional networks, recurrent networks, and transformers each encode different inductive biases -- locality, sequential memory, and content-dependent pairwise interaction -- and have remained mathematically distinct since their inception. We show that this fragmentation reflects not a fundamental diversity in how signals should be processed, but rather incomplete views of a single underlying mathematical object: a learnable integral transform. We introduce the Integral Transform Network (ITNet), a unified architecture built around a learnable kernel that depends jointly on positions and features. This kernel is implemented as a small neural network, specifically an MLP, that models pairwise interactions, enabling the model to adapt its behavior from data. We show that convolution, self-attention (including multi-head), and autoregressive recurrence (including LSTM, GRU, S4, and Mamba) arise as special cases under appropriate parameterizations, and that ITNet is a universal approximator of continuous operators. To make this practical, we develop tiled kernel fusion, importance-weighted Monte Carlo integration, and learned low-rank factorization, enabling efficient and scalable computation. A single ITNet architecture with a shared operator and lightweight modality-specific encoders matches or exceeds specialized baselines on ImageNet-1K , GLUE, ModelNet40, VQA\,v2 and NLVR2. The results demonstrate that a single learned interaction mechanism can recover the behavior of all three architectural families from data.

卷积网络、循环网络和Transformer各自编码了不同的归纳偏置——局部性、序列记忆和内容依赖的成对交互——自其诞生以来在数学上始终相互独立。我们表明，这种分化并非源于信号处理方式上的根本差异，而是对单一底层数学对象的不完整理解。我们提出积分变换网络（ITNet），一种围绕可学习核构建的统一架构，该核联合依赖于位置和特征。该核由一个小型神经网络实现，即多层感知机（MLP），用于建模成对交互，使模型能够从数据中自适应行为。我们证明，卷积、自注意力（包括多头形式）和自回归循环（包括LSTM、GRU、S4和Mamba）在适当的参数化下都可以作为特例导出，且ITNet是连续算子的通用近似器。为实现实际应用，我们开发了分块核融合、重要性加权蒙特卡洛积分和学习的低秩分解技术，以实现高效且可扩展的计算。配备共享算子和轻量级模态特定编码器的单一ITNet架构在ImageNet-1K、GLUE、ModelNet40、VQA v2和NLVR2数据集上的表现达到或超越了专业基线模型。结果表明，单一可学习的交互机制能够从数据中恢复三类架构家族的全部行为。

The increasing complexity of modern power systems, driven by the integration of inverter-based and distributed energy resources, challenges the reliability of conventional protection schemes and motivates the use of machine learning for protection tasks. However, published results are often difficult to compare because datasets, sensing assumptions, and decision horizons vary across studies. This paper presents a controlled comparison of machine learning models for fault classification (FC) and fault localization (FL) under identical sensing, timing, and validation conditions on a common electromagnetic transient dataset, using decision windows of 10-50 ms to reflect protection-relevant time scales. For FC, the best-performing nonlinear models achieve F1 scores above 0.98 already at 10 ms, while lower-capacity models degrade at shorter horizons but improve with longer windows, indicating that relevant fault-type information is already present in the earliest transient. For FL, the top-performing models reach a stable localization error of about 10 % of normalized line length across all evaluated horizons, while weaker models form a clearly separated second performance tier. Line-resolved analysis shows that localization accuracy varies across grid segments, indicating topology-dependent difficulty rather than insufficient temporal context alone. These findings provide a controlled reference for comparing machine learning models across two protection tasks with fundamentally different information requirements.

现代电力系统因可再生能源和分布式能源的并网而日益复杂，这对传统保护方案的可靠性提出了挑战，并推动了机器学习在保护任务中的应用。然而，已发表的研究结果往往难以比较，因为不同研究使用的数据集、传感假设和决策时域存在差异。本研究在统一的电磁暂态数据集上，在相同的传感条件、时序要求和验证条件下，对用于故障分类（FC）和故障定位（FL）的机器学习模型进行了受控对比研究，决策窗口设置为10-50毫秒，以反映保护相关的时间尺度。对于故障分类，性能最优的非线性模型在10毫秒时即可达到0.98以上的F1分数，而容量较低的模型在较短窗口时性能下降，但随着窗口延长而改善，表明相关故障类型信息在早期暂态中已经存在。对于故障定位，表现最佳的模型在所有评估时域下均达到约为归一化线路长度10%的稳定定位误差，而较弱的模型则形成明显分离的第二性能层级。线路分辨率分析表明，定位精度在不同电网区段间存在差异，表明这取决于拓扑结构的难度，而非仅仅是时间上下文不足。这些发现为两种具有根本不同信息需求的保护任务中的机器学习模型对比提供了受控参考。

Scaling robot policy learning for autonomous surgery is challenging, as expert demonstrations are expensive and in vivo exploration poses substantial safety risks. Surgical world models address this by generating realistic, action-conditioned future frames from an initial observation, but existing methods exhibit two persistent failure modes: spatial interaction incoherence, where visible instrument contact fails to induce spatially consistent tissue deformation, and temporal fidelity collapse, where prediction errors compound across autoregressive rollouts and progressively corrupt visual quality. We present SurgVista, a surgical world model that mitigates both failures through two training recipes. Deformation Consistency Regularization extracts scene-point trajectories from training videos and enforces cross-frame coherence through latent contrastive learning, strengthening physically consistent instrument-tissue dynamics. Drift Adaptation Training mitigates long-horizon drift by perturbing conditioning frames with online prediction residuals and photometric augmentations calibrated to long-horizon drift statistics, sustaining visual fidelity over extended rollouts. To enable rigorous evaluation, we further introduce SurgWorld-Bench, featuring diverse procedure types, long-range rollouts, and decoupled metrics for instrument-motion accuracy and tissue-response fidelity. Extensive experiments show that SurgVista consistently outperforms state-of-the-art methods across visual quality, temporal consistency, and interaction fidelity, with gains widening as the prediction horizon grows.

缩放机器人策略学习以实现自主手术具有挑战性，因为专家演示成本高昂，且体内探索存在重大安全风险。手术世界模型通过从初始观察生成逼真的、动作条件化的未来帧来解决这一问题，但现有方法存在两种持续的失败模式：空间交互不一致，即可见的器械接触未能诱导空间一致的组织变形；以及时间保真度崩溃，即预测误差在自回归 rollout 过程中累积，并逐渐破坏视觉质量。我们提出了 SurgVista，一个通过两种训练策略来缓解这两种失败模式的手术世界模型。变形一致性正则化从训练视频中提取场景点轨迹，并通过潜在对比学习强化跨帧一致性，从而增强物理上一致的器械-组织动力学。漂移适应训练通过用在线预测残差和针对长程漂移统计校准的光度增强来扰动条件帧，从而缓解长期漂移，在扩展 rollout 中保持视觉保真度。为了实现严格评估，我们进一步引入了 SurgWorld-Bench，其特点包括多样化的手术类型、长程 rollout 以及解耦的器械运动精度和组织响应保真度指标。大量实验表明，SurgVista 在视觉质量、时间一致性和交互保真度方面始终优于最先进方法，且随着预测视野的增长，性能优势进一步扩大。

High-quality 4D light field representation with efficient angular feature modeling is crucial for scene perception, as it can provide discriminative spatial-angular cues to identify moving targets. However, recent developments still struggle to deliver reliable angular modeling in the temporal domain, particularly in complex low-light scenes. In this paper, we propose a novel light field epipolar-plane structure image (ESI) representation that explicitly defines the geometric structure within the light field. By capitalizing on the abrupt changes in the angles of light rays within the epipolar plane, this representation can enhance visual expression in low-light scenes and reduce redundancy in high-dimensional light fields. We further propose an angular-temporal interaction network (ATINet) for light field object tracking that learns angular-aware representations from the geometric structural cues and angular-temporal interaction cues of light fields. Furthermore, ATINet can also be optimized in a self-supervised manner to enhance the geometric feature interaction across the temporal domain. Finally, we introduce a large-scale light field low-light dataset for object tracking. Extensive experimentation demonstrates that ATINet achieves state-of-the-art performance in single object tracking. Furthermore, we extend the proposed method to multiple object tracking, which also shows the effectiveness of high-quality light field angular-temporal modeling.

高质量的4D光场表示与高效的角度特征建模对于场景感知至关重要，因为它能够提供区分性的空间-角度线索来识别运动目标。然而，现有方法在时序域中仍难以实现可靠的角度建模，特别是在复杂的低光场景中。本文提出了一种新颖的光场极平面结构图像（ESI）表示方法，该方法明确刻画了光场内的几何结构。通过利用极平面内光线角度的剧烈变化，该表示方法能够增强低光场景中的视觉表达，并降低高维光场的冗余度。进一步地，本文提出了一种用于光场目标跟踪的角-时序交互网络（ATINet），该网络从光场的几何结构线索和角-时序交互线索中学习角度感知表示。此外，ATINet还可以通过自监督方式进行优化，以增强时序域中的几何特征交互。最后，本文构建了一个大规模的光场低光目标跟踪数据集。大量实验表明，ATINet在单目标跟踪任务中实现了最先进的性能。进一步地，我们将所提出的方法扩展到多目标跟踪任务，也验证了高质量光场角-时序建模的有效性。

Multimodal models for text-to-image generation have achieved strong visual fidelity, yet they remain brittle under compositional structural constraints, notably generative numeracy, attribute binding, and part-level relations. To address these challenges, we propose Shape-of-Thought (SoT), a visual CoT framework for process-supervised progressive shape assembly in the rendered 2D domain, without external engines at inference time. SoT trains a unified multimodal autoregressive model to generate interleaved textual plans and rendered intermediate states, helping the model capture shape-assembly logic without producing explicit geometric representations. Unlike text-only CoT, each decision is grounded in a rendered state, making counts, attachments, topology, and intermediate part-addition errors inspectable across the trajectory. To support this paradigm, we introduce SoT-26K, a large-scale dataset of grounded assembly traces derived from part-based CAD hierarchies, and T2S-CompBench, a benchmark for evaluating structural integrity and trace faithfulness. Fine-tuning on SoT-26K achieves 88.4% on component numeracy and 84.8% on structural topology, outperforming direct generation by +24.2 points on component numeracy and +19.3 points on structural topology. SoT establishes a transparent testbed for rendered-domain structure-aware generation. The code is available at https://github.com/yuhuo03/Shape-of-Thought.

用于文本到图像生成的多模态模型已实现较高的视觉保真度，但在组合结构约束下仍表现脆弱，尤其是在生成计数能力、属性绑定和部件级关系方面。为解决这些挑战，我们提出了思维形态（Shape-of-Thought，SoT），一个用于渲染2D域中过程监督渐进式形状组装的视觉思维链框架，无需推理时的外部引擎。SoT训练了一个统一的多模态自回归模型来生成交错的文本计划和渲染中间状态，帮助模型捕获形状组装逻辑，而无需产生显式的几何表示。与纯文本思维链不同，每个决策都基于渲染状态，使计数、附件、拓扑结构以及中间部件添加错误在整个轨迹中可被检查。为支持这一范式，我们引入了SoT-26K，一个基于部件级CAD层次结构的大规模接地组装轨迹数据集，以及T2S-CompBench，一个用于评估结构完整性和轨迹忠实度的基准。在SoT-26K上微调后，模型在组件计数上达到88.4%，在结构拓扑上达到84.8%，相比直接生成方法在组件计数上提升24.2个百分点，在结构拓扑上提升19.3个百分点。SoT为渲染域结构感知生成建立了一个透明的测试平台。代码可访问https://github.com/yuhuo03/Shape-of-Thought。

Unified Multimodal Modeling aims to integrate visual understanding and generation within a single system. However, existing approaches typically rely on two disparate visual tokenizers, which splits the representation space and hinders truly unified modeling. We propose UniAR, a unified autoregressive framework where a single discrete visual tokenizer serves as the key bridge between understanding and generation, enabling a shared context in which the model can directly interpret its own generated visual tokens without additional re-encoding. UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme, preserving both high-level semantics and low-level details while scaling the effective visual vocabulary at minimal cost. Building on this, the unified autoregressive model adopts parallel-bitwise-prediction to jointly predict spatially grouped, multi-level visual codes, substantially reducing visual sequence length and accelerating generation. Finally, a diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images. Through large-scale pre-training, followed by supervised fine-tuning and reinforcement learning, UniAR achieves state-of-the-art performance on image generation and image editing while remaining competitive on multimodal understanding benchmarks. The project page is available at https://sharelab-sii.github.io/uniar-web.

统一多模态建模旨在将视觉理解和生成集成到单一系统中。然而，现有方法通常依赖两个不同的视觉分词器，这导致表示空间分裂，阻碍了真正的统一建模。我们提出UniAR，这是一个统一的自回归框架，其中单一离散视觉分词器作为理解和生成之间的关键桥梁，使模型能够在共享上下文中直接解释自身生成的视觉 tokens，无需额外的重新编码。UniAR 采用预训练视觉编码器配合多级特征融合和无查找位量化方案，在以极低成本扩展有效视觉词表的同时，保留高层语义和低层细节。在此基础上，统一自回归模型采用并行位预测来联合预测空间分组的多级视觉编码，大幅缩短视觉序列长度并加速生成。最后，基于扩散的视觉解码器对离散视觉 tokens 进行解码，生成高保真图像。通过大规模预训练，随后进行监督微调和强化学习，UniAR 在图像生成和图像编辑任务上达到了最先进水平，同时在多模态理解基准测试中保持竞争力。项目页面见 https://sharelab-sii.github.io/uniar-web。

Style-captioned text-to-speech systems use natural language to control voice characteristics, but how individual words influence acoustic output remains unclear. Understanding this is critical for diagnosing failure modes and improving controllability in expressive TTS. We propose cross-attention attribution for speech diffusion models, adapting the DAAM framework to the speech domain for the first time, and apply it to CapSpeech-TTS. Our method extracts per-token heatmaps across 25 layers and 24 ODE steps. We analyze 3,600 (style caption, text transcript) combinations comprising 120 style captions conditioning the generation of 30 text transcripts each, revealing how caption tokens shape waveforms. Results show: (1) style tokens have lower temporal variance than content/function tokens, confirming global conditioning; (2) style attention correlates with F0 and energy; (3) style conditioning peaks in early steps and deep layers; (4) attention entropy reaches its minimum at layer 17, co-occurring with the style importance peak, indicating maximal network selectivity at the most style-critical stage. This is the first study of how natural language influences cross-attention in speech diffusion models

风格字幕文本到语音系统使用自然语言来控制语音特性，但单个词汇如何影响声学输出仍然不明确。理解这一点对于诊断失败模式及提升情感语音合成系统的可控性至关重要。我们提出了面向语音扩散模型的跨注意力归因方法，首次将DAAM框架适配到语音领域，并将其应用于CapSpeech-TTS。我们的方法提取了25层和24个ODE步骤的每标记热图。我们分析了3600个（风格字幕、文本转录）组合，包括120个风格字幕各自条件生成30个文本转录，揭示了字幕标记如何塑造波形。结果表明：（1）风格标记的时间方差低于内容标记和功能标记，证实了全局条件作用；（2）风格注意力与F0和能量相关；（3）风格条件作用在早期步骤和深层达到峰值；（4）注意力熵在第17层达到最小值，与风格重要性峰值同时出现，表明在最关键的风格阶段网络具有最大的选择性。这是首次研究自然语言如何影响语音扩散模型中跨注意力的研究。

Computational enzyme design requires generating proteins that scaffold catalytic residues and ligands, a task that demands both geometric accuracy and structural diversity from the underlying generative model. Current all-atom generators inherit expensive architectures from structure prediction, leading to high training costs and limited sample diversity. We argue that much of this complexity is unnecessary for generators, which condition on sparse geometric constraints rather than rich co-evolutionary signals. Emyx is a 140M-parameter conditional flow matching model that concentrates capacity within standard transformer blocks, replacing heavy embedding stacks with lightweight conditional representations and sparse connectivity. We additionally derive an exact reparametrisation of the flow matching interpolant into the EDM noise-level framework, bridging flow matching training efficiency with state-of-the-art sampling methods designed for diffusion models without retraining. Despite being the smallest model, Emyx outperforms both Prote\'ina-Complexa and RFdiffusion3 against the AME enzyme design benchmark across success rate under strict evaluation requiring both global fold recovery and catalytic geometry accuracy, structural novelty, scaffold diversity, and geometric validity, while training in just $682$ GPU-hours, roughly $4\times$ less than RFdiffusion3.

计算酶设计需要生成能够支架催化残基和配体的蛋白质，这一任务要求底层生成模型同时具备几何精度和结构多样性。当前的全原子生成模型继承了结构预测中昂贵的架构，导致训练成本高且样本多样性有限。我们认为，对于生成模型而言，大部分这种复杂性是不必要的，因为生成模型基于稀疏几何约束而非丰富的共进化信号进行条件生成。Emyx是一个拥有1.4亿参数的条件流匹配模型，它将计算能力集中于标准Transformer块中，用轻量级条件表示和稀疏连接替代了繁重的嵌入堆栈。此外，我们推导了流匹配插值器在EDM噪声水平框架下的精确重参数化，无需重新训练即可将流匹配的训练效率与为扩散模型设计的最先进采样方法相结合。尽管Emyx是规模最小的模型，但在AME酶设计基准测试中，它在成功率（严格评估要求同时满足全局折叠恢复和催化几何精度）、结构新颖性、支架多样性和几何有效性方面均优于Proteína-Complexa和RFdiffusion3，且仅需682 GPU小时训练时间，约为RFdiffusion3的1/4。

Diffusion models have achieved remarkable success in image and video generation and editing. While recent studies have extended these efforts toward motion editing, simultaneously transforming both motion and location-despite its practical importance-remains largely unexplored. To better understand robust motion-location editing, we first analyze the fundamental factors that degrade its quality. Based on this analysis, we propose TeleMorpher, one of the first one-shot frameworks to the best of our knowledge, for simultaneous motion-location editing. Our approach leverages motion priors, a target motion-centric video generated from an off-the-shelf model as motion-editing guidance, and the ground truth motion to enable more controllable and precise motion-location editing. Via this, our framework works as follows: (1) we first disentangle the protagonist and the background via pre-trained segmentation and inpainting models. (2) Then, we introduce a training-free pose warping that edits the protagonist's motion with the motion prior as the guidance. (3) The result of warped motion video is directly injected into a baseline motion editor during inference, mitigating the difference between source and target motions while preserving the appearance of the source video. (4) To enhance the reliability of quantitative evaluations, we propose two new LPIPS-based metrics that measure the background consistency before and after the motion editing and the fidelity of motion editing performance via measuring the difference between the extracted protagonist&x27;s skeletons from source and target videos. Experiments with in-the-wild videos and the TaiChi dataset demonstrate that TeleMorpher achieves superior performance across both quantitative and qualitative measurements (real-human evaluation), underscoring its effectiveness.

扩散模型在图像和视频生成与编辑领域取得了显著成功。虽然近期研究将工作扩展到运动编辑领域，但同时转换运动和位置——尽管具有重要的实际意义——仍然 largely未被探索。为了更好地理解稳健的运动-位置编辑，我们首先分析了导致其质量下降的基本因素。基于此分析，我们提出了TeleMorpher，据我们所知，这是首批实现同步运动-位置编辑的单样本框架之一。我们的方法利用运动先验、基于现成模型生成的以目标运动为中心的视频作为运动编辑引导，以及真实运动来实现更可控和精确的运动-位置编辑。具体而言，我们框架的工作流程如下：（1）首先通过预训练的分割和修复模型将主角与背景分离。（2）然后，我们引入一种无训练的姿态变形方法，以运动先验为指导来编辑主角的运动。（3）将变形运动视频的结果在推理阶段直接注入基线运动编辑器中，在保留源视频外观的同时减轻源运动与目标运动之间的差异。（4）为增强定量评估的可靠性，我们提出了两种基于LPIPS的新指标，分别用于衡量运动编辑前后背景的一致性，以及通过测量源视频和目标视频中提取的主角骨架差异来评估运动编辑的性能保真度。真实场景视频和TaiChi数据集上的实验表明，TeleMorpher在定量和定性测量（真人评估）方面均取得了卓越性能，突出了其有效性。

Planning under uncertainty is an essential capability for autonomous robots. The Partially Observable Markov Decision Process (POMDP) provides a powerful framework for such a capability. Although POMDP-based planning has advanced significantly, its application to real-world problems is often limited by the difficulty of obtaining faithful POMDP models. We present Vectorized Online planning wIth Learned diffusion model for POMDP Agents (VOiLA), a framework that learns task-agnostic POMDP models for online planning under uncertainty. VOiLA learns transition and observation samplers using conditional diffusion models and learns observation-likelihood models for particle-based belief updates. To enable efficient online planning, the diffusion samplers are distilled into compact feedforward generators and integrated with Vectorized Online POMDP Planner (VOPP), an online POMDP planner designed to leverage GPU parallelization. Experimental results indicate the distillation strategy reduces sampling cost by up to nearly three orders of magnitude, making learned generative POMDP models practical for online planning. Evaluation of VOiLA on three benchmark problems indicate that VOiLA achieves equal or better performance than Recurrent Soft Actor Critic while using less than 10% training data, and generalizes much better to unseen environment configurations. Physical robot evaluation indicates VOiLA uses the models learned using only simulated data and generates a policy that successfully accomplish the task in 10 of 10 runs.

不确定性规划是自主机器人的核心能力之一。部分可观测马尔可夫决策过程（POMDP）为这一能力提供了强大的理论框架。尽管基于POMDP的规划已取得显著进展，但其在实际问题中的应用往往受限于难以获得精确POMDP模型的困境。本文提出VOiLA（面向POMDP智能体的基于学习扩散模型的向量化在线规划方法），该框架能够学习任务无关的POMDP模型以实现不确定性条件下的在线规划。VOiLA利用条件扩散模型学习转移采样器和观测采样器，并学习观测似然模型以支持基于粒子的信念更新。为了实现高效的在线规划，本文将扩散采样器蒸馏为紧凑的前馈生成器，并将其与向量化在线POMDP规划器（VOPP）集成，后者是专为利用GPU并行化而设计的在线POMDP规划器。实验结果表明，蒸馏策略将采样成本降低了近三个数量级，使得学习生成的POMDP模型能够实际应用于在线规划。在三个基准问题上的评估表明，VOiLA在使用不到10%训练数据的情况下达到了与循环软Actor-Critic相当或更优的性能，并对未见过的环境配置具有更强的泛化能力。实体机器人实验表明，VOiLA仅使用模拟数据学习模型即可生成策略，在10次运行中成功完成全部任务。

Art has long stood as a pivotal expression of human creativity. Embodied artificial intelligence offers a route for generative models to participate in that creativity through physical action rather than disembodied digital content. In robotic music co-creation, it is challenging to connect semantic musical understanding with real-time and physically executable performance. We present Co-policy, a framework for human-robot musical co-creation that separates semantic intent grounding, constrained musical variation, and visuomotor execution. To ground musical semantics, Co-policy uses pre-inference semantic anchors and a fine-tuned Qwen-vl planner (F-Qwen) to transform speech, live musical seeds, and visual observations into structured co-creation plans. To support low-latency execution, Co-policy introduces a Gaussian-Mixture Visuomotor Policy (GMP), implemented as a conditional mixture-density policy that maps target notes and visual context to multimodal robot actions in a single forward pass. Unlike robotic playback systems that merely reproduce user-specified notes, Co-policy generates complementary musical responses under both musical and physical constraints. Real-robot chime experiments, ablations, and expert evaluation show improved intent alignment, execution accuracy, and response frequency over diffusion-policy and ablated baselines, supporting physically grounded action generation as a key requirement for embodied human-AI co-creation.

艺术长期以来一直是人类创造力的重要表达方式。具身人工智能为生成模型参与这种创造力提供了一条途径，通过物理动作而非脱节的数字内容进行创作。在机器人音乐协同创作中，将语义音乐理解与实时物理可执行表演联系起来是一项挑战。我们提出了Co-policy，这是一个人机音乐协同创作框架，将语义意图锚定、受限音乐变异和视觉运动执行相分离。为了锚定音乐语义，Co-policy使用预推理语义锚点和微调后的Qwen-vl规划器（F-Qwen），将语音、实时音乐种子和视觉观察转换为结构化的协同创作计划。为了支持低延迟执行，Co-policy引入了高斯混合视觉运动策略（GMP），实现为一种条件混合密度策略，在单次前向传播中将目标音符和视觉上下文映射到多模态机器人动作。与仅重现用户指定音符的机器人播放系统不同，Co-policy在音乐和物理约束下生成互补的音乐响应。真实机器人风铃实验、消融实验和专家评估表明，与扩散策略和消融基线相比，Co-policy在意图对齐、执行准确性和响应频率方面均有改善，这支持将物理基础的动作生成作为具身人类-AI协同创作的一项关键要求。

Latent Diffusion Models (LDMs) have become dominant in visual synthesis, but their quality-compute trade-off is largely constrained by the tokenizer's fixed compression ratio. Variable-length tokenizers (VLTs) promise adaptive compression by varying token counts, allowing diffusion models to flexibly balance quality and compute. However, conventional VLTs modulate length by truncating ordered token sequences, which makes token semantics depend on token position and breaks representational alignment across lengths. This leads to a cross-length shift in the latent distribution that hinders a single variable-length diffusion model from operating effectively. To address this, we propose a novel variable-length tokenizer that modulates length by merging tokens. We show that encouraging similar tokens to merge enables direct cross-length representation alignment when the diffusion transformer operates according to the merging pattern. Since conventional merging methods are data-dependent, making the merging pattern inaccessible during generation, we introduce learnable global merging, which is data-independent, to ensure compatibility with diffusion transformers. On ImageNet 256$\times$256 generation, our merging-based variable-length tokenizer integrated with a diffusion transformer achieves a superior gFID-compute trade-off compared to prior VLT methods. Code is available at this https URL

潜在扩散模型（LDMs）在视觉合成领域已成为主流，但其质量-计算权衡在很大程度上受限于分词器的固定压缩比。变长分词器（VLTs）通过调整token数量实现自适应压缩，使扩散模型能够灵活平衡质量与计算成本。然而，传统VLTs通过截断有序token序列来调节长度，导致token语义依赖于token位置，破坏了不同长度间的表示对齐。这造成潜在分布的跨长度偏移，阻碍了单一变长扩散模型的有效运作。为解决这一问题，我们提出了一种通过合并token来调节长度的新型变长分词器。研究表明，鼓励相似token合并能够实现直接的跨长度表示对齐，前提是扩散Transformer按照合并模式运作。由于传统合并方法依赖于数据，导致生成过程中无法获取合并模式，我们引入了数据无关的可学习全局合并策略，以确保与扩散Transformer的兼容性。在ImageNet 256×256生成任务中，我们基于合并的变长分词器与扩散Transformer集成后，在gFID-计算权衡方面优于现有VLT方法。代码可访问此https链接。

Makeup transfer models enable fun augmented reality (AR) experiences as well as virtual try-on (VTO) for online makeup shopping. While recent state-of-the-art diffusion based solutions such as Stable-Makeup dramatically improve the accuracy and realism of makeup transfer, they still face limitations in identity and skin color preservation, making production-level VTO for makeup shopping unrealistic. In this work, we propose MakeupMirror, a diffusion-based approach to makeup transfer that makes significant progress towards preserving facial features and skin tone. We introduce several technical innovations over Stable-Makeup: (1) integration of facial geometry conditioning with ControlNets to maintain facial fidelity; (2) region-specific makeup transfer control to enable precise makeup application across facial regions such as skin, eyes and lips; (3) skin tone-based makeup transfer modulation that prevent skin tone alteration in cross-subject transfer scenarios; and (4) integration of a Levenberg-Marquardt Langevin sampler to speed up inference while maintaining generation quality. Our experiments on CPM-Real, Makeup Wild, and (herein newly collected, more diverse) MakeupSelfies datasets show that MakeupMirror improves relative facial recognition similarity by +60%, reduces relative skin tone difference by -50% over Stable-Makeup, with a latency of 0.7s, while achieving expert acceptance rate of 94% across core facial identity preservation criteria.

妆容迁移模型支持有趣的增强现实(AR)体验以及在线化妆品虚拟试妆(VTO)。尽管最新的基于扩散模型的前沿解决方案（如Stable-Makeup）显著提升了妆容迁移的准确性和真实性，但其在身份和肤色保留方面仍存在局限性，使得化妆品在线购物所需的生产级VTO仍不切实际。本工作提出MakeupMirror，这是一种基于扩散模型的妆容迁移方法，在保留面部特征和肤色方面取得显著进展。相较于Stable-Makeup，本工作引入了若干技术创新：(1) 集成面部几何条件控制的ControlNets以保持面部保真度；(2) 区域特定的妆容迁移控制，实现面部各区域（如皮肤、眼部、唇部）的精准妆容应用；(3) 基于肤色的妆容迁移调制，防止跨主体迁移场景中的肤色改变；(4) 集成Levenberg-Marquardt Langevin采样器，在保持生成质量的同时加速推理。在CPM-Real、Makeup Wild以及（本工作中新收集的更具多样性的）MakeupSelfies数据集上的实验表明，MakeupMirror相比Stable-Makeup将相对人脸识别相似度提升+60%，将相对肤色差异降低-50%，延迟为0.7秒，同时在核心面部身份保留标准上达到94%的专家接受率。

Audio editing aims to modify specific content in an existing audio clip according to a natural language instruction while preserving the remaining acoustic content. Despite the remarkable progress of diffusion models, existing training-based editing methods mainly rely on the local inductive biases and cross-attention interaction in convolutional U-Net backbones, which often hinder long-range semantic alignment and precise understanding and localization of instructions. In contrast, diffusion transformers provide stronger global modeling and multimodal fusion, but existing editing architectures usually adopt a simple stack of MMDiT and DiT blocks. Applying joint attention over concatenated audio and text tokens in all blocks results in quadratic complexity with respect to token length. To balance editing performance and efficiency, we propose a hybrid two-stage diffusion transformer architecture for instruction-guided audio editing based on rectified flow matching. It performs joint attention over audio and text tokens to establish coarse semantic alignment at low-resolution stage, then switches to alternating joint-attention and cross-attention blocks to refine editing details at high-resolution stage. This coarse-to-fine strategy enables efficient and accurate instruction-guided audio editing. Experiments show that the proposed framework achieves notable performance gains on challenging editing tasks involving overlapping audio events and complex instructions, while substantially improving editing efficiency with a compact model.

音频编辑旨在根据自然语言指令修改现有音频片段中的特定内容，同时保留其余声学内容。尽管扩散模型取得了显著进展，但现有基于训练的编辑方法主要依赖于卷积U-Net骨干网络的局部归纳偏置和交叉注意力交互，这往往阻碍了长距离语义对齐以及对指令的精准理解与定位。相比之下，扩散变换器提供了更强的全局建模和多模态融合能力，但现有编辑架构通常采用MMDiT和DiT模块的简单堆叠。在所有模块中对拼接的音频和文本标记应用联合注意力会导致标记长度的二次复杂度。为平衡编辑性能和效率，我们提出了一种基于修正流匹配的混合两阶段扩散变换器架构用于指令引导音频编辑。该方法在低分辨率阶段对音频和文本标记执行联合注意力以建立粗粒度语义对齐，随后在高分辨率阶段切换到交替的联合注意力和交叉注意力模块来细化编辑细节。这种粗到细的策略实现了高效且精准的指令引导音频编辑。实验表明，所提出的框架在涉及重叠音频事件和复杂指令的挑战性编辑任务中取得了显著的性能提升，同时凭借紧凑的模型大幅提升了编辑效率。

Flow matching has emerged as a standard paradigm for robotic manipulation owing to its strong expressive power for modelling complex, multimodal action distributions, alongside similar approaches like diffusion policy. However, existing methods rely on discretized action chunks, making them brittle to demonstrations collected at heterogeneous control frequencies and prone to temporally inconsistent actions that degrade control stability. In this paper, we propose Frequency-Aware Flow Matching (FAFM), which outputs continuous, temporally consistent actions. To handle heterogeneous frequency input, we transform discrete action sequences into the frequency domain with the discrete cosine transform (DCT), perform flow matching over the resulting coefficients, and reconstruct continuous actions via cosine basis expansion. To generate temporally consistent actions, we regularize the first-order temporal derivative to promote smooth actions. This corresponds to a Sobolev-type constraint that suppresses high-frequency errors and discourages abrupt action changes. Our FAFM is simple, introduces no additional network parameters and applies to standalone flow-matching policies and vision-language action models. Across synthetic toy benchmark, obstacle avoidance, LapGym, and LIBERO, FAFM improves success rates, multimodal expressivity, motion smoothness, convergence speed, robustness to mechanical bias and mixed-frequency input. These gains are consistent when deployed on a real-world Franka robot. Code available at https://anonymous.4open.science/r/FAFM.

流匹配已成为机器人操作的标准范式，得益于其在建模复杂多模态动作分布方面的强大表达能力，类似于扩散策略等方法。然而，现有方法依赖于离散动作块，对异构控制频率下收集的演示数据较为脆弱，且容易产生时间不一致的动作，从而降低控制稳定性。本文提出频率感知流匹配（FAFM），用于输出连续且时间一致的动作。为处理异构频率输入，我们利用离散余弦变换（DCT）将离散动作序列变换到频域，在得到的系数上进行流匹配，并通过余弦基展开重构连续动作。为生成时间一致的动作，我们对一阶时间导数进行正则化以促进动作平滑。这对应于一种索伯罗夫型约束，能够抑制高频误差并阻止动作突变。我们的FAFM方法简单，无需引入额外网络参数，可应用于独立流匹配策略和视觉-语言动作模型。在合成玩具基准、障碍物规避、LapGym和LIBERO等任务中，FAFM提升了成功率、多模态可表达性、运动平滑度、收敛速度、对机械偏差和混合频率输入的鲁棒性。在真实Franka机器人上的部署也取得了一致的效果。代码可访问 https://anonymous.4open.science/r/FAFM。

Leveraging Vision Foundation Models (VFMs) for camera-to-LiDAR knowledge distillation offers a promising solution to the scarcity of annotated data needed to represent the immense geometric and kinematic diversity of real-world autonomous driving (AD). However, current approaches typically treat VFMs as black-box teachers, relying exclusively on frame-wise feature similarity. Consequently, they do not fully exploit the teacher's layer-wise semantic structure and global context, as well as the rich spatiotemporal information inherent in LiDAR sequences. We propose HilDA, a self-supervised pretraining framework for LiDAR backbones that better captures the semantic what and geometric where needed for driving tasks. HilDA combines hierarchical distillation comprising multi-layer distillation for progressive semantic alignment and global context distillation for scene-level semantics, with a temporal occupancy diffusion objective promoting spatiotemporal consistency. Models pre-trained with HilDA achieve state-of-the-art results on cross-modal distillation benchmarks and outperform models trained via prior distillation approaches on 3D object detection, scene flow, and semantic occupancy prediction. Code available at: https://maxiuw.github.io/hilda.

利用视觉基础模型（VFMs）进行相机到激光雷达的知识蒸馏，为解决自动驾驶中刻画真实世界巨大几何与运动多样性所需的标注数据稀缺问题提供了一种有前景的解决方案。然而，当前方法通常将VFMs视为黑盒教师，仅依赖逐帧特征相似度。因此，这些方法未能充分利用教师的逐层语义结构和全局上下文，也无法利用激光雷达序列中固有的丰富时空信息。我们提出了HilDA，这是一种用于激光雷达骨干网络的自监督预训练框架，能够更好地捕捉驾驶任务所需的语义“是什么”和几何“在哪里”。HilDA结合了分层蒸馏（包含用于渐进语义对齐的多层蒸馏和用于场景级语义的全局上下文蒸馏）与促进时空一致性的时序占用扩散目标。使用HilDA预训练的模型在跨模态蒸馏基准上取得了最先进的结果，并在3D目标检测、场景流和语义占用预测任务上优于先前蒸馏方法训练的模型。代码可访问：https://maxiuw.github.io/hilda.

Classifier guidance is a way to control diffusion generation by using a noise-conditioned classifier to steer the sampling process toward a target class. One drawback of classifier guidance is that it requires two separately trained models: a classifier and a diffusion model. We therefore study a more compact alternative in which a conventionally trained speech classifier is repurposed as the backbone for diffusion generation. Starting from a frozen noise-conditioned classifier in log-Mel space, we attach a lightweight subnetwork that reuses intermediate classifier representations and train only this subnetwork under a Denoising Score Matching objective. Our work shows that a pretrained classifier can be repurposed for conditional generation, providing an appealing bridge between discriminative modeling and conditional speech synthesis resulting in high speech quality within a single-backbone model, with reduced memory footprint and computational cost.

分类器引导是一种通过使用噪声条件分类器来引导采样过程趋向目标类别，从而控制扩散生成的方法。分类器引导的一个缺点是它需要两个分别训练的模型：一个分类器和一个扩散模型。因此，我们研究了一种更紧凑的替代方案，将常规训练的语音分类器改造为扩散生成的骨干网络。从冻结的log-Mel空间噪声条件分类器出发，我们连接了一个轻量子网络，重用中间分类器表示，并仅在这个子网络上进行去噪分数匹配目标的训练。我们的研究表明，预训练分类器可以被改造用于条件生成，从而在判别建模与条件语音合成之间搭建了一座有吸引力的桥梁，实现了单骨干模型内的高语音质量，同时降低了内存占用和计算成本。

We study conditional generation in diffusion models under hard constraints, where generated samples must satisfy prescribed events with probability one. Such constraints arise naturally in safety-critical applications and in rare-event simulation, where soft or reward-based guidance methods offer no guarantee of constraint satisfaction. Building on a probabilistic interpretation of diffusion models, we develop a principled conditional diffusion guidance framework based on Doob's h-transform, martingale representation and quadratic variation process. Specifically, the resulting guided dynamics augment a pretrained diffusion with an explicit drift correction involving the logarithmic gradient of a conditioning function, without modifying the pretrained score network. Leveraging martingale and quadratic-variation identities, we propose two novel off-policy learning algorithms based on a martingale loss and a martingale-covariation loss to estimate h and its gradient using only trajectories from the pretrained model. We provide non-asymptotic guarantees for the resulting conditional sampler in both total variation and Wasserstein distances, explicitly characterizing the impact of score approximation and guidance estimation errors. Numerical experiments demonstrate the effectiveness of the proposed methods in enforcing hard constraints and generating rare-event samples. The code of the numerical experiments can be found at https://github.com/ZhengyiGuo2002/CDG_Finance.

本文研究扩散模型在硬约束条件下的条件生成问题，其中生成样本必须以概率1满足预定事件。这类约束在安全关键应用和稀有事件模拟中自然存在，而基于软约束或奖励的引导方法无法保证约束的满足。基于扩散模型的概率学解释，本文构建了一个基于Doob's h变换、鞅表示和二次变差过程的原则性条件扩散引导框架。具体而言，所得的引导动态在预训练扩散模型基础上增加了显式的漂移校正项，涉及条件函数的对数梯度，而不改变预训练的评分网络。利用鞅和二次变差恒等式，本文提出了两种基于鞅损失和鞅协变损失的离线策略学习算法，仅使用预训练模型的轨迹来估计h及其梯度。本文对所得条件采样器在全变差距离和Wasserstein距离下提供了非渐近保证，明确刻画了评分近似误差和引导估计误差的影响。数值实验验证了所提方法在强制执行硬约束和生成稀有事件样本方面的有效性。数值实验代码可访问https://github.com/ZhengyiGuo2002/CDG_Finance。

Existing RL post-training methods for text-to-image generation usually convert the final-image reward into a single scalar advantage and apply it with the same strength to the entire generative trajectory. However, text-to-image generation naturally has temporal and spatial structure: different denoising steps are responsible for different generation stages, and the content that truly determines text alignment often appears only in part of the image. This granularity mismatch makes it difficult for policy updates to focus on the generative components that actually affect the reward. To address this issue, we propose \textbf{SpatioTemporal Adaptive Reward (STAR) Allocation} for RL post-training of text-to-image diffusion and flow models. STAR uses text-image attention inside the generative model and starts from the core content that the user truly cares about in the prompt. It constructs spatial allocation maps that dynamically vary across denoising steps and rollouts, and allocates the same group-relative advantage to more relevant latent regions with almost no additional computational overhead. STAR then applies stronger policy updates to these regions through a spatially resolved policy objective. We use Stable Diffusion 3.5 Medium as the base model and evaluate on three tasks: GenEval, OCR text rendering, and PickScore. Experimental results show that STAR improves compositional semantic alignment, text rendering, and preference optimization without changing the external reward source, achieving $\mathbf{0.9759}$, $\mathbf{0.9757}$, and $\mathbf{23.60}$ on GenEval, OCR, and PickScore, respectively.

现有的文生图生成强化学习后训练方法通常将最终图像奖励转换为单个标量优势，并以相同强度应用于整个生成轨迹。然而，文生图生成天然具有时空结构：不同的去噪步骤负责不同的生成阶段，而真正决定文本对齐的内容往往仅出现在图像的部分区域。这种粒度不匹配使得策略更新难以聚焦于实际影响奖励的生成组件。为解决这一问题，我们提出了面向文生图扩散模型和流模型的时空自适应奖励分配（STAR）方法。STAR利用生成模型内部的文本-图像注意力，从用户提示中真正关心的核心内容出发，构建了跨去噪步骤和采样 rollout 动态变化的空间分配图，并将相同的组相对优势分配给更相关的潜在区域，几乎不产生额外计算开销。随后，STAR通过空间解析的策略目标对这些区域应用更强的策略更新。我们使用 Stable Diffusion 3.5 Medium 作为基础模型，在三个任务上进行评估：GenEval、OCR 文本渲染和 PickScore。实验结果表明，STAR 在不改变外部奖励源的情况下，显著提升了组合语义对齐、文本渲染和偏好优化性能，在 GenEval、OCR 和 PickScore 上分别达到了 0.9759、0.9757 和 23.60 的分数。

Advances in radiance fields have enabled photorealistic novel view synthesis. In several domains, large-scale real-world datasets have been developed to support comprehensive benchmarking and to facilitate progress beyond scene-specific reconstruction. However, for distractor-free radiance fields, a large-scale dataset with clean and cluttered images per scene remains lacking, limiting the development. To address this gap, we introduce DF3DV-1K, a large-scale real-world dataset comprising 1,048 scenes, each providing clean and cluttered image sets for benchmarking. In total, the dataset contains 89,924 images captured using consumer cameras to mimic casual capture, spanning 128 distractor types and 161 scene themes across indoor and outdoor environments. A curated subset of 41 scenes, DF3DV-41, is systematically designed to evaluate the robustness of distractor-free radiance field methods under challenging scenarios. Using DF3DV-1K, we benchmark nine recent distractor-free radiance field methods and 3D Gaussian Splatting, identifying the most robust methods and the most challenging scenarios. Beyond benchmarking, we demonstrate an application of DF3DV-1K by fine-tuning a diffusion-based 2D enhancer to improve radiance field methods, achieving average improvements of 0.96 dB PSNR and 0.057 LPIPS on the held-out set (e.g., DF3DV-41) and the On-the-go dataset. We hope DF3DV-1K facilitates the development of distractor-free vision and promotes progress beyond scene-specific approaches. The dataset and leaderboard are available at https://johnnylu305.github.io/df3dv1k_web/.

辐射场的进展已使逼真的新视角合成成为可能。在若干领域中，研究者已开发了大规模真实场景数据集，以支持全面的基准测试并推动超越场景特定重建的发展。然而，针对无干扰辐射场，目前仍缺乏包含每场景清洁图像和杂乱图像的大规模数据集，这限制了该领域的发展。为解决这一空白，我们提出了DF3DV-1K，这是一个大规模真实场景数据集，包含1,048个场景，每个场景均提供清洁图像集和杂乱图像集用于基准测试。该数据集共包含89,924张图像，使用消费级相机拍摄以模拟随意采集，涵盖128种干扰物类型和161种场景主题，涵盖室内外环境。我们精心筛选了41个场景构成DF3DV-41子集，系统性地设计用于评估无干扰辐射场方法在挑战性场景下的鲁棒性。利用DF3DV-1K，我们对九种最新的无干扰辐射场方法及3D Gaussian Splatting进行了基准测试，识别出最鲁棒的方法和最具挑战性的场景。除基准测试外，我们还展示了DF3DV-1K的应用，即微调基于扩散的2D增强器以改进辐射场方法，在保留集（如DF3DV-41）和On-the-go数据集上实现了平均0.96 dB PSNR和0.057 LPIPS的提升。我们希望DF3DV-1K能够促进无干扰视觉领域的发展，并推动超越场景特定方法的研究进展。数据集和排行榜可访问https://johnnylu305.github.io/df3dv1k_web/。

Multi-representation diffusion models can improve visual synthesis by denoising complementary views of an image, but their performance depends critically on the asynchronous schedule that determines when each representation is denoised. We propose to learn this schedule. Our method formulates asynchronous flow matching over multiple representation spaces and uses a schedule-corrected objective that keeps each representation's local noising-time weights fixed as the schedule changes. We instantiate the schedule with a flexible parametric class that is convex and monotone by construction, and learn it using a fast joint probe with less than 1% additional training compute. On ImageNet 256x256, the learned schedule substantially improves both convergence speed and final quality under a matched 675M-parameter XL backbone. With AutoGuidance, our 200-epoch model reaches FID 1.05, matching the 800-epoch SFD-XL baseline with 4x less training. Training to 600 epochs further improves to FID 1.02, outperforming the 1B-parameter SFD-XXL result of FID 1.04 while using a smaller model. In the unguided setting, our 200-epoch model reaches FID 2.37, already below the best 800-epoch SFD-XL result (2.54) at 4x less training, and improves to FID 2.14 at 600 epochs. Code is available at https://github.com/bsq532087/LWD

多表示扩散模型通过同时对图像的互补视图进行去噪来提升视觉合成效果，但其性能高度依赖于决定各表示何时去噪的异步调度策略。我们提出学习这一调度方法。我们的方法在多个表示空间上构建异步流匹配，并使用调度校正目标函数，使每个表示的局部噪声时间权重在调度变化时保持不变。我们用一种构建设计上凸且单调的灵活参数化类来实例化调度，并使用不到1%额外训练计算的快速联合探针进行学习。在ImageNet 256x256上，学习到的调度在匹配的675M参数XL骨干网络下显著提升了收敛速度和最终质量。使用AutoGuidance，我们的200轮模型达到FID 1.05，匹敌800轮SFD-XL基线且训练量减少4倍。训练至600轮进一步提升至FID 1.02，优于1B参数的SFD-XXL模型（FID 1.04），同时使用更小的模型。在无引导设置下，我们的200轮模型达到FID 2.37，已低于最佳800轮SFD-XL结果（2.54）且训练量减少4倍，训练至600轮提升至FID 2.14。代码见https://github.com/bsq532087/LWD

This paper addresses the challenge of one-shot novel view and pose human image synthesis. The existing methods transfer the reference human image to a target pose using a set of 2D pose keypoints or synthesize human images based on generalizable human NeRF which uses human model priors to extract point-wise features. However, pose transfer based methods can not handle complex human pose using ambiguous 2D pose as the condition, while generalizable human NeRFs may be inaccurate to recover occluded/invisiable human parts without extracted reliable features. To solve these problems, we propose a novel approach for novel view and pose synthesis from a singe human image via conditional denoising diffusion model. Our diffusion model divides the novel view and pose synthesis problem into a sequence of conditional denoising steps. Specifically, to generate humans with complex and arbitrary poses, we introduce 3D human priors, i.e., 3D normal map and color prompt, as geometry and color conditions into the generation process. By transferring the reference human into the target human with a series of diffusion steps, our diffusion model enables high-quality synthesis including the occluded/invisible parts. Further, we propose a self-reconstruction based customized refinement to enhance fine details when tested on novel persons.Experimental results on different public datasets demonstrate that our approach significantly outperforms previous methods and also shows better generalization ability across datasets. The code will be made publicly available at https://github.com/Yankeegsj/3DPGDM.

本文针对单张图像新视角和姿态人体图像合成的挑战展开研究。现有方法通过一组2D姿态关键点将参考人体图像迁移到目标姿态，或基于可泛化的人体NeRF（神经辐射场）利用人体模型先验提取点级特征进行图像合成。然而，基于姿态迁移的方法无法处理以模糊2D姿态为条件的复杂人体姿态，而可泛化的人体NeRF在提取不到可靠特征时可能无法准确恢复被遮挡/不可见的人体部位。为解决这些问题，本文提出了一种基于条件去噪扩散模型从单张人体图像合成新视角和姿态的新方法。我们的扩散模型将新视角和姿态合成问题分解为一系列条件去噪步骤。具体而言，为了生成具有复杂和任意姿态的人体，我们引入3D人体先验（即3D法线图和颜色提示）作为几何和颜色条件融入生成过程。通过将参考人体在一系列扩散步骤中迁移到目标人体，我们的扩散模型能够实现高质量合成，包括遮挡/不可见部位。此外，我们提出了一种基于自重建的定制细化方法，以增强测试新人物时的细节。在不同公共数据集上的实验结果表明，我们的方法显著优于先前方法，并展现出更好的跨数据集泛化能力。代码将开源至 https://github.com/Yankeegsj/3DPGDM。

Adversarial camouflage in the physical world remains highly challenging, particularly under UAV reconnaissance where targets undergo continuous geometric changes and extreme illumination variations. Existing methods either optimize 2D digital perturbations that fail to generalize to dynamic viewpoints or produce visually unnatural textures that cannot be deployed in real scenarios. Therefore, we propose an end-to-end framework for adversarial camouflage generation that automatically produces wearable adversarial patterns and maintains stable attack performance in real physical environments with changing viewpoints, poses, and lighting conditions. Our method integrates UV-volume rendering with a diffusion-based texture generator, enabling consistent appearance under varying scales, poses, and lighting conditions. To ensure environmental realism, we propose an illumination color consistency estimator that extracts dominant background attributes and guides a natural texture loss to align the generated UV texture with the surrounding environment. A multi-scale dynamic training strategy further enhances robustness against viewpoint shifts and body deformation. Extensive experiments across multiple mainstream detectors demonstrate that our method achieves strong and stable physical attack performance while maintaining high perceptual naturalness, reducing human detection rates without introducing unnatural artifacts.

物理世界中的对抗性伪装仍然面临极大挑战，尤其是在无人机侦察场景中，目标会经历持续的几何变化和极端的光照变化。现有的方法要么优化无法推广到动态视角的2D数字扰动，要么产生在真实场景中无法部署的视觉不自然纹理。因此，我们提出了一种端到端的对抗性伪装生成框架，能够自动生成可穿戴的对抗性图案，并在视角、姿态和光照条件不断变化的真实物理环境中保持稳定的攻击性能。我们的方法将UV体渲染与基于扩散的纹理生成器相结合，使生成的外观在不同的尺度、姿态和光照条件下保持一致。为确保环境真实性，我们提出了光照颜色一致性估计器，用于提取主导背景属性并引导自然纹理损失，使生成的UV纹理与周围环境对齐。多尺度动态训练策略进一步增强了对视角变化和身体变形的鲁棒性。在多个主流检测器上的广泛实验表明，我们的方法在保持高感知自然度的同时实现了强且稳定的物理攻击性能，能够降低人类检测率且不会引入不自然的伪影。

Handwritten Mathematical Expression Generation (HMEG) is challenging due to the complex two-dimensional layouts and long-range structural dependencies of mathematical expressions. Existing methods typically rely on explicit spatial supervision, such as symbol-level bounding boxes, which incurs high annotation costs and limits scalability. In this work, we propose DiffMath, a symbol- and graph-aware latent diffusion framework that leverages the hierarchical structure inherent in LaTeX as a structural prior, eliminating the need for positional supervision. First, we design a Relational Abstract Syntax Tree (RelAST), a generation-oriented representation that distills MathML trees into compact triplet sequences [S, R, D], where each token directly encodes a symbol identity, spatial relation, or nesting depth. Second, we introduce MathVAE, which learns structure-preserving latent representations through symbol-aware and relation-aware perceptual regularization, ensuring that the latent space captures both character semantics and spatial topology. Third, MathDiT performs conditional denoising in this structured latent space, further guided by a global symbol-count prior via Adaptive Layer Normalization (AdaLN) to improve structural coherence. Experiments show that DiffMath produces structurally consistent handwritten expressions, achieves superior performance over existing methods, and improves the accuracy of downstream OCR models through synthetic data augmentation.

手写数学表达式生成（HMEG）因其复杂的二维布局和数学表达式的长程结构依赖而具有挑战性。现有方法通常依赖显式空间监督，如符号级边界框，这导致标注成本高昂且限制了可扩展性。本工作提出DiffMath，一个符号和图感知的潜在扩散框架，利用LaTeX中固有的层级结构作为结构先验，从而无需位置监督。首先，我们设计了关系抽象语法树（RelAST），这是一种生成导向的表示方法，将MathML树压缩为紧凑的三元组序列[S, R, D]，其中每个token直接编码符号标识、空间关系或嵌套深度。其次，我们提出了MathVAE，通过符号感知和关系感知的感知正则化学习保结构的潜在表示，确保潜在空间同时捕获字符语义和空间拓扑。第三，MathDiT在此结构化潜在空间中执行条件去噪，并通过自适应层归一化（AdaLN）引入全局符号计数先验来进一步提升结构一致性。实验表明，DiffMath能够生成结构一致的手写表达式，在性能上优于现有方法，并通过合成数据增强提升了下游OCR模型的准确率。

Traditional animation production relies heavily on manual drawing and iterative refinement, particularly for key-pose design, in-betweening, and character coloring. While existing animation and video generation methods have made notable progress, they typically depend on RGB boundary frames, dense frame-wise conditions, or complete sketch sequences, limiting their applicability under low-cost input conditions. We present SketchKeyAnime, a video diffusion framework for generating structurally controllable, appearance-consistent, and temporally coherent animations from sparse key-sketch inputs. Given a single reference RGB image and a few temporally indexed key sketches, SketchKeyAnime introduces a dual-branch conditioning mechanism to encode local geometric constraints alongside semantic-temporal context. It leverages Sketch Cross Attention to fuse reference image and sketch conditions with learnable gating, and incorporates an Adaptive Weighted Loss to strengthen supervision on key-sketch frames and line-art regions. Experimental results on the Aesthetic subset of Sakuga-42M show that our approach consistently outperforms representative animation interpolation and sketch-guided generation baselines. Compared to the best-performing baseline, SketchKeyAnime reduces EDMD by 31.9\% and FVD by 9.5\%, demonstrating superior sketch fidelity and temporal coherence, while achieving the best overall performance across most quantitative metrics. These results validate the proposed framework and highlight its potential for low-cost, highly controllable animation creation.

传统动画制作高度依赖手动绘制和迭代细化，特别是在关键姿态设计、中间帧生成和角色上色方面。尽管现有动画和视频生成方法已取得显著进展，但它们通常依赖于RGB边界帧、密集帧条件或完整草图序列，这在低成本输入条件下限制了其适用性。我们提出SketchKeyAnime，一个用于从稀疏关键草图输入生成结构可控、外观一致且时间连贯动画的视频扩散框架。给定单张参考RGB图像和若干时间索引关键草图，SketchKeyAnime引入双分支条件机制来编码局部几何约束以及语义-时间上下文。它利用Sketch Cross Attention通过可学习门控融合参考图像和草图条件，并采用自适应加权损失来加强对关键草图帧和线条区域的监督。Sakuga-42M的Aesthetic子集上的实验结果表明，我们的方法始终优于代表性的动画插值和草图引导生成基线方法。与最佳基线相比，SketchKeyAnime将EDMD降低31.9%，FVD降低9.5%，展现出卓越的草图保真度和时间连贯性，同时在大多数定量指标上实现了最佳整体性能。这些结果验证了所提框架的可行性，并凸显了其在低成本、高可控动画创作中的潜力。

Latent diffusion models (LDMs) enable efficient image-to-image translation but discard fine spatial details during compression, degrading downstream perception tasks. We identify two bottlenecks: the autoencoder, which loses spatial information, and the conditioning pathway, which further degrades the source signal through naive downsampling. We propose two lightweight, backbone-agnostic fixes: a Source-Conditioned Autoencoder (SCAE) that injects high-resolution source features into the decoder via skip connections, and a Learnable Guidance Encoder (LGE) that replaces naive downsampling with a learned conditioning signal. Evaluated on RGB-to-SWIR translation for driving scenes with two denoiser backbones (U-Net and DiT), our approach improves detection mAP by up to 2x over the latent diffusion baseline, with up to 3.4x gains on small objects (COCO-small, <32^2 px^2), while achieving state-of-the-art FID. We further show that FID and detection performance are poorly correlated, motivating multi-axis evaluation. Results generalise zero-shot to the public RASMD benchmark. We will publicly release test data with annotations, all checkpoints, and training code.

潜在扩散模型（LDMs）虽然能够实现高效的图像到图像转换，但在压缩过程中会丢弃细粒度的空间细节，从而降低下游感知任务的效果。我们识别出两个瓶颈：一个是自编码器在压缩过程中造成空间信息丢失，另一个是条件通路通过简单下采样进一步削弱源信号。我们提出了两种轻量级且与骨干网络无关的解决方案：源条件自编码器（SCAE），通过跳跃连接将高分辨率源特征注入解码器；以及可学习引导编码器（LGE），用学习到的条件信号替代简单的下采样。我们在驾驶场景的RGB到SWIR转换任务上使用两种去噪器骨干网络（U-Net和DiT）进行评估，结果表明我们的方法相比潜在扩散基线将检测mAP提升至2倍，在小目标（COCO-small，<32^2像素^2）上提升至3.4倍，同时达到了最先进的FID分数。我们进一步表明FID与检测性能之间相关性较弱，这推动了多维度评估的需求。该方法在公开的RASMD基准上实现了良好的零样本泛化。我们将公开发布测试数据及标注、所有模型检查点和训练代码。

Most diffusion and flow-matching generators define the prior, probability path, and prediction target in the same representation space. Latent diffusion improves efficiency by moving this path into an autoencoder latent space, but the final sample is still produced by a separately trained decoder. This separation creates a mismatch: the generator is optimized for latent-space prediction, while final quality depends on how the decoder handles generated latents that may differ from clean encoder outputs. We introduce CrossFlow, a cross-space flow formulation that maps noisy latent inputs directly to pixel-space images. The key technical step is a velocity-free one-step objective: the latent trajectory defines the training path, but the supervised prediction is an image rather than a latent displacement. This lets one model act both as a one-step latent-to-pixel generator and as a decoder replacement for latent diffusion pipelines. On class-conditional ImageNet-1k at $256\times256$, CrossFlow-XL achieves 1.62 FID with one function evaluation. Ablations show that the latent encoder and pixel-space perceptual and adversarial losses are important for fidelity. These results indicate that cross-space flow objectives can combine the efficiency of latent representations with direct pixel-space supervision, without requiring a separate decoder at inference.

大多数扩散和流匹配生成器将先验、概率路径和预测目标定义在同一表示空间中。潜在扩散通过将该路径迁移到自编码器潜在空间来提高效率，但最终样本仍由单独训练的解码器生成。这种分离造成了不匹配：生成器针对潜在空间预测进行优化，而最终质量取决于解码器如何处理可能与干净编码器输出不同的生成潜在变量。我们提出CrossFlow，这是一种跨空间流公式，可以将噪声潜在输入直接映射到像素空间图像。关键技术步骤是无速度的一步目标：潜在轨迹定义训练路径，但监督预测是图像而不是潜在位移。这使得一个模型既可以作为一步潜在到像素生成器，也可以作为潜在扩散管道的解码器替代品。在256×256分辨率的类别条件ImageNet-1k上，CrossFlow-XL通过一次函数评估达到1.62的FID。消融实验表明，潜在编码器和像素空间感知及对抗损失对保真度很重要。这些结果表明，跨空间流目标可以将潜在表示的效率与直接像素空间监督相结合，无需在推理时使用单独的解码器。

Cinematic compositing aims to integrate green-screen characters into novel environments while maintaining physical and photometric realism. Previous methods often fail to capture the complex bidirectional interactions between characters and their surroundings, which we characterize as Character-to-Environment (C2E) physical interaction and Environment-to-Character (E2C) lighting harmonization. To address this, we propose an end-to-end video diffusion framework that jointly models C2E and E2C interactions, specifically handling the challenges of interactive props. Our approach introduces a tri-mask-guided architecture with RGB-D joint denoising to ensure physically consistent interactions among the character, props, and environment. We further develop an efficient prior-driven data curation pipeline to construct high-quality relighting pairs without expensive rendering. Finally, a reference-conditioned mechanism enables controllable environment synthesis and precise prop replacement. Extensive experiments demonstrate that our framework significantly outperforms existing methods in cinematic-quality dynamic video compositing.

电影级合成旨在将绿幕角色融入新环境，同时保持物理真实感和光度真实感。以往的方法往往难以捕捉角色与周围环境之间复杂的双向交互，我们将这种交互表征为角色到环境（C2E）物理交互和环境到角色（E2C）光照协调。为解决这一问题，我们提出了一种端到端的视频扩散框架，可联合建模C2E和E2C交互，并专门处理交互道具的挑战。我们的方法引入了一种三掩码引导架构，配合RGB-D联合去噪，以确保角色、道具和环境之间的物理一致交互。我们进一步开发了一种高效的先验驱动数据整理流水线，无需昂贵渲染即可构建高质量重光照配对数据。此外，参考条件机制可实现可控的环境合成和精确的道具替换。大量实验表明，我们的框架在电影级动态视频合成任务上显著优于现有方法。

Generative artificial intelligence has the potential to improve productivity and transform the production of creative content. However, existing research indicates that image generation models are significantly influenced by biases. This work investigates the inherent biases and language-induced biases present in text-to-image models within the context of occupation-related image generation, complementing established metrics with human preference feedback. We present a comprehensive evaluation of five current text-to-image models: Midjourney v6.1, Stable Diffusion 3 Medium, DALL-E 3, Playground v2.5, and FLUX.1-dev , focusing on gender and ethnicity bias, image quality, and prompt alignment. To facilitate this evaluation, we developed the "Battle-Arena for Fair Image Synthesis" (BAFIS), a platform designed to collect human feedback on bias in generated images. Furthermore, we created a dataset comprising 21,140 synthetic images generated using multilingual prompts, which serves as a basis for our analysis. We further place our results within a broader social context by comparing them to official statistics from the German Federal Employment Agency. Our findings reveal systematic biases in text-to-image models, with established evaluation metrics in partial correlation with subjective user ratings. Thus, our research emphasizes the need for including human preferences to develop fairer and more inclusive text-to-image models.

生成式人工智能有潜力提高生产力并改变创意内容的生产方式。然而，现有研究表明，图像生成模型明显受到偏见的影响。本研究探讨了文本到图像模型在职业相关图像生成情境中固有的偏见和语言诱导的偏见，并用人类偏好反馈来补充已有的评估指标。我们对五种当前主流文本到图像模型进行了全面评估：Midjourney v6.1、Stable Diffusion 3 Medium、DALL-E 3、Playground v2.5和FLUX.1-dev，重点关注性别和种族偏见、图像质量以及提示词对齐程度。为实现这一评估，我们开发了“公平图像合成竞技场”（Battle-Arena for Fair Image Synthesis，简称BAFIS），这是一个专门用于收集生成图像偏见相关人类反馈的平台。此外，我们构建了一个包含21,140张合成图像的数据集，这些图像使用多语言提示词生成，为我们的分析提供了基础。我们通过与德国联邦劳工局的官方统计数据进行比较，将研究结果置于更广泛的社会情境中。研究结果显示，文本到图像模型存在系统性偏见，且已有评估指标与主观用户评分部分相关。因此，本研究强调需要在开发更公平、更具包容性的文本到图像模型时纳入人类偏好。

Few-shot anomaly detection remains challenging due to limited training data. Multi-modal anomaly detection (MAD) offers a viable solution, leveraging 3D geometric cues to enrich 2D RGB representations and compensate for this scarcity. However, existing MAD methods apply spatially uniform feature processing, conflating stable macroscopic structures with high-frequency localized defect signals, exacerbating cross-modal misalignment and inflating false-positive rates. To overcome this, we present CMDS-AD, a Cross-Modal Dual-Stream Anomaly Detection framework. A LoRA-guided diffusion model generates diverse RGB samples to mitigate extreme data scarcity. For 3D normal augmentation, we employ a pre-trained diffusion model as a normal estimator. Crucially, this estimator inherently acts as a non-linear low-pass filter, directly extracting low-frequency normal representations from RGB inputs. This establishes an auxiliary estimated stream of purely low-frequency information, anchoring robust structural templates and assisting the uncompressed real stream, containing coupled high- and low-frequency components, to precisely isolate micro-defects. A Coordinate-Aware Hierarchical Feature Mapper adaptively aligns cross-modal semantics, while a multiplicative scoring mechanism filters modality-specific noise. Under the extreme 1-shot setting, CMDS-AD achieves absolute performance gains of 5.7% (I-AUROC) and 2.0% (AUPRO) on MVTec 3D-AD, alongside 7.7% and 5.6% improvements on EyeCandies, establishing a new state-of-the-art.

少样本异常检测由于训练数据有限而仍然具有挑战性。多模态异常检测（MAD）提供了一种可行的解决方案，利用3D几何线索丰富2D RGB表示，以弥补数据的稀缺。然而，现有的MAD方法采用空间均匀的特征处理，将稳定的宏观结构与高频局部缺陷信号混为一谈，加剧了跨模态对齐问题并推高了假阳性率。为克服这一问题，我们提出了CMDS-AD，一个跨模态双流异常检测框架。LoRA引导的扩散模型生成多样化的RGB样本以缓解极端数据稀缺问题。对于3D法向量增强，我们采用预训练的扩散模型作为法向量估计器。关键在于，该估计器本身作为一种非线性低通滤波器，直接从RGB输入中提取低频法向量表示。这建立了一个仅包含低频信息的辅助估计流，作为稳健的结构模板，并协助包含耦合高低频分量的非压缩真实流精确分离微缺陷。坐标感知层次特征映射器自适应地对齐跨模态语义，而乘法评分机制则过滤模态特定噪声。在极端的1样本设置下，CMDS-AD在MVTec 3D-AD上实现了5.7%（I-AUROC）和2.0%（AUPRO）的绝对性能提升，同时在EyeCandies上分别提升了7.7%和5.6%，确立了新的最先进水平。

Evaluating video generation with clean, pixel-based reward models disconnects evaluation from the noisy diffusion process and incurs massive VAE decoding costs. In this paper, we challenge this paradigm by asking a fundamental question: Can a powerful video generator inherently discriminate preferences directly from noisy latents? To answer this, we introduce \textbf{PRISM} (\textbf{P}reference \textbf{R}epresentation in \textbf{I}ntermediate \textbf{S}tates of Diffusion \textbf{M}odels). PRISM employs a lightweight Query-based Aggregation head with a frozen video diffusion backbone to decode preference signals from noisy latents. Surprisingly, PRISM not only achieves SOTA preference accuracy but also unlocks strong noise-robustness, which enables early-stage Best-of-$N$ sampling. This allows for filtering suboptimal candidates at the very beginning of denoising, drastically reducing computation while boosting video quality. We also reveal a strong positive correlation between a backbone's generative performance and its inherent evaluative power, enabling self-improving video backbones.

使用基于干净像素的奖励模型评估视频生成会将评估与噪声扩散过程分离，并产生巨大的VAE解码成本。本文通过提出一个基本问题来挑战这一范式：强大的视频生成器能否直接从噪声潜变量中固有地辨别偏好？为此，我们引入了PRISM（Preference Representation in Intermediate States of Diffusion Models，扩散模型中间状态的偏好表示）。PRISM采用一个轻量级的基于查询的聚合头，配合冻结的视频扩散骨干网络，从噪声潜变量中解码偏好信号。令人惊讶的是，PRISM不仅实现了最先进的偏好准确率，还解锁了强大的噪声鲁棒性，从而实现了早期阶段的最优选择采样。这允许在去噪开始时过滤次优候选，在大幅降低计算量的同时提升视频质量。我们还揭示了骨干网络生成性能与其固有评估能力之间存在强正相关关系，从而实现了自我改进的视频骨干网络。

Conditional diffusion and flow models routinely fail to satisfy the very constraints that define their task. For instance, a depth-conditioned model often produces images whose re-extracted depth disagrees with the input, even though the forward operator--the depth predictor defining the constraint--is available during both training and inference. Existing approaches generally fall into two categories: supervised models that treat the conditioning signal as a static cue and ignore alignment information at inference, and guidance-based methods that consult it through hand-tuned linear updates, typically trading fidelity to the condition against the plausibility of the generated sample. We argue that the fundamental gap in both paradigms is that the model is never trained to utilize its own alignment error. We introduce FlowBender, a closed-loop framework that treats this error as a first-class input, training the network to learn a correction policy conditioned on inference-time feedback. At each step, an unguided look-ahead pass estimates the clean signal, a task-specific deviation is computed via the forward operator, and a refinement pass consumes this signal to produce a corrected velocity. We propose several variants of FlowBender, including a gradient-based formulation for differentiable operators and a zero-order variant for non-differentiable settings such as JPEG compression. For efficient sampling, we introduce a prior-step shortcut that enables closed-loop correction at a minimal additional computational cost. Across image-to-image translation, restoration, and 3D mesh texturing, FlowBender consistently outperforms standard supervised baselines, alignment-loss-augmented training, and state-of-the-art inference-time guidance, improving fidelity and plausibility simultaneously rather than trading them against each other. Project page: https://flow-bender.github.io/

条件扩散模型和流模型经常无法满足定义其任务的约束条件。例如，即使前向算子（即定义约束的深度预测器）在训练和推理阶段都可用，深度条件模型也经常产生重新提取的深度与输入不一致的图像。现有方法通常分为两类：监督模型将条件信号视为静态线索，在推理时忽略对齐信息；基于引导的方法通过手动调整的线性更新来利用约束，通常在满足条件保真度与生成样本合理性之间进行权衡。我们认为，这两种范式的根本问题在于模型从未被训练利用自身的对齐误差。我们提出FlowBender，一种闭环框架，将该误差视为一等输入，训练网络学习以推理时反馈为条件的修正策略。在每一步中，无引导的前瞻传递估计干净信号，通过前向算子计算任务特定偏差，精化传递利用该信号产生修正后的速度。我们提出了FlowBender的多种变体，包括面向可微分算子的基于梯度形式，以及面向JPEG压缩等不可微分设置的零阶变体。为实现高效采样，我们引入前一步捷径，使闭环修正以极低的额外计算成本实现。在图像到图像转换、修复和3D网格纹理化任务上，FlowBender始终优于标准监督基线、对齐损失增强训练以及最先进的推理时引导方法，同时改善保真度和合理性，而非在两者之间进行权衡。

Creating 3D visual illusions, a single 3D mesh that reveals entirely different semantics from various viewing angles, is a fascinating but tough challenge. Existing optimization-based methods are slow and can produce oversaturated colors. In contrast, naive stitching approaches fail to produce geometrically coherent objects. This results in visible unnatural seams and semantic leaks. In this paper, we present a fast and training-free framework for generating text-driven 3D visual illusions. Our approach decouples the generation into two stages. First, we propose a cross-space dual-branch denoising process. This process dynamically decodes 3D latents into voxel space for CLIP-guided orientation alignment and Signed Distance Field (SDF) blending, which ensures seamless geometric fusion. Second, we introduce a view-conditioned texture synthesis module that projects and aggregates view-specific 2D diffusion priors onto the fused geometry. Extensive experiments demonstrate that our method generates highly realistic, dual-semantic 3D illusions in just 3-5 minutes. It significantly outperforms existing methods in geometric integrity, semantic recognizability, and efficiency. Project page: https://siang1105.github.io/JanusMesh.github.io/

创建3D视觉错觉，即从不同视角呈现完全不同语义的单一3D网格，是一项引人入胜但极具挑战性的任务。现有的基于优化的方法速度较慢，且可能产生过度饱和的颜色。相比之下，简单拼接的方法无法生成几何上一致的对象，导致明显的接缝和语义泄漏。本文提出了一种快速且无需训练的文本驱动3D视觉错觉生成框架。我们的方法将生成过程解耦为两个阶段。首先，我们提出了跨空间双分支去噪过程，该过程将3D潜在变量动态解码到体素空间中进行CLIP引导的方向对齐和符号距离场（SDF）混合，以确保无缝的几何融合。其次，我们引入了视图条件纹理合成模块，将视图特定的2D扩散先验投影并聚合到融合的几何体上。大量实验表明，我们的方法仅需3-5分钟即可生成高度逼真的双语义3D错觉。它在几何完整性、语义可识别性和效率方面均显著优于现有方法。项目主页：https://siang1105.github.io/JanusMesh.github.io

Diffusion models rely heavily on explicit timestep embeddings to modulate the denoising process across various noise scales. In this work, we challenge the necessity of these temporal signals by analyzing their impact on U-Net and Diffusion Transformer architectures. Beyond empirical evidence, we provide a theoretical framework demonstrating that, under certain conditions, the global minimizer of the diffusion training objective can be achieved without explicit timestep conditioning. Our findings reveal a surprising robustness when timestep embeddings are completely removed. Extensive ablation studies on the CelebA and CIFAR-10 datasets show that these time-agnostic models can maintain high structural fidelity and even surpass their conditioned counterparts in competitive metrics, including FID, precision, and recall. Our analysis suggests these architectures can implicitly infer noise scales from the corrupted input under specific assumptions, rendering explicit temporal conditioning redundant. This study challenges long-standing temporal conditioning paradigms and paves the way for more efficient and structurally focused generative architectures.

扩散模型严重依赖显式时间步嵌入来调节不同噪声尺度下的去噪过程。在本工作中，我们通过分析其对U-Net和Diffusion Transformer架构的影响来挑战这些时间信号的必要性。除了实证证据外，我们还提供了一个理论框架，证明在某些条件下，扩散训练目标的全局最优解可以在不进行显式时间步条件化的情况下实现。我们的发现揭示了完全移除时间步嵌入时令人惊讶的鲁棒性。在CelebA和CIFAR-10数据集上的广泛消融实验表明，这些时间无关模型能够保持较高的结构保真度，甚至在FID、精确率和召回率等竞争性指标上超越其条件化对应物。我们的分析表明，在特定假设下，这些架构可以从被破坏的输入中隐式推断噪声尺度，从而使得显式的时间条件化变得冗余。本研究挑战了长期存在的时间条件化范式，并为更高效、更有利于结构的生成架构开辟了道路。

Unsupervised learning of latent motion from Internet videos is crucial for robot learning. Existing discrete methods generally mitigate the shortcut learning caused by extracting excessive static backgrounds through vector quantization with a small codebook size. However, they suffer from information loss and struggle to capture more complex and fine-grained dynamics. Moreover, there is an inherent gap between the distribution of discrete latent motion and continuous robot action, which hinders the joint learning of a unified policy. We propose CoMo, which aims to learn more precise continuous latent motion from internet-scale videos. CoMo employs an early temporal difference (Td) mechanism to increase the shortcut learning difficulty and explicitly enhance motion cues. Additionally, to ensure latent motion better captures meaningful foregrounds, we further propose a temporal contrastive learning (Tcl) scheme. Specifically, positive pairs are constructed with a small future frame temporal offset, while negative pairs are formed by directly reversing the temporal direction. The proposed Td and Tcl work synergistically and effectively ensure that the latent motion focuses better on the foreground and reinforces motion cues. Critically, CoMo exhibits strong zeroshot generalization, enabling it to generate effective pseudo action labels for unseen videos. Extensive simulated and real-world experiments show that policies co-trained with CoMo pseudo action labels achieve superior performance with both diffusion and auto-regressive architectures.

从互联网视频中进行潜在运动的无监督学习对机器人学习至关重要。现有离散方法通常通过小规模码本的矢量量化来缓解因提取过多静态背景而导致的捷径学习问题。然而，这些方法存在信息丢失问题，难以捕捉更复杂、更细粒度的动态特征。此外，离散潜在运动与连续机器人动作之间存在固有的分布差距，阻碍了统一策略的联合学习。我们提出了CoMo，旨在从互联网规模的视频中学习更精确的连续潜在运动。CoMo采用早期时间差分（Td）机制来增加捷径学习难度并显式增强运动线索。此外，为确保潜在运动更好地捕捉有意义的前景，我们进一步提出了时间对比学习（Tcl）方案。具体而言，正样本对通过小量未来帧时间偏移构建，而负样本对通过直接反转时间方向形成。提出的Td和Tcl协同工作，有效确保潜在运动更好地聚焦于前景并强化运动线索。关键的是，CoMo展现出强大的零样本泛化能力，能够为未见过的视频生成有效的伪动作标签。大量模拟和真实世界实验表明，使用CoMo伪动作标签协同训练的策略在扩散和自回归架构上均取得了优异的性能。

3D content inherently encompasses multi-modal characteristics and can be projected into different modalities (e.g., RGB images, RGBD, and point clouds). Each modality exhibits distinct advantages in 3D asset modeling: RGB images contain vivid 3D textures, whereas point clouds define fine-grained 3D geometries. However, most existing 3D-native generative architectures either operate predominantly within single-modality paradigms-thus overlooking the complementary benefits of multi-modality data-or restrict themselves to 3D structures, thereby limiting the scope of available training datasets. To holistically harness multi-modalities for 3D modeling, we present TriMM, the first feed-forward 3D-native generative model that learns from basic multi-modalities (e.g., RGB, RGBD, and point cloud). Specifically, 1) TriMM first introduces collaborative multi-modal coding, which integrates modality-specific features while preserving their unique representational strengths. 2) Furthermore, auxiliary 2D and 3D supervision are introduced to raise the robustness and performance of multi-modal coding. 3) Based on the embedded multi-modal code, TriMM employs a triplane latent diffusion model to generate 3D assets of superior quality, enhancing both the texture and the geometric detail. Extensive experiments on multiple well-known datasets demonstrate that TriMM, by effectively leveraging multi-modality, achieves competitive performance with models trained on large-scale datasets, despite utilizing a small amount of training data. Furthermore, we conduct additional experiments on recent RGB-D datasets, verifying the feasibility of incorporating other multi-modal datasets into 3D generation.

3D内容本身具有多模态特性，可以投影到不同的模态（如RGB图像、RGBD和点云）。每种模态在3D资产建模中展现出独特的优势：RGB图像包含生动的3D纹理，而点云则定义了细粒度的3D几何形状。然而，大多数现有的原生3D生成架构主要在单模态范式下运行，因此忽略了多模态数据的互补优势，或者仅局限于3D结构，从而限制了可用训练数据集的范围。为了全面利用多模态进行3D建模，我们提出了TriMM，这是首个从基本多模态（如RGB、RGBD和点云）学习的前馈式原生3D生成模型。具体而言：1）TriMM首先引入协作式多模态编码，在保留各模态独特表征优势的同时整合模态特定特征；2）此外，引入辅助2D和3D监督以提高多模态编码的鲁棒性和性能；3）基于嵌入的多模态编码，TriMM采用三平面潜在扩散模型生成高质量的3D资产，同时增强纹理和几何细节的表现。在多个知名数据集上进行的广泛实验表明，TriMM通过有效利用多模态，尽管只使用了少量训练数据，却能达到与在大规模数据集上训练的模型相当的竞争性能。此外，我们在最新的RGB-D数据集上进行了额外实验，验证了将其他多模态数据集纳入3D生成的可行性。

Concept erasure aims to selectively unlearning undesirable content in diffusion models (DMs) to reduce the risk of sensitive content generation. As a novel paradigm in concept erasure, most existing methods employ adversarial training to identify and suppress target concepts, thus reducing the likelihood of sensitive outputs. However, these methods often neglect the specificity of adversarial training in DMs, resulting in only partial mitigation. In this work, we investigate and quantify this specificity from the perspective of concept space, i.e., can adversarial samples truly fit the target concept space? We observe that existing methods neglect the role of conceptual semantics when generating adversarial samples, resulting in ineffective fitting of concept spaces. This oversight leads to the following issues: 1) when there are few adversarial samples, they fail to comprehensively cover the object concept; 2) conversely, they will disrupt other target concept spaces. Motivated by the analysis of these findings, we introduce S-GRACE (Semantics-Guided Robust Adversarial Concept Erasure), which grace leveraging semantic guidance within the concept space to generate adversarial samples and perform erasure training. Experiments conducted with seven state-of-the-art methods and three adversarial prompt generation strategies across various DM unlearning scenarios demonstrate that S-GRACE significantly improves erasure performance 26%, better preserves non-target concepts, and reduces training time by 90%. Our code is available at https://github.com/Qhong-522/S-GRACE.

概念擦除旨在对扩散模型（Diffusion Models, DMs）进行选择性遗忘，以降低敏感内容生成的风险。作为概念擦除的一种新范式，现有大多数方法采用对抗训练来识别和抑制目标概念，从而减少敏感输出的可能性。然而，这些方法往往忽视了对抗训练在扩散模型中的特殊性，导致仅能部分缓解问题。在本工作中，我们从概念空间的角度研究和量化这种特殊性，即对抗样本是否真正符合目标概念空间？我们观察到，现有方法在生成对抗样本时忽略了概念语义的作用，导致无法有效拟合概念空间。这种忽视导致以下问题：1）当对抗样本较少时，它们无法全面覆盖目标概念；2）相反地，它们会干扰其他目标概念空间。基于这些发现的分析，我们提出了S-GRACE（语义引导的鲁棒对抗概念擦除），该方法利用概念空间内的语义引导来生成对抗样本并进行擦除训练。在多种扩散模型遗忘场景下，使用七种最先进方法和三种对抗提示生成策略进行的实验表明，S-GRACE显著提升了26%的擦除性能，更好地保留了非目标概念，并减少了90%的训练时间。我们的代码已发布于https://github.com/Qhong-522/S-GRACE。

Unified Multimodal Modeling aims to integrate visual understanding and generation within a single system. However, existing approaches typically rely on two disparate visual tokenizers, which splits the representation space and hinders truly unified modeling. We propose UniAR, a unified autoregressive framework where a single discrete visual tokenizer serves as the key bridge between understanding and generation, enabling a shared context in which the model can directly interpret its own generated visual tokens without additional re-encoding. UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme, preserving both high-level semantics and low-level details while scaling the effective visual vocabulary at minimal cost. Building on this, the unified autoregressive model adopts parallel-bitwise-prediction to jointly predict spatially grouped, multi-level visual codes, substantially reducing visual sequence length and accelerating generation. Finally, a diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images. Through large-scale pre-training, followed by supervised fine-tuning and reinforcement learning, UniAR achieves state-of-the-art performance on image generation and image editing while remaining competitive on multimodal understanding benchmarks. The project page is available at https://sharelab-sii.github.io/uniar-web.

统一多模态建模旨在将视觉理解和生成集成到单一系统中。然而，现有方法通常依赖两个不同的视觉分词器，这导致表示空间分裂，阻碍了真正的统一建模。我们提出UniAR，这是一个统一的自回归框架，其中单一离散视觉分词器作为理解和生成之间的关键桥梁，使模型能够在共享上下文中直接解释自身生成的视觉 tokens，无需额外的重新编码。UniAR 采用预训练视觉编码器配合多级特征融合和无查找位量化方案，在以极低成本扩展有效视觉词表的同时，保留高层语义和低层细节。在此基础上，统一自回归模型采用并行位预测来联合预测空间分组的多级视觉编码，大幅缩短视觉序列长度并加速生成。最后，基于扩散的视觉解码器对离散视觉 tokens 进行解码，生成高保真图像。通过大规模预训练，随后进行监督微调和强化学习，UniAR 在图像生成和图像编辑任务上达到了最先进水平，同时在多模态理解基准测试中保持竞争力。项目页面见 https://sharelab-sii.github.io/uniar-web。

Generative joint source-channel coding (GJSCC) has emerged as a new Deep JSCC paradigm for achieving high-fidelity and robust image transmission under extreme wireless channel conditions, such as ultra-low bandwidth and low signal-to-noise ratio. Recent studies commonly adopt diffusion models as generative decoders, but they frequently produce visually realistic results with limited semantic consistency. This limitation stems from a fundamental mismatch between reconstruction-oriented JSCC encoders and generative decoders, as the former lack explicit semantic discriminability and fail to provide reliable conditional cues. In this paper, we propose DiT-JSCC, a novel GJSCC backbone that can jointly learn a semantics-prioritized representation encoder and a diffusion transformer (DiT) based generative decoder, our open-source project aims to promote the future research in GJSCC. Specifically, we design a semantics-detail dual-branch encoder that aligns naturally with a coarse-to-fine conditional DiT decoder, prioritizing semantic consistency under extreme channel conditions. Moreover, a training-free adaptive bandwidth allocation strategy inspired by Kolmogorov complexity is introduced to further improve the transmission efficiency, thereby indeed redefining the notion of information value in the era of generative decoding. Extensive experiments demonstrate that DiT-JSCC consistently outperforms existing JSCC methods in both semantic consistency and visual quality, particularly in extreme regimes.

生成式联合信源信道编码（GJSCC）已成为一种新型深度联合信源信道编码范式，能够在极端无线信道条件下（如超低带宽和低信噪比）实现高保真、鲁棒的图像传输。近期研究普遍采用扩散模型作为生成式解码器，但产生的结果往往视觉逼真而语义一致性有限。这一局限性源于重建导向的联合信源信道编码编码器与生成式解码器之间的根本不匹配：前者缺乏明确的语义判别能力，无法提供可靠的条件线索。本文提出DiT-JSCC，一种新型GJSCC主干网络，能够联合学习语义先验表示编码器和基于扩散Transformer（DiT）的生成式解码器，我们的开源项目旨在推动GJSCC的未来研究。具体而言，我们设计了一个语义-细节双分支编码器，与粗到细的条件扩散Transformer解码器自然对齐，在极端信道条件下优先保障语义一致性。此外，受柯尔莫戈罗夫复杂度启发的无训练自适应带宽分配策略被引入以进一步提升传输效率，从而真正重新定义了生成式解码时代的信息价值概念。大量实验表明，DiT-JSCC在语义一致性和视觉质量方面均始终优于现有联合信源信道编码方法，尤其是在极端信道条件下。

Latent Diffusion Models (LDMs) have become dominant in visual synthesis, but their quality-compute trade-off is largely constrained by the tokenizer's fixed compression ratio. Variable-length tokenizers (VLTs) promise adaptive compression by varying token counts, allowing diffusion models to flexibly balance quality and compute. However, conventional VLTs modulate length by truncating ordered token sequences, which makes token semantics depend on token position and breaks representational alignment across lengths. This leads to a cross-length shift in the latent distribution that hinders a single variable-length diffusion model from operating effectively. To address this, we propose a novel variable-length tokenizer that modulates length by merging tokens. We show that encouraging similar tokens to merge enables direct cross-length representation alignment when the diffusion transformer operates according to the merging pattern. Since conventional merging methods are data-dependent, making the merging pattern inaccessible during generation, we introduce learnable global merging, which is data-independent, to ensure compatibility with diffusion transformers. On ImageNet 256$\times$256 generation, our merging-based variable-length tokenizer integrated with a diffusion transformer achieves a superior gFID-compute trade-off compared to prior VLT methods. Code is available at this https URL

潜在扩散模型（LDMs）在视觉合成领域已成为主流，但其质量-计算权衡在很大程度上受限于分词器的固定压缩比。变长分词器（VLTs）通过调整token数量实现自适应压缩，使扩散模型能够灵活平衡质量与计算成本。然而，传统VLTs通过截断有序token序列来调节长度，导致token语义依赖于token位置，破坏了不同长度间的表示对齐。这造成潜在分布的跨长度偏移，阻碍了单一变长扩散模型的有效运作。为解决这一问题，我们提出了一种通过合并token来调节长度的新型变长分词器。研究表明，鼓励相似token合并能够实现直接的跨长度表示对齐，前提是扩散Transformer按照合并模式运作。由于传统合并方法依赖于数据，导致生成过程中无法获取合并模式，我们引入了数据无关的可学习全局合并策略，以确保与扩散Transformer的兼容性。在ImageNet 256×256生成任务中，我们基于合并的变长分词器与扩散Transformer集成后，在gFID-计算权衡方面优于现有VLT方法。代码可访问此https链接。

Enabling efficient deep neural network (DNN) inference on edge devices with different hardware constraints is a challenging task that typically requires DNN architectures to be specialized for each device separately. To avoid the huge manual effort, one can use neural architecture search (NAS). However, many existing NAS methods are resource-intensive and time-consuming because they require the training of many different DNNs from scratch. Furthermore, they do not take the resource constraints of the target system into account. To address these shortcomings, we propose PrototypeNAS, a zero-shot NAS method to accelerate and automate the selection, compression, and specialization of DNNs to different target microcontroller units (MCUs). We propose a novel three-step search method that decouples DNN design and specialization from DNN training for a given target platform. First, we present a novel search space that not only cuts out smaller DNNs from a single large architecture, but instead combines the structural optimization of multiple architecture types, as well as optimization of their pruning and quantization configurations. Second, we explore the use of an ensemble of zero-shot proxies during optimization instead of a single one. Third, we propose the use of Hypervolume subset selection to distill DNN architectures from the Pareto front of the multi-objective optimization that represent the most meaningful tradeoffs between accuracy and FLOPs. We evaluate the effectiveness of PrototypeNAS on 12 different datasets in three different tasks: image classification, time series classification, and object detection. Our results demonstrate that PrototypeNAS is able to identify DNN models within minutes that are small enough to be deployed on off-the-shelf MCUs and still achieve accuracies comparable to the performance of large DNN models.

在具有不同硬件约束的边缘设备上实现高效的深度神经网络（DNN）推理是一项具有挑战性的任务，通常需要为每种设备分别定制DNN架构。为避免巨大的手动工作量，可以使用神经架构搜索（NAS）。然而，许多现有的NAS方法需要从头训练许多不同的DNNs，资源消耗大且耗时。此外，这些方法并未考虑目标系统的资源约束。为解决这些不足，我们提出了PrototypeNAS，这是一种零样本NAS方法，旨在加速和自动化针对不同目标微控制器单元（MCUs）的DNN选择、压缩和定制化过程。我们提出了一种新颖的三步搜索方法，将给定目标平台的DNN设计与定制化从DNN训练中解耦。首先，我们提出了一种新颖的搜索空间，该搜索空间不仅从单一大型架构中裁剪出更小的DNN，还综合优化多种架构类型的结构以及剪枝和量化配置。其次，我们在优化过程中探索使用零样本代理集合而非单一代理。第三，我们提出使用超体积子集选择从多目标优化的帕累托前沿中提炼DNN架构，以呈现精度与FLOPs之间最具意义的权衡。我们在三个不同任务的12个数据集上评估了PrototypeNAS的有效性：图像分类、时间序列分类和目标检测。我们的结果表明，PrototypeNAS能够在数分钟内识别出足够小以部署在现成MCUs上的DNN模型，同时实现与大型DNN模型相当的精度。

Latent diffusion models (LDMs) enable efficient image-to-image translation but discard fine spatial details during compression, degrading downstream perception tasks. We identify two bottlenecks: the autoencoder, which loses spatial information, and the conditioning pathway, which further degrades the source signal through naive downsampling. We propose two lightweight, backbone-agnostic fixes: a Source-Conditioned Autoencoder (SCAE) that injects high-resolution source features into the decoder via skip connections, and a Learnable Guidance Encoder (LGE) that replaces naive downsampling with a learned conditioning signal. Evaluated on RGB-to-SWIR translation for driving scenes with two denoiser backbones (U-Net and DiT), our approach improves detection mAP by up to 2x over the latent diffusion baseline, with up to 3.4x gains on small objects (COCO-small, <32^2 px^2), while achieving state-of-the-art FID. We further show that FID and detection performance are poorly correlated, motivating multi-axis evaluation. Results generalise zero-shot to the public RASMD benchmark. We will publicly release test data with annotations, all checkpoints, and training code.

潜在扩散模型（LDMs）虽然能够实现高效的图像到图像转换，但在压缩过程中会丢弃细粒度的空间细节，从而降低下游感知任务的效果。我们识别出两个瓶颈：一个是自编码器在压缩过程中造成空间信息丢失，另一个是条件通路通过简单下采样进一步削弱源信号。我们提出了两种轻量级且与骨干网络无关的解决方案：源条件自编码器（SCAE），通过跳跃连接将高分辨率源特征注入解码器；以及可学习引导编码器（LGE），用学习到的条件信号替代简单的下采样。我们在驾驶场景的RGB到SWIR转换任务上使用两种去噪器骨干网络（U-Net和DiT）进行评估，结果表明我们的方法相比潜在扩散基线将检测mAP提升至2倍，在小目标（COCO-small，<32^2像素^2）上提升至3.4倍，同时达到了最先进的FID分数。我们进一步表明FID与检测性能之间相关性较弱，这推动了多维度评估的需求。该方法在公开的RASMD基准上实现了良好的零样本泛化。我们将公开发布测试数据及标注、所有模型检查点和训练代码。

Conditional diffusion and flow models routinely fail to satisfy the very constraints that define their task. For instance, a depth-conditioned model often produces images whose re-extracted depth disagrees with the input, even though the forward operator--the depth predictor defining the constraint--is available during both training and inference. Existing approaches generally fall into two categories: supervised models that treat the conditioning signal as a static cue and ignore alignment information at inference, and guidance-based methods that consult it through hand-tuned linear updates, typically trading fidelity to the condition against the plausibility of the generated sample. We argue that the fundamental gap in both paradigms is that the model is never trained to utilize its own alignment error. We introduce FlowBender, a closed-loop framework that treats this error as a first-class input, training the network to learn a correction policy conditioned on inference-time feedback. At each step, an unguided look-ahead pass estimates the clean signal, a task-specific deviation is computed via the forward operator, and a refinement pass consumes this signal to produce a corrected velocity. We propose several variants of FlowBender, including a gradient-based formulation for differentiable operators and a zero-order variant for non-differentiable settings such as JPEG compression. For efficient sampling, we introduce a prior-step shortcut that enables closed-loop correction at a minimal additional computational cost. Across image-to-image translation, restoration, and 3D mesh texturing, FlowBender consistently outperforms standard supervised baselines, alignment-loss-augmented training, and state-of-the-art inference-time guidance, improving fidelity and plausibility simultaneously rather than trading them against each other. Project page: https://flow-bender.github.io/

条件扩散模型和流模型经常无法满足定义其任务的约束条件。例如，即使前向算子（即定义约束的深度预测器）在训练和推理阶段都可用，深度条件模型也经常产生重新提取的深度与输入不一致的图像。现有方法通常分为两类：监督模型将条件信号视为静态线索，在推理时忽略对齐信息；基于引导的方法通过手动调整的线性更新来利用约束，通常在满足条件保真度与生成样本合理性之间进行权衡。我们认为，这两种范式的根本问题在于模型从未被训练利用自身的对齐误差。我们提出FlowBender，一种闭环框架，将该误差视为一等输入，训练网络学习以推理时反馈为条件的修正策略。在每一步中，无引导的前瞻传递估计干净信号，通过前向算子计算任务特定偏差，精化传递利用该信号产生修正后的速度。我们提出了FlowBender的多种变体，包括面向可微分算子的基于梯度形式，以及面向JPEG压缩等不可微分设置的零阶变体。为实现高效采样，我们引入前一步捷径，使闭环修正以极低的额外计算成本实现。在图像到图像转换、修复和3D网格纹理化任务上，FlowBender始终优于标准监督基线、对齐损失增强训练以及最先进的推理时引导方法，同时改善保真度和合理性，而非在两者之间进行权衡。

Training-free detectors of AI-generated images promise generator-agnostic deployment without classifier training, yet their reported numbers are rarely compared under a single controlled protocol. We audit two representative training-free scores -- an autoencoder-reconstruction score (AEROBLADE-style) and a noise-perturbation feature-similarity score (RIGID-style) -- plus a naive feature-kNN control, on a common 1,500-image GenImage-derived benchmark spanning seven generators and JPEG compression at quality 70 and 50. The audit yields three cautionary findings. (i) Implementation details masquerade as method differences: replacing the LPIPS backbone (AlexNet -> VGG-16) changes overall AUROC by +0.085, and switching between resize-to-512 and native-resolution preprocessing flips per-generator conclusions by up to 0.38 AUROC. (ii) Score direction is not a property of the method but of its hyperparameters: the RIGID-style score is inverted (AUROC < 0.5) on SD1.5 and Wukong at noise level sigma=0.05, recovers to >0.5 for every generator at sigma=0.01, and collapses to 0.15 at sigma=0.3. (iii) Dataset format bias inflates robustness claims: without unified re-encoding, AUROC under JPEG-50 exceeds the clean condition for the AlexNet-backbone reconstruction score; after bias correction the residual anomaly localizes to a single generator (BigGAN). The audited scores have complementary per-generator failure sets, but naive z-score fusion does not beat the best single score, indicating that exploiting complementarity requires direction-aware combination.

免训练的AI生成图像检测器承诺实现生成器无关的部署，无需训练分类器，然而其报告的数值很少在单一受控协议下进行比较。本文对两种代表性的免训练分数进行了审计——自编码器重建分数（AEROBLADE风格）和噪声扰动特征相似度分数（RIGID风格）——以及一种朴素特征kNN对照方法，在涵盖七个生成器且经过JPEG质量70和50压缩的通用1500图像GenImage衍生基准上进行测试。审计得出三个警示性发现。(i) 实现细节伪装成方法差异：更换LPIPS主干网络（AlexNet → VGG-16）使整体AUROC改变+0.085，而在resize至512与原生分辨率预处理之间切换会使每个生成器的结论翻转高达0.38 AUROC。(ii) 分数方向不是方法的属性，而是超参数的属性：RIGID风格分数在SD1.5和Wukong上于噪声水平σ=0.05时发生翻转（AUROC<0.5），在σ=0.01时恢复至所有生成器均>0.5，而在σ=0.3时崩溃至0.15。(iii) 数据集格式偏差膨胀了鲁棒性声称：未进行统一重编码时，JPEG-50条件下的AUROC超过干净条件下的AlexNet主干重建分数；经偏差校正后，残余异常仅定位于单一生成器（BigGAN）。被审计的分数在每个生成器上具有互补的失败集，但朴素z-score融合未能击败最佳单一分数，表明利用互补性需要方向感知组合。

Deploying vision transformers (ViTs) on sensor-edge systems is limited not only by on-device compute, but also by the energy and bandwidth required to transmit high-dimensional image data from the sensor to the processor. While in-sensor and near-sensor computing reduce this cost through early feature extraction, existing methods often provide only modest compression. We observe that the frequency domain provides a naturally compact representation of visual information and can be exploited at the sensor level to reduce sensor-to-processor data movement. Building on this insight, we present FrequencyFormer, a co-designed sensor-to-processor pipeline for efficient ViT inference. FrequencyFormer includes: (1) a multi-scale DCT tokenizer that compresses a 224x224 image into compact frequency-domain tokens, achieving up to 128x reduction in off-chip data volume with modest accuracy loss; (2) a LUT-based near-sensor hardware implementation that leverages fixed DCT coefficients for multiplier-free, energy- and area-efficient tokenization; and (3) a modified MIPI-based low-power communication architecture that further reduces transfer energy. FrequencyFormer serves as a drop-in replacement for standard ViT patch embedding and remains compatible with pretrained backbones across classification, detection, and segmentation tasks. The pipeline achieves 28.8 TOPS/W, reduces communication energy by 230x, and lowers total sensor-side energy by 2.22x, demonstrating frequency-domain tokenization as a scalable foundation for in-sensor ViT deployment.

在传感器边缘系统上部署视觉变换器（ViT）不仅受到设备端算力的限制，还受到从传感器向处理器传输高维图像数据所需的能量和带宽的限制。尽管传感器内计算和近传感器计算通过早期特征提取降低了这一成本，但现有方法通常只能提供有限的压缩效果。我们观察到频域能够提供视觉信息的天然紧凑表示，可以在传感器层面加以利用以减少传感器到处理器的数据移动。基于这一洞察，我们提出了FrequencyFormer，一个面向高效ViT推理的协同设计传感器-处理器流水线。FrequencyFormer包括：（1）一个多尺度DCT分词器，将224×224图像压缩为紧凑的频域标记，实现高达128倍的片外数据量缩减，同时保持可接受的精度损失；（2）一个基于查找表（LUT）的近传感器硬件实现，利用固定DCT系数实现无乘法器、能量和面积高效的分词；以及（3）一个改进的基于MIPI的低功耗通信架构，进一步降低传输能量。FrequencyFormer可作为标准ViT块嵌入的即插即用替代方案，并与预训练骨干网络保持兼容，适用于分类、检测和分割等任务。该流水线实现28.8 TOPS/W的能效比，将通信能量降低230倍，并将传感器端总能量降低2.22倍，证明了频域分词可作为传感器内ViT部署的可扩展基础。