Heterogeneous Graph Attention Network With Multimodal Feature Selection for Cancer Multiomics Integration
arXiv cs.LGarXiv:2408.02845v2 Announce Type: replace
Abstract: The increase in high-dimensional multiomics data demands advanced integration models to capture the complexity of human diseases. Graph-based deep learning integration models, despite their promise, struggle with small patient cohorts and high-dimensional features, often applying independent feature selection without modeling relationships among omics. Furthermore, conventional graph-based omics models focus on homogeneous graphs, lacking multiple types of nodes and edges to capture diverse structures. We introduce a Heterogeneous Graph ATtention network for omics integration (HeteroGATomics) to improve cancer diagnosis. HeteroGATomics performs multimodal feature selection through a multi-agent system, creating dedicated networks of feature and patient similarity for each omics modality. These networks are then combined into one heterogeneous graph for learning holistic omics-specific representations and integrating predictions across modalities. Experiments on three cancer multiomics datasets show that HeteroGATomics outperforms baseline methods across most evaluation metrics in cancer diagnosis. External validation on an independent breast cancer cohort further demonstrates its generalizability, where HeteroGATomics outperforms graph-based multiomics integration methods across evaluation metrics. Moreover, HeteroGATomics enhances interpretability by identifying important biomarkers contributing to the diagnosis outcomes.
KathDB-FAO: Optimizing Multimodal Queries through Operator Grouping and Code Reuse
arXiv cs.AIarXiv:2609.28761v2 Announce Type: replace-cross
Abstract: Advances in multimodal large language models (multimodal LLMs) allow data systems to answer analytical queries over text and images alongside relational tables, but at a significant cost for large-scale data. Existing systems reduce this cost through rewrite rules written in advance, which cannot use the logic of an individual query or the semantics of the data it processes. To address this challenge, we design, implement, and evaluate KathDB-FAO, a query evaluation system over multimodal data. KathDB-FAO decomposes a complex query into an execution plan of fine-grained steps, which we call Function-as-Operators (FAOs). To reduce execution cost, it searches for subsets of these FAOs to implement together as one function, enabling query-specific optimizations that make fewer model calls. Synthesizing these functions with a frontier model has its own cost, yet we observe that synthesized FAOs often recur across queries. To further reduce plan generation cost, KathDB-FAO uses pre-built functions and reuses functions synthesized for earlier queries. On SemBench and an additional workload, KathDB-FAO achieves quality comparable to the strongest competing system while reducing end-to-end cost per query by 76% on average.
Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation
arXiv cs.AIarXiv:2609.15334v2 Announce Type: replace-cross
Abstract: Medical imaging modalities such as ultrasound and X-ray are widely used in clinical practice, where diagnosis follows a structured, evidence-driven workflow aligned with standardized criteria. While multimodal large language models (MLLMs) show promise for automated medical report generation, most existing systems rely on end-to-end multimodal fusion without modeling clinically defined intermediate attributes, leading to limited grounding and interpretability. To address this issue, we propose CORAL (COncept-grounded ReAsoning with Localization), a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process. CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module. The resulting textual concept tokens are combined with mask-modulated visual features within an MLLM to enable structured report generation and diagnostic prediction. Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs, indicating that concept-grounded reasoning better aligns generation with clinical decision processes.
2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation
arXiv cs.AIarXiv:2609.11308v2 Announce Type: replace-cross
Abstract: Long-horizon robot manipulation requires memory, but not necessarily inside the action policy. To address such tasks, current agentic systems often combine VLAs with planners and geometric tools, sometimes using additional depth or calibrated geometry. These systems confound attribution: gains may come from richer observations or alternative motor tools, while failures may stem from either the policy or an under-specified language interface. We isolate this question through a deliberately constrained design: less tool breadth, but greater interface bandwidth. 2AM makes a multimodal Agent the sole holder of task memory and a single RGB-based, episodically stateless Action Model the sole executor of task-relevant motion. The Agent compiles interaction history into subtask language and optional 2D grasp, place, and move hints that bind its physical intention at different time scales. To teach this steerability to the VLA, we augment demonstrations with structured hint labels and train under condition dropout, spatial noise, and temporal jitter to tolerate imperfect Agent outputs. On LIBERO-Mem, without depth, online geometry, or planner-based object motion, 2AM reaches 76.3% average completion, a 61.5-point improvement over the strongest reported baseline of 14.8%, together with 63.0% relaxed and 11.8% strict success. These results show that task memory can remain Agent-side. They further show that Action Model capability depends not only on what the policy has learned, but on how precisely the Agent can steer it. Project website: agent-mem-action-model.github.io
Cost-Aware Hierarchical Multi-Agent Ransomware Detection and Family Attribution
arXiv cs.AIarXiv:2609.04820v3 Announce Type: replace-cross
Abstract: Sandbox execution and memory forensics are among the most resources intensive stages of malware triage. Static analysis can scale to millions of files while dynamic and memory analysis require minutes per sample. Because multimodal ransomware detectors often apply every modality to every sample, costs and latency rise even when static evidence suffice.We proposea cost aware Hierarchical Multi Agent System (HMAS) that frames evidence acquisition as a budgeted sequential decision problem enabling timely ransomware detection. Specialist agents generate schema validated risk signals for each modality while domain controllers aggregate evidence and coordinate analysis. The orchestrator starts with static evidence and escalates only if confidence is low or signals disagree. A locally hosted language model provides a limited, accoutable review without replacing the deterministic pipeline. Experiments on 16 ransomware families show 94.12% binary accuracy (F1 0.9376, ROC-AUC 0.9989). For multi class family attribution, it achieves 98.90% accuracy (macro-F1 0.9748). The system resolves 57.95% of the samples using static analysis alone whhile only 6.21% require the full pipeline. Compared with exhaustive cost of 12.0 units,the proposed system reduces te average cost to 6.724 units, representing approximately 44% saving. Zero day experiments further examine generalization to unseen ransomware families. Additional evaluations examine cost sensitivity and component level contribution. LLM-assisted review provides only marginal improvement in F1 while introducing additional computational overhead. Overall, the results show the potential of adaptive hierarchical multi agent orchestration to balance ransomware detection performance, family attribution analysis cost and decision latency.
Rethinking AI in Medical Imaging: Assumptions, Reality, and Reframing; Towards Physician-Centered Agentic Reasoning
arXiv cs.AIarXiv:2607.27428v2 Announce Type: replace-cross
Abstract: Medical imaging has served as primary proving ground for clinical AI, yet a decade of intense research has not translated into proportionate bedside impact. We argue that this gap is not primarily a product of insufficient algorithmic performance, inadequate regulation, or limited explainability. Rather, it reflects a structural misalignment, between how AI systems are designed and evaluated, and how clinical decisions are made. This work identifies 8 interconnected dimensions of this misalignment: 1) the dominance of pixel-only models in a multimodal clinical world; 2) the erosion of physician trust through opaque and inflexible systems; 3) the unfulfilled promise of foundation models in data-sparse medical domains; 4) the persistent bottleneck of non-shareable under-curated datasets; 5) the gap between validated algorithms and deployable clinical platforms; 6) the slow, unmonitored translation of evidence into routine practice; 7) the risk of AI displacing rather than extending physician judgment; and 8) the failure of prediction-centric AI to generate actionable, agentic clinical guidance. For each dimension, we reframe the problem and propose a path forward, culminating in a vision of physician-centered AI solutions with agentic reasoning that extend, rather than replace, clinical judgment.
ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
arXiv cs.AIarXiv:2607.24743v3 Announce Type: replace-cross
Abstract: Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.
SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
arXiv cs.AIarXiv:2606.25990v2 Announce Type: replace-cross
Abstract: As multimodal conversational systems increasingly engage in spoken interaction, their ability to navigate paralinguistic social cues has become a critical bottleneck for natural human-AI communication. However, existing evaluations of machine emotional intelligence assess reasoning exclusively through isolated text or passive acoustic perception, overlooking the complex cross-modal reasoning required for active, multi-turn dialogue. We introduce \textsc{SpeechEQ}, a comprehensive framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs). The framework includes a validated dataset of 2,265 dialogues across 15 Emotional Quotient (EQ) subscales grounded in EQ-i 2.0 theory, along with a multi-turn evaluation protocol measured by our proposed Spoken EQ (SEQ) score inspired by human EQ assessments. Experiments show limitations in how both existing Speech Emotion Recognition and end-to-end Speech-Language Models understand and apply paralinguistic cues through speech. While end-to-end architectures outperform cascaded systems, \textsc{SpeechEQ} reveals that current multimodal models remain bottlenecked by a text-reliant ``modality shortcut,'' an alignment-induced ``safety trap,'' and ``contextual amnesia,'' highlighting the barriers to truly emotionally aware AI. Our benchmark can be accessed at https://huggingface.co/datasets/SpeechEQ/SpeechEQ and demo page at https://speecheq.github.io
ReLATE: Learning Relative Representations for Post-Hoc Multimodal Alignment
arXiv cs.AIarXiv:2605.16834v2 Announce Type: replace-cross
Abstract: Post-hoc multimodal alignment connects independently pretrained vision and language encoders while keeping them frozen, offering an alternative to joint pretraining when paired data are scarce. A central design challenge is to construct comparable representations across heterogeneous feature spaces while retaining local information relevant to distinct semantic concepts within each input. We propose Relative Alignment with Learnable Anchors and Token Evidence (ReLATE), which learns modality-specific anchors that jointly define relative representations and determine token contributions to each relative coordinate. Image patches and text tokens are represented by their similarities to anchors within their respective frozen feature spaces. Anchor-wise Similarity Pooling (ASP) then weights and aggregates these similarities separately for each anchor to construct the corresponding relative coordinate. A sample-level symmetric contrastive objective aligns the resulting image and text representations by optimizing only the anchors, without updating the encoders. Experiments on 13 general-domain benchmarks across four vision-language tasks demonstrate improvements over the compared post-hoc alignment methods under matched paired-data budgets, with further evaluations across data scales and encoder combinations. Beyond the general domain, we evaluate ReLATE in pathology as a specialized-domain application. By aligning frozen pathology image and biomedical language encoders, ReLATE matches or exceeds the evaluated pathology-specific vision-language models in balanced accuracy on three zero-shot classification benchmarks.
DiffCap-Bench: A Comprehensive, Challenging, Robust Benchmark for Image Difference Captioning
arXiv cs.AIarXiv:2605.04503v2 Announce Type: replace-cross
Abstract: Image Difference Captioning (IDC) generates natural language descriptions that precisely identify differences between two images, serving as a key benchmark for fine-grained change perception, cross-modal reasoning, and image editing data construction. However, existing benchmarks lack diversity and compositional complexity, and standard lexical-overlap metrics (e.g., BLEU, METEOR) fail to capture semantic consistency or penalize hallucinations, which together prevent a comprehensive and robust evaluation of multimodal large language models (MLLMs) on IDC. To address these gaps, we introduce DiffCap-Bench, a comprehensive IDC benchmark covering ten distinct difference categories to ensure diversity and compositional complexity. Furthermore, we propose an LLM-as-a-Judge evaluation protocol grounded in human-validated Difference Lists, enabling a robust assessment of models' ability to both capture and describe visual changes. Through extensive evaluation of state-of-the-art MLLMs, we reveal significant performance gaps between proprietary and open-source models, highlight the critical importance of reasoning capability, and identify clear limitations in model scaling. Our framework also demonstrates strong alignment with human expert judgments and strong correlation with downstream image editing data construction quality. These findings establish DiffCap-Bench as both a reliable IDC evaluation framework and a practical predictor of downstream utility. The benchmark and code will be made publicly available to support further research.
CTM-AI: Evolving Multi-Agent Intelligence Inspired by a Model of Consciousness
arXiv cs.AIarXiv:2605.04097v2 Announce Type: replace-cross
Abstract: Despite remarkable advances, today's AI systems remain narrow in scope, falling short of the adaptive and continual intelligence that characterizes human capabilities. This gap has fueled longstanding debates about whether AI might one day achieve human-like generality or even consciousness. Flipping the debate, we present a concrete blueprint for a multi-agent AI system inspired by a formal machine model of consciousness called the Conscious Turing Machine (CTM). Together, CTM-AI contains a large collection of powerful processors ranging from specialized experts (e.g., language models, multimodal models, tools, and APIs) to unspecialized general-purpose learners poised to develop their own expertise. Crucially, for whatever problem must be dealt with, information from many processors is competitively selected, integrated, and exchanged appropriately to solve the task. Feedback mechanisms in CTM-AI enable the system to iteratively evolve its internal connections and communication. CTM-AI achieves state-of-the-art accuracy on MUStARD (72.28) and UR-FUNNY (72.13), outperforming multimodal and multi-agent frameworks. On tool-using and agentic tasks, CTM-AI achieves 10+ points of improvement on StableToolBench and WebArena-Lite. Overall, CTM-AI offers a principled and testable multi-agent AI system inspired by a model of consciousness. The code is available at https://github.com/consciousness-lab/ctm-ai.
Bolek: A Multimodal Language Model for Molecular Reasoning
arXiv cs.AIarXiv:2605.02745v2 Announce Type: replace-cross
Abstract: Molecular property models increasingly support high-stakes drug-discovery decisions, but their outputs are often difficult to audit: classical predictors return scores without rationale, while language models can produce fluent explanations weakly grounded in the input molecule.
We introduce Bolek, a compact multimodal language model that grounds natural-language reasoning in molecular structure by injecting a Morgan fingerprint embedding into an instruction-tuned text decoder. Bolek is fine-tuned on molecular alignment tasks, including molecule description, RDKit descriptor prediction, and substructure detection, and on downstream reasoning over 15 TDC binary classification tasks using synthetic chains-of-thought anchored in concrete molecular features.
Across these tasks, Bolek outperforms its Qwen3-4B-Instruct base on all endpoints in yes/no mode and on 13 of 15 in chain-of-thought mode, raising mean ROC/PR AUC from 0.55 to 0.76. It also outperforms TxGemma-9B-Chat on 13 of 15 binary classification tasks despite being less than half its size. Bolek's explanations are more grounded than those of the baseline LLMs: it cites numerical descriptors 10-100x more often per chain-of-thought, and the cited values agree strongly with RDKit for key descriptors such as TPSA, MolLogP, and MolWt (Spearman rho = 0.87-0.91). Generalisation extends beyond the training panel: on 15 unseen TDC classification endpoints, Bolek matches TxGemma on five, and it produces non-trivial rank correlations on three held-out regression endpoints despite never seeing downstream regression during training.
These results suggest that targeted modality injection and reasoning supervision tied to verifiable molecular features can yield compact, auditable molecular reasoning models.
Hierarchy-Aware Multimodal Unlearning for Medical AI
arXiv cs.AIarXiv:2512.09867v4 Announce Type: replace-cross
Abstract: Multimodal large language models (MLLMs) are increasingly used in sensitive domains such as medical AI, where privacy regulations, including HIPAA and GDPR, require the removal of specific individuals' or institutions' data. This motivates machine unlearning, which aims to remove the influence of target data from a trained model. However, existing unlearning benchmarks fail to reflect the hierarchical and multimodal structure of real-world medical data, limiting their ability to properly evaluate unlearning in practice. Therefore, we introduce MedForget, a hierarchy-aware multimodal unlearning benchmark that models data from medical institutions as a nested structure, enabling fine-grained evaluation across retain and forget splits. Experiments show that current unlearning methods struggle to achieve effective hierarchy-aware forgetting without degrading retained utility, measured by performance on clinically relevant prediction tasks. To address this limitation, we propose Cross-modal Hierarchy-Informed Projection for unlearning (CHIP), a training-free, hierarchy-aware multimodal unlearning method that deletes information by selectively removing target-specific weight subspaces while preserving sibling-shared information. Our results show that CHIP achieves the highest forget-retain performance gap across all hierarchy levels while maintaining competitive downstream utility compared to existing methods. Overall, MedForget provides a practical benchmark for evaluating hierarchy-aware multimodal unlearning for medical data, while CHIP offers an effective solution that generalizes across architectures and balances deletion with utility.
AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agents
arXiv cs.AIarXiv:2510.04257v2 Announce Type: replace-cross
Abstract: Multimodal agents built on large vision-language models are increasingly deployed in open-world settings but remain highly vulnerable to prompt injection. Typographic text-in-image attacks such as FigStep have been evaluated mainly on standalone vision-language models. For agents, user goals, webpage context, and action history make the effect of embedded instructions uncertain, and robustness to behavior-guided, adaptive visual injections remains insufficiently studied. We introduce AgentTypo, a black-box red-teaming framework that launches adaptive typographic prompt injection by embedding optimized text into webpage images. We formulate an attention-guided automatic typographic prompt-injection objective that favors the attacker's expected answer over the clean-page answer, together with task-conditioned visual attention from small question-answering models. A novel black-box optimization method searches text position, contrast, and color. To further strengthen the attack, we develop a multi-LLM system that iteratively refines injection prompts using evaluation feedback and retrieves successful past examples for strategy learning. Experiments on the VWA-Adv and Mind2Web benchmarks show that AgentTypo significantly outperforms state-of-the-art image-based prompt injection attacks. Across seven closed- and open-source model backends, AgentTypo-pro achieves 45%-83% ASR, outperforming the CLIP-based AgentAttack by 29-47% and previous typographic attacks by 25%-37% on VWA-Adv.
Large Language Models for Crash Detection in Video: A Survey of Methods, Datasets, and Challenges
arXiv cs.AIarXiv:2507.02074v3 Announce Type: replace-cross
Abstract: Crash detection from video feeds is a critical problem in intelligent transportation systems. Recent developments in large language models (LLMs) and vision-language models (VLMs) have transformed how we process, reason about, and summarize multimodal information. This paper surveys recent methods leveraging LLMs for crash detection from video data. We present a structured taxonomy of fusion strategies, summarize key datasets, analyze model architectures, compare performance benchmarks, and discuss ongoing challenges and opportunities. Our review provides a foundation for future research in this fast-growing intersection of video understanding and foundation models.
Towards General Continuous Memory for Vision-Language Models
arXiv cs.AIarXiv:2505.17670v3 Announce Type: replace-cross
Abstract: Language models (LMs) and their extension, vision-language models (VLMs), have achieved remarkable performance across various tasks. However, they still struggle with complex reasoning tasks that require multimodal or multilingual real-world knowledge. To support such capabilities, an external memory system that can efficiently provide relevant multimodal information is essential. Existing approaches generally concatenate image and text tokens into a long sequence as memory, which, however, may drastically increase context length and even degrade performance. In contrast, we propose using continuous memory, a compact set of dense embeddings to more effectively and efficiently represent multimodal and multilingual knowledge. Our key insight is that a VLM can serve as its own continuous memory encoder. We empirically show that this design improves performance on complex multimodal reasoning tasks. Building on this, we introduce a data-efficient and parameter-efficient method to fine-tune the VLM into a memory encoder, requiring only 1.2% of the model's parameters and a small corpus of 15.6K self-synthesized samples. Our approach CoMEM utilizes VLM's original capabilities to encode arbitrary multimodal and multilingual knowledge into just 8 continuous embeddings. Since the inference-time VLM remains frozen, our memory module is plug-and-play and can be flexibly integrated as needed. Extensive experiments across eight multimodal reasoning benchmarks demonstrate the effectiveness of our approach.
Can Large Language Models Capture Video Game Engagement?
arXiv cs.AIarXiv:2502.04379v3 Announce Type: replace-cross
Abstract: Can out-of-the-box pretrained Large Language Models (LLMs) detect human affect successfully when observing a video? To address this question, for the first time, we evaluate comprehensively the capacity of popular LLMs for successfully predicting continuous affect annotations of videos when prompted by a sequence of text and video frames in a multimodal fashion. In this paper, we test LLMs' ability to correctly label changes of in-game engagement in 80 minutes of annotated videogame footage from 20 first-person shooter games of the GameVibe corpus. We run over 4,800 experiments to investigate the impact of LLM architecture, model size, input modality, prompting strategy, and ground truth processing method on engagement prediction. Our findings suggest that while LLMs rightfully claim human-like performance across multiple domains and able to outperform traditional machine learning baselines, they generally fall behind continuous experience annotations provided by humans. We examine some of the underlying causes for a fluctuating performance across games, highlight the cases where LLMs exceed expectations, and draw a roadmap for the further exploration of automated emotion labelling via LLMs.
Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning
arXiv cs.AIarXiv:2606.29984v2 Announce Type: replace
Abstract: Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.
When No Answer Is Correct: Diagnosing Absent Answer Detection for MLLMs in Video Understanding
arXiv cs.AIarXiv:2606.08239v2 Announce Type: replace
Abstract: Multimodal large language models (MLLMs) have made substantial advancements in video understanding, yet the reliability of their responses remains underexplored. This work presents a diagnostic study of absent answer detection for MLLMs in video understanding, where the correct answer is deliberately excluded from the candidate set and a reliable model is expected to recognize that no valid option exists. We evaluate the absent answer detection behavior under three settings: multiple-choice questions augmented with a ``None of the Above'' option, open-ended generation with a detection instruction, and standard evaluation without any guidance. Across a diverse set of models and benchmarks, we find that MLLMs overwhelmingly select plausible distractors rather than detecting the absent answer. This failure is more pronounced in temporal reasoning tasks and worsens with denser frame sampling. We further explore chain-of-thought prompting as a mitigation strategy and find that while it substantially improves detection rates, performance remains unsatisfactory, suggesting that prompting-based strategies alone are insufficient to fully address this limitation. These findings expose a systematic failure in absent answer detection and highlight the need for explicit detection mechanisms in multimodal systems.
Anthropogenic Regional Adaptation in Multimodal Vision-Language Model
arXiv cs.AIarXiv:2604.11490v3 Announce Type: replace
Abstract: While the field of vision-language (VL) has achieved remarkable success in integrating visual and textual information across multiple languages and domains, there is still no dedicated framework for assessing human-centric alignment in vision-language systems. We offer two contributions to address this gap. First, we introduce Anthropogenic Regional Adaptation: a novel paradigm that aims to optimize model relevance to specific regional contexts while ensuring the retention of global generalization capabilities. Second, we present a simple, but effective adaptation method named Geographical-generalization-made-easy (GG-EZ), which utilizes regional data filtering and model merging. Through comprehensive experiments on 3 VL architectures: large vision-language models, text-to-image diffusion models, and vision-language embedding models, and a case study in Southeast Asia (SEA) regional adaptation, we demonstrate the importance of Anthropogenic Regional Adaptation and the effectiveness of GG-EZ, showing 5-15% gains in cultural relevance metrics across SEA while maintaining over 98% of global performance and even occasionally surpassing it. Our findings establish Anthropogenic Regional Alignment as a foundational paradigm towards applicability of multimodal vision-language models in diverse regions and demonstrate a simple-yet-effective baseline method that optimizes regional value alignment while preserving global generalization.
继续滚动加载更多…