arXiv:2609.04820v3 Announce Type: replace-cross

Abstract: Sandbox execution and memory forensics are among the most resources intensive stages of malware triage. Static analysis can scale to millions of files while dynamic and memory analysis require minutes per sample. Because multimodal ransomware detectors often apply every modality to every sample, costs and latency rise even when static evidence suffice.We proposea cost aware Hierarchical Multi Agent System (HMAS) that frames evidence acquisition as a budgeted sequential decision problem enabling timely ransomware detection. Specialist agents generate schema validated risk signals for each modality while domain controllers aggregate evidence and coordinate analysis. The orchestrator starts with static evidence and escalates only if confidence is low or signals disagree. A locally hosted language model provides a limited, accoutable review without replacing the deterministic pipeline. Experiments on 16 ransomware families show 94.12% binary accuracy (F1 0.9376, ROC-AUC 0.9989). For multi class family attribution, it achieves 98.90% accuracy (macro-F1 0.9748). The system resolves 57.95% of the samples using static analysis alone whhile only 6.21% require the full pipeline. Compared with exhaustive cost of 12.0 units,the proposed system reduces te average cost to 6.724 units, representing approximately 44% saving. Zero day experiments further examine generalization to unseen ransomware families. Additional evaluations examine cost sensitivity and component level contribution. LLM-assisted review provides only marginal improvement in F1 while introducing additional computational overhead. Overall, the results show the potential of adaptive hierarchical multi agent orchestration to balance ransomware detection performance, family attribution analysis cost and decision latency.