Evaluation of the impact of image mutations on the origin classification of digital images

Context: Image origin classifier tools are increasingly deployed to verify the origin and authenticity of digital images, especially for detecting AI-generated or manipulated content. Despite their usage in forensic and media applications, the robustness of these systems under real-world image perturbations remains largely unexplored. Objective: This study aims to evaluate the robustness of a proprietary tool (developer claimed consistency >90%) against a wide spectrum of systematic image mutations to uncover biases, failure points, and inconsistencies in performance. Methods: We designed a large-scale mutation-driven automated testing approach using ImageMagick, an open-source software suite for image processing, to apply 128 distinct mutation types — categorized into 101 discrete and 27 continuous filters — across a dataset of 40 original images, yielding over 31,000 mutated samples. The classifier’s output was collected via an automated Python pipeline and analyzed after parsing structured data from the tool’s JSON responses. Results: The classifier showed significant sensitivity to specific mutation types, decreasing consistency to 61% under discrete mutations and dropping further to 30%–41% under continuous parameterized mutations. Mutations involving grayscale conversions and rotations were particularly detrimental. Additionally, performance degraded with lower resolutions, with a critical threshold observed to be ∼280 pixels in height or width. Conclusion: Our empirical findings underscore black-box-based image classification systems’ vulnerability to structured image-level mutations, without requiring access to their internal logic. Building on this observation, the proposed mutation-based methodology provides a generic, automated, scalable and reusable framework for systematically generating diverse visual transformations for different image-based testing strategies