XAIminer: Understanding AI Decisions Interactively
Why interactive rather than prescribed?
How useful an XAI method is depends on the task, the AI model, its performance and the image. A method that carries in one context can be uninformative or even misleading in another. This is why the XRAISE project prescribes no single method: assessors should be able to choose, switch and place results side by side as the situation demands. Whether that works is examined in an experimental user study and a focus group with safety assessors.
In the image above both models detect the person reliably (100 % and 91 %). The Grad-CAM maps show, however, that only ConvNeXt-T (centre) looks at the person — VGG16 (right) relies on the lower right edge of the image. Right answer, questionable reasoning.
What the system does
XAIminer — developed in the XRAISE research project, see the front of this column — runs in the browser and is modular: any number of panels side by side, each with its own AI model, training state, XAI method (Grad-CAM, LRP, CRAFT, CRP) and filters. Panels can be linked so that all of them show the same image. For every image the classification probabilities and the true class are shown; anything noticeable can be annotated.
In the image above, the same network type in two training states: one reports “person” (87 %) although there is none — the LRP map (centre) shows that it responds to the building at the edge of the image (Clever Hans effect). The clean training state (right) decides correctly.
Duration: 2024-2026
Funding: DZSF beim Eisenbahn-Bundesamt
Contact:Sascha Weber, Carsten Knoll