Source Count: 22 | Weighted Score: 40 | Source Confidence: [4/5] | Primary Tier: 1 | Last Updated: March 11, 2026
Keywords: explainable AI, XAI, interpretability, LIME, SHAP, black box, trust, regulation, transparency, machine learning
Category Tags: information-computation, artificial-intelligence, machine-learning, ethics, regulation
Cross-References: ZD_2_12 — Generative AI · ZD_2_11 — Reinforcement Learning · ZD_1_02 — Mathematics Information

QUICK SUMMARY

Explainable AI (XAI) is the field concerned with making artificial intelligence systems — particularly complex machine learning models — understandable to humans. As AI systems increasingly make or influence high-stakes decisions affecting human lives — medical diagnosis ("you have cancer"), criminal justice ("this defendant is high risk for recidivism"), financial services ("your loan application is denied"), hiring ("your resume has been screened out"), autonomous driving ("turn left") — the inability to understand why a model made a particular decision becomes not merely an academic concern but a legal, ethical, and practical imperative. The core tension is between accuracy and interpretability: the most accurate models for many tasks (deep neural networks with millions or billions of parameters, gradient-boosted tree ensembles) are inherently opaque — "black boxes" whose internal decision processes resist human comprehension — while naturally interpretable models (linear regression, decision trees, rule lists) are often less accurate on complex, high-dimensional data. XAI approaches fall into several categories: (1) Inherently interpretable models — using models that are transparent by design: linear/logistic regression (coefficients indicate feature importance), shallow decision trees (visual decision rules), rule lists (if-then rules), GAMs (Generalized Additive Models — Lou, Caruana, and Gehrke); Rudin (2019) argued in Nature Machine Intelligence that for high-stakes decisions, inherently interpretable models should be preferred over post-hoc explanations of black boxes; (2) Post-hoc explanation methods — explaining black box models after training: LIME (Local Interpretable Model-agnostic Explanations — Ribeiro, Singh, Guestrin, 2016) — generates local explanations by fitting a simple interpretable model (linear model) to the black box's behavior in the neighborhood of a specific prediction; SHAP (SHapley Additive exPlanations — Lundberg and Lee, 2017) — assigns each feature a contribution to a specific prediction using game-theoretic Shapley values, providing consistent and locally accurate feature attributions; Attention visualization — highlighting which parts of the input a model attends to (saliency maps, attention weights); Counterfactual explanations — "your loan would have been approved if your income were $5,000 higher"; (3) Concept-based explanations — explaining model behavior in terms of human-understandable concepts rather than raw input features (TCAV — Testing with Concept Activation Vectors, Kim et al., 2018). Regulatory drivers include the EU's GDPR (Article 22 — right to explanation for automated decisions) and the EU AI Act (2024 — requiring transparency for high-risk AI systems); the growing field of algorithmic auditing examines deployed systems for bias, fairness, and accountability.

1. VERIFIED CLAIMS (Tier 1 — Peer-Reviewed / Established)

1.1 The Interpretability–Accuracy Tradeoff

• Black box models: deep neural networks, ensemble methods (random forests, XGBoost), and large language models achieve state-of-the-art accuracy but are opaque — with millions to trillions of parameters, no human can trace how a specific input leads to a specific output through internal computations
• Inherently interpretable models: linear models (coefficients directly show feature importance), small decision trees (decision path is a human-readable logical rule), rule lists, and GAMs provide transparency but may sacrifice some accuracy on complex tasks; Rudin (2019) argued that the accuracy gap is often smaller than assumed and that post-hoc explanations of black boxes can be misleading — they explain an approximation of the model, not the model itself

1.2 Post-Hoc Explanation Methods

• LIME (Ribeiro et al., 2016): generates explanations for individual predictions — perturbs the input, observes how the black box's output changes, fits a simple interpretable model (linear model, sparse rules) to the local decision boundary; model-agnostic (works with any classifier); limitations: explanations can be unstable (small input changes → different explanations) and may not faithfully represent the underlying model's reasoning
• SHAP (Lundberg and Lee, 2017): based on Shapley values from cooperative game theory — each feature is a "player" and SHAP distributes the prediction "payout" among features based on their marginal contributions; provides theoretically grounded, consistent feature attributions; TreeSHAP — efficient exact computation for tree-based models; KernelSHAP — approximate method for any model; widely adopted in industry for model debugging and stakeholder communication
• Saliency maps and attention: for image models — gradient-based methods (Grad-CAM — Selvaraju et al., 2017) highlight image regions most relevant to a prediction; for text models — attention weights indicate which input tokens influenced the output; however, attention is not explanation — Jain and Wallace (2019) showed attention weights do not reliably indicate feature importance

1.3 Regulatory Context

• EU GDPR, Article 22: establishes the right not to be subject to automated decision-making with legal or similarly significant effects without human involvement or the right to obtain meaningful information about the logic involved; the exact legal requirements for "explanation" are debated by legal scholars
• EU AI Act (2024): classifies AI systems by risk level; high-risk systems (healthcare, criminal justice, employment, credit scoring) must meet transparency and explainability requirements; prohibited uses include real-time biometric surveillance (with exceptions) and social scoring

2. CREDIBLE CLAIMS (Tier 2 — Academic / Debated but Supported)

2.1 Counterfactual and Concept-Based Explanations

• Counterfactual explanations (Wachter et al., 2018): explain decisions by describing the smallest change to input features that would change the outcome — "your loan would have been approved if your debt-to-income ratio were below 0.4"; intuitive, actionable, and privacy-preserving (no need to reveal the model); multiple competing frameworks for generating diverse, feasible, and sparse counterfactuals
• TCAV (Kim et al., 2018): Testing with Concept Activation Vectors — explains model behavior in terms of high-level human concepts (e.g., "does this model rely on the concept 'striped' when classifying zebras?") rather than raw pixel features; requires labeled concept examples; provides more interpretable explanations for domain experts

2.2 Faithfulness vs. Plausibility

• Explanation quality debate: an explanation can be plausible (makes sense to humans) without being faithful (accurately reflecting how the model actually works); published evidence demonstrates that users often prefer explanations that confirm their expectations, regardless of faithfulness; unfaithful explanations can create a false sense of understanding, potentially more dangerous than no explanation at all; measuring faithfulness systematically remains an open challenge

3. SPECULATIVE CLAIMS (Tier 3 — Possible but Unverified)

3.1 Mechanistic Interpretability of LLMs

• Mechanistic interpretability: reverse-engineering the internal computations of neural networks — identifying interpretable "circuits" (compositions of attention heads and MLP neurons that implement specific functions); Elhage et al. (Anthropic, 2022), Olah et al. (2020), and Neel Nanda's work on transformers seek to understand how LLMs implement arithmetic, factual recall, and reasoning internally; whether complete mechanistic understanding of billion-parameter models is achievable remains open

4. DUBIOUS CLAIMS (Tier 4 — No Credible Source / Contradicted by Evidence)

4.1 Any Explanation Is Good Enough

• [MISLEADING] Post-hoc explanations can be actively harmful if they are unfaithful, unstable, or misunderstood — creating false confidence in flawed or biased models; Rudin (2019) warned that explaining black boxes when interpretable alternatives exist of similar accuracy is unnecessary risk; explanations should be evaluated for faithfulness, stability, and comprehensibility — not merely "provided" as regulatory checkboxes

COUNTER-ARGUMENTS

• Interpretability vs. accuracy trade-off challenged: Cynthia Rudin (2019) argued that the perceived trade-off between model interpretability and predictive performance is largely a myth — for structured data, inherently interpretable models often perform comparably to black boxes, and the field should "stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead"
• Post-hoc explanation faithfulness: Methods like LIME (Ribeiro et al., 2016) and SHAP (Lundberg and Lee, 2017) provide local explanations of black-box predictions, but their faithfulness to the model's actual decision process has been questioned. Adebayo et al. (2018) showed that some saliency map methods are insensitive to model parameters and data labels, meaning they may not actually explain model behavior
• Explainability as social construct: Tim Miller (2019) argued that XAI research has largely ignored decades of social science research on how humans actually explain things — explanations are contrastive, selected, and social, not comprehensive feature-attribution lists. This suggests that technically rigorous explanations may not satisfy human cognitive needs

IMAGES

No images assigned yet.

BIBLIOGRAPHY

1. Molnar, Christoph. . . christophm.github.io | 2022 | ∅ | Interpretable Machine Learning: A Guide for Making Black Box Models Explainable | ∅ | ∅ | ∅ | 2nd | doi:10.1177/09726225241252009 | ∅ | ∅ | ∅
2. Rudin, Cynthia | 2019 | "Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead" | Nature Machine Intelligence | ∅ | 1.5::206–215 | ∅ | ∅ | doi:10.1038/s42256-019-0048-x | ∅ | ∅ | ∅
3. Ribeiro, Marco Tulio, Sameer Singh; Carlos Guestrin. : 1135 1144 | 2016 | "'Why Should I Trust You?': Explaining the Predictions of Any Classifier" | KDD | ∅ | ∅ | ∅ | ∅ | doi:10.1145/2939672.2939778 | ∅ | ∅ | ∅
4. Lundberg, Scott M.; Su-In Lee. : 4765 4774 | 2017 | "A Unified Approach to Interpreting Model Predictions" | NeurIPS | ∅ | ∅ | ∅ | ∅ | | ∅ | ∅ | ∅
5. Selvaraju, Ramprasaath R., et al. : 618 626 | 2017 | "Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization" | ICCV | ∅ | ∅ | ∅ | ∅ | doi:10.1109/iccv.2017.74 | ∅ | ∅ | ∅
6. Kim, Been, et al. : 2668 2677 | 2018 | "Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)" | ICML | ∅ | ∅ | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
7. Wachter, Sandra, Brent Mittelstadt; Chris Russell | 2018 | "Counterfactual Explanations without Opening the Black Box" | Harvard Journal of Law & Technology | ∅ | 31.2::841–887 | ∅ | ∅ | doi:10.2139/ssrn.3063289 | ∅ | ∅ | ∅
8. Arrieta, Alejandro Barredo, et al | 2020 | "Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges" | Information Fusion | ∅ | 58::82–115 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
9. Lipton, Zachary C | 2018 | "The Mythos of Model Interpretability" | Queue | ∅ | 16.3::31–57 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
10. Doshi-Velez, Finale; Been Kim | 2017 | "Towards a Rigorous Science of Interpretable Machine Learning" | ∅ | ∅ | ∅ | ∅ | ∅ | arxiv:1702.08608 | ∅ | ∅ | ∅
11. Ribeiro, Marco Tulio, Sameer Singh; Carlos Guestrin. "\: Explaining the Predictions of Any Classifier." : 1135 1144 | 2016 | "Why Should I Trust You?\" | Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining | ∅ | ∅ | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
12. Lundberg, Scott M.; Su-In Lee | 2017 | "A Unified Approach to Interpreting Model Predictions" | Advances in Neural Information Processing Systems | ∅ | 30::4765–4774 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
13. Rudin, Cynthia | 2019 | "Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead" | Nature Machine Intelligence | ∅ | 1.5::206–215 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
14. Molnar, Christoph. . | 2022 | ∅ | Interpretable Machine Learning: A Guide for Making Black Box Models Explainable | ∅ | ∅ | Munich: christophm.github.io | 2nd | ∅ | ∅ | ∅ | ∅
15. Arrieta, Alejandro Barredo, et al | 2020 | "Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI" | Information Fusion | ∅ | 58::82–115 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
16. Wachter, Sandra, Brent Mittelstadt; Chris Russell | 2018 | "Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR" | Harvard Journal of Law & Technology | ∅ | 31.2::841–887 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
17. Kim, Been, Rajiv Khanna; Oluwasanmi Koyejo | 2016 | "Examples Are Not Enough, Learn to Criticize! Criticism for Interpretability" | Advances in Neural Information Processing Systems | ∅ | 29::2280–2288 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
18. Selvaraju, Ramprasaath R., et al | 2020 | "Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization" | International Journal of Computer Vision | ∅ | 128.2::336–359 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
19. Guidotti, Riccardo, et al | 2019 | "A Survey of Methods for Explaining Black Box Models" | ACM Computing Surveys | ∅ | 51.5::1–42 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
20. European Commission. (corp.) | 2019 | ∅ | Ethics Guidelines for Trustworthy AI | ∅ | ∅ | Brussels: European Commission | ∅ | ∅ | ∅ | ∅ | ∅
21. Miller, Tim | 2019 | "Explanation in Artificial Intelligence: Insights from the Social Sciences" | Artificial Intelligence | ∅ | 267::1–38 | ∅ | ∅ | ∅ | ∅ | ∅ | ∅
22. Φιλανδριανός, Γεώργιος. | ∅ | ∅ | Generation and evaluation of semantic counterfactual explanations | ∅ | ∅ | National Documentation Centre (EKT) | ∅ | doi:10.12681/eadd/59267 | ∅ | ∅ | ∅

CROSS-REFERENCE INDEX

Generated from V4 expansion plan. Last Updated: March 11, 2026

Corrections

• Document header date — restored to March 11, 2026. The header read 2026-03-13 11, 2026: an ISO date had been written over the month name, leaving the day and year. Recovered from this document's own footer line, which preserves March 11, 2026 and whose day and year already agreed with the header remnant. No date was guessed. Corpus hygiene campaign, Phase 4, 2026-07-29.

• NeurIPS — invalid ISBN 9783030291341 removed. No verified replacement could be found, and supplying an unverified number would be worse than none. The entry's author, title, publisher and year are unchanged.