[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16]

Journal of Information Science and Engineering, Vol. 39 No. 4, pp. 711-724

XAI for Image Captioning using SHAP

1Department of Information Technology
Satya Wacana Christian University
Salatiga 50517, Indonesia

2Department of Information Management
Chaoyang University of Technology
Taichung City, 413310 Taiwan

3School of Creative Technologies
University of Portsmouth, PO1 2UP UK
4Department of Mathematics and Computer Science
University of Münster
D-48149 Münster, Germany
E-mail: crching@cyut.edu.tw

In the fields of computer vision (CV) and natural language processing (NLP), they attempt to create a textual description of a given image is known as image captioning. Captioning is the process of creating an explanation for an image. Recognizing the sig-nificant items in an image, their qualities, and their connections are required for image captioning. It must also be able to construct phrases that are valid in both syntax and semantics. Deep-learning-based approaches are deal with the intricacies and problems of image captioning. This article provides a simple and effective Explainable Artificial Intelligence (XAI) technique for image text. Deep learning techniques have been widely applied to this work in recent years, and the results have been relatively positive. This work employs Azure Cognitive Service and Open-Source Image Captioning model to get image caption. We implement Explainable Artificial Intelligence (XAI) Image Cap-tioning (Image to Text) using Shapley Additive explanations (SHAP). This work applies Cosine similarity by spaCy and Term Frequency Inverse Document Frequency (TF-IDF transform) to evaluate the sentence similarity. Our research work found that Azure Cog-nitive Services provides better descriptions for images compared to the Open-Source Image Captioning Model.

Keywords: SHAP, explainable artificial intelligence, image captioning, azure cognitive service, API

  Retrieve PDF document (JISE_202304_01.pdf)