This post is also available in: heעברית (Hebrew)

By comparing a machine-learning algorithm with human reasoning, the user can be exposed to the ‘thinking’ patterns of the system and be able to understand how it works. 

In the field of machine learning, understanding why a particular decision was made is often as important as assessing its value. There are many tools available to experts now that enable them to explain how the computerized process works, but these methods can typically only provide data on one decision at a time, requiring manual evaluation, which takes considerable time. In training, Machine Learning often deals with massive amounts of data, overcoming human limitations in terms of the time required for in-depth processing and identifying and extracting clear patterns. 

Research from the Massachusetts Institute of Technology (MIT) and IBM Research has recently developed a method for collecting, sorting, and evaluating individual explanations to analyze a machine learning model’s behavior quickly. Shared Interest, the technique developed by the researchers, incorporates quantifiable metrics that allow a direct comparison between the model’s thinking and that of humans. This allows the user to identify patterns in the model’s decision-making process, for example, perhaps the model becomes easily distracted and confused by objects in the background of pictures. By analyzing these insights, the user can quickly determine if the model is reliable and ready to be used in the real world.

Researchers hoped that the speeded up analysis process would help human users better understand why a model did certain things and how machine learning makes certain decisions (saliency method). When the model classifies images, the methods focus on areas in the images that are relevant to the decision-making process. Using a salinity map that overlaps the original image, data is collected for different areas. When the model classifies the image as a dog, it highlights the dog’s head as these pixels were essential to the model’s classification of the image.

In this way, the saliency data generated by the model is compared to the ground truth data generated by humans for the same image to determine whether they are aligned. The technique can also be applied to text-based data, which will highlight and categorize keywords instead of images, according to According to the researchers, as long as the Shared Interest method is based on reliable methods, it has good performance, and they intend to apply it soon to other types of data, such as tabular data.