This post is also available in:
עברית (Hebrew)
A research initiative from Singapore Management University is setting out to address a longstanding problem in artificial intelligence: the difficulty in evaluating how AI models reach their decisions. While traditional software follows explicit instructions, AI—particularly systems based on neural networks—operates in ways that resist conventional analysis. This gap makes it hard to test, verify, or explain many AI-driven applications.
The project, “The Science of Certified AI Systems”, supported under Singapore’s Tier 3 research funding, is developing a formal basis for assessing AI behavior using concepts like abstraction, causality, and interpretability—core ideas in software engineering that must now be adapted to suit machine-learned systems, according to TechXplore.
Unlike standard code where cause-and-effect is easily traced, AI often draws conclusions based on correlations found during training. For instance, two people with similar profiles might receive opposite outcomes in a credit model due to factors hidden within the model’s learned parameters. This lack of transparency makes it difficult to guarantee fairness or safety.
To address this, the researchers aim to create practical tools—such as debugging environments and model checkers—specifically designed for neural networks. These will support the development of AI certification standards, similar to those used in safety-critical software today. Also, the researchers intend to create guidelines for developing AI systems.
An additional focus of the project is the integration of AI models into real-world systems. This includes identifying where deployment issues typically arise and what practices can help improve reliability. For example, ensuring open-source models include clear documentation—such as supported hardware or fallback mechanisms—can significantly reduce errors in field deployment.
In a related study, researchers examined how code-generating AI models often memorize and reproduce snippets from their training data. This raises concerns about security and intellectual property, especially if the model outputs proprietary or sensitive code. A recently proposed technique called machine unlearning aims to reduce this risk by training models to “forget” specific data without losing broader capabilities.
Together, these efforts reflect a growing recognition that building trustworthy AI requires not just accuracy, but clarity, control, and oversight—especially as these systems move from labs into critical infrastructure and commercial tools.
