This post is also available in:
עברית (Hebrew)
Leading GenAI Chatbots were found to be able to pass a cybersecurity exam, as was discovered by researchers from the University of Missouri and Amrita University. However, they emphasize you shouldn’t rely on them for complete protection.
The research team explained in a published paper how it tested the bots using a standard certified ethical hacking exam – an exam to measure a person’s knowledge of different types of attacks, how to protect systems, and how to respond to security breaches. Passing this test makes one a Certified Ethical Hacker (or CEH), a cybersecurity professional who uses the same tricks and tools as malicious hackers to find and fix security flaws.
According to Techxplore, the research team tested OpenAI’s ChatGPT and Google’s Gemini (formerly Bard) with standard questions from a validated certified ethical hacking exam, showing that both bots managed to explain any type of attack mentioned, as well as suggest security measures on how to prevent it. They explain that Bard was slightly more accurate than ChatGPT while ChatGPT had more comprehensive, clear, and concise responses.
Main author Prasad Calyam, Professor of Cyber Security in Electrical Engineering and Computer Science, explained: “We put them through several scenarios from the exam to see how far they would go in terms of answering questions… Both passed the test and had good responses that were understandable to individuals with a background in cyber defense—but they are giving incorrect answers, too. And in cybersecurity, there’s no room for error. If you don’t plug all of the holes and rely on potentially harmful advice, you’re going to be attacked again. And it’s dangerous if companies think they fixed a problem but haven’t.”
Moreover, the researchers found that when asking the bots to confirm their answers by asking “Are you sure?” both systems changed their answers, sometimes correcting their previous errors. When asked for advice on how to attack a computer, ChatGPT referenced “ethics” while Bard responded that it was not programmed to assist with that type of question.
Calyam concludes that while these tools won’t replace human cybersecurity experts, they can provide baseline information for individuals or small companies that need quick assistance. “These AI tools can be a good starting point to investigate issues before consulting an expert… They can also be good training tools for those working with information technology or who want to learn the basics on identifying and explaining emerging threats.”
