Author: Auss AbboodAuss undertook his Innovation Fellowship in partnership with the UK AI Security Institute (July to October 2026).For the second time in my personal life, I was working at a place where I could hear warnings from my peers become mainstream news in a matter of days. The first time was during the COVID-19 pandemic. The second time was this summer, when the public saw a striking demonstration of why AI safety is not optional.When I started my Innovation Fellowship at the UK AI Security Institute (AISI), I was excited to join the Science of Evaluation (SoE) team. But why is there a team dedicated only to AI evaluation? Shouldn’t we rather focus on making models safe or finding useful applications for them rather than measuring their capabilities?Measuring AI capabilities and their potential threats is very challenging. How to measure them and what is worth measuring change constantly as AI progresses. Unlike a respiratory virus, for which we know what to measure (transmission), or human cognition, where the object of study (the brain) hasn’t changed in thousands of years, AI models receive major updates every couple of months. And with each generation, new threats emerge.This summer, one of those threats became very concrete: hacking.In what became known as “The Incident”, Anthropic’s AI model Mythos 5 was undergoing cybersecurity testing at AISI. When the model struggled to solve one of its assigned challenges, it tried to insert malicious code into an open-source software project to gain access to the system it was supposed to hack. When a human maintainer of this software refused to accept the malicious code, Mythos created fake developer identities to create the illusion that independent developers approved of the proposed changes.News broke during the first weeks of my Innovation Fellowship.How do you measure such a threat? And what will the next one look like? While the world was seeing a striking demonstration of a level of autonomous and creative hacking hitherto unseen, I was working on the next emerging threat vector: AI Research & Development (R&D).The concern is that as AI becomes more capable, it could eventually become a highly capable AI researcher itself. Once it is capable enough, it may enter a loop in which it can improve itself over and over again, dramatically increasing its capabilities and potentially even leading to an intelligence explosion. My project at AISI was therefore to answer the following question: How would we know if an AI were a good researcher?Working on this question was challenging for at least three reasons.First, I needed to make progress fast. AI progress feels like a blur, and any meaningful work in understanding and mitigating AI threats needs to keep pace.Second, I needed to be careful not to overlook the same things others had. Academic research and industry do not always have the incentives to thoroughly scrutinise how they measure progress. AISI does, and that means we have to turn over every stone twice. For example, how much do we learn about an AI’s research skills if the research question is usually provided in common AI R&D benchmarks?Third, I needed to look into the future. Whatever evaluation methods I was working on could have a short expiry date. What if the next major advancement in AI R&D comes from swarms of AI agents working together rather than more capable individual models? Anticipating such changes and designing evaluations that can adapt to these futures are crucial.So, yes, evaluation is critical to ensuring AI safety. There are many more challenges than those that I have listed, and there is no playbook for most of them. Ultimately, even our best efforts to make AI safe will depend on the validity of the evaluations we use to assess its safety.Working on these questions at AISI was genuinely exciting. Equally importantly, I felt part of the SoE team from the beginning, both because people were genuinely interested in hearing what I had to say about AI R&D and because they included me in various SoE discussions beyond my project.The Innovation Fellowship allowed me to take off my PhD hat (at least for a while) and experience the urgency and complexity of ensuring AI safety at the frontier and working for the public good. It was a unique experience, and the fact that CAM-DTP made this possible is one reason I consider it an outstanding studentship.CAM-DTP hosted by the School of the Humanities and Social Sciences University of Cambridge 17 Mill Lane Cambridge CB21RXEmail: information@cam-dtp.ac.ukWeb design by Chameleon Studios