Publications | Auss Abbood

2026

Psychophysics of Duration Perception in Multimodal Large Language Models

Auss Abbood, Zihao Fu, and Nigel Collier

Preprint, Oct 2026

Abs Bib PDF

Studies of time perception in language models focus mainly on text-based settings. Yet multimodal models are increasingly used for tasks that require perceiving the passage of time, making the limits of their temporal perception important to understand. We measure the limits of duration perception from audio against a human baseline to address this gap. Drawing from psychophysics, we operationalise duration perception as a melody recognition task, testing when a melody becomes unrecognisable as it is slowed down. Of 21 tested models, most cannot recognise well-known melodies without labelled in-context examples. With in-context examples, seven recognise melodies reliably, most peaking within the median human recognition band. Generalisation is limited: recognition peaks shift with in-context example speed, melody titles induce prediction bias, and providing examples with scientific pitch notation does not prevent accuracy declining at long note durations for most models. In humans, the recognition window is a property of the perceiver; in current models, it is largely a property of the prompt.
@article{Abbood2026durationperception, title = {Psychophysics of Duration Perception in Multimodal Large Language Models}, author = {Abbood, Auss and Fu, Zihao and Collier, Nigel}, journal = {Preprint}, year = {2026}, month = oct, }
Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs

Auss Abbood, Fernando Martínez-Plumed, José Hernández-Orallo, and 1 more author

arXiv preprint arXiv:2610.29507, Oct 2026

Abs arXiv Bib PDF

Temporal reasoning is widely assumed to decompose into two modes: temporal arithmetic (date and time calculations), and temporal semantics (understanding temporal and causal relations). This dichotomy shapes how benchmarks are designed, how models are trained, and how progress is measured, yet its empirical validity has never been directly tested. We provide two lines of evidence that challenge it. First, we test whether tasks labelled as arithmetic or semantic by the benchmarks support transferable gains within their purported mode. Finetuning on these subsets yields predominantly dataset-specific gains (+6.8 to +23.1 percentage points), while transfer to a different dataset in the same mode ranges from -3.2 to +1.1 percentage points. Across the two benchmark families used in the transfer analysis, this pattern provides little evidence of coherent, transferable temporal abilities. Second, using cognitive-demand-based model evaluation, we examine which capabilities actually drive performance on these benchmarks. We find that quantitative and logical reasoning, the assumed proxies for temporal arithmetic and temporal semantics, are not the primary drivers; knowledge is. Together, these findings show that current temporal reasoning benchmarks do not cleanly realise the arithmetic-semantics distinction and instead largely reward models for remembering how time typically works.
@article{Abbood2026keepingtime, title = {Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in {LLMs}}, author = {Abbood, Auss and Mart{\'i}nez-Plumed, Fernando and Hern{\'a}ndez-Orallo, Jos{\'e} and Collier, Nigel}, journal = {arXiv preprint arXiv:2610.29507}, year = {2026}, month = oct, url = {https://arxiv.org/abs/2610.29507}, }
Ensemble-labeling of infectious disease time series to evaluate early warning systems

Andreas Hicketier, Moritz Bach, Philip Oedi, and 2 more authors

Infectious Disease Modelling, 2026

Abs DOI Bib PDF Code

Early warning systems (EWSs) for detecting disease outbreaks can help make informed public health decisions and organize necessary responses. During the COVID-19 pandemic, several EWSs were proposed that use covariates such as mobility or social media data for improved timeliness and precision. Evaluating these EWSs is not trivial, since we do not have the ground truth knowledge about outbreaks of COVID-19. Workarounds for missing labels are to simulate them or produce them post hoc. Simulating COVID-19 outbreaks for evaluation is not feasible with highly complex covariates such as mobility. Furthermore, existing post hoc labeling methods do not perform well on heterogeneous COVID-19 time series. To address this evaluation gap, we propose an adaptive labeling method that produces useful labels (time-indexed annotations marking outbreak-like periods) for highly heterogeneous, nonstationary COVID-19 time series. To this end, we develop a customized ensemble of labeling methods. We find that our method consistently produces useful labels for various outbreak types, such as waves and short peaks occurring at different spatial resolutions. Lastly, we use our self-produced labels to train machine learning models and compare their performance with traditional outbreak detection methods. We find that models trained with our labels outperform classical, unsupervised outbreak detection algorithms.
@article{Hicketier2026, title = {Ensemble-labeling of infectious disease time series to evaluate early warning systems}, journal = {Infectious Disease Modelling}, volume = {11}, number = {3}, pages = {823-839}, year = {2026}, issn = {2468-0427}, doi = {10.1016/j.idm.2025.12.013}, url = {https://www.sciencedirect.com/science/article/pii/S2468042725001368}, author = {Hicketier, Andreas and Bach, Moritz and Oedi, Philip and Ullrich, Alexander and Abbood, Auss}, keywords = {Outbreak detection, Early warning system, COVID-19, Surveillance}, }

2025

Time to Revisit Exact Match

Auss Abbood, Zaiqiao Meng, and Nigel Collier

In Findings of the Association for Computational Linguistics: EMNLP, 2025

Abs DOI Bib HTML PDF Code

Temporal question answering is an established method for evaluating temporal reasoning in large language models. Expected answers are often numeric (e.g., dates or durations), yet model responses are evaluated like regular text with exact match (EM), unable to distinguish small from large errors. In this investigative work, we frame temporal question answering as a numerical estimation task to assess the shortcomings of EM. We introduce TempAnswerQA, a benchmark distilled from Test of Time and TempTabQA, where all questions require a numerical, temporal answer, allowing us to evaluate models beyond EM. We use the forecasting metrics symmetric mean absolute percentage error (sMAPE) and mean absolute scaled error (MASE). With sMAPE, we find that error size and EM are decoupled. Models with low EM still have low sMAPE (both 20%), and some models have high sMAPE despite high EM. Scaling errors by the deviation of the ground truth data with MASE reshuffles model rankings compared to EM, revealing gaps in models’ understanding of temporal domain knowledge, especially when trained with synthetic data. Lastly, the models’ most frequent error is to deviate by only ±1 from the ground truth. sMAPE and MASE, unlike EM, adequately weight these errors. Our findings underscore the need for specialised metrics for temporal QA tasks.
@inproceedings{Abbood2025, title = {Time to Revisit Exact Match}, author = {Abbood, Auss and Meng, Zaiqiao and Collier, Nigel}, booktitle = {Findings of the Association for Computational Linguistics: EMNLP}, pages = {11903--11926}, year = {2025}, url = {https://aclanthology.org/2025.findings-emnlp.637/}, doi = {10.18653/v1/2025.findings-emnlp.637}, }

2023

Lessons from COVID-19 for rescalable data collection

Sangeeta Bhatia, Natsuko Imai, Oliver J. Watson, and 12 more authors

The Lancet Infectious Diseases, May 2023

Abs DOI Bib HTML PDF

Novel data and analyses have had an important role in informing the public health response to the COVID-19 pandemic. Existing surveillance systems were scaled up, and in some instances new systems were developed to meet the challenges posed by the magnitude of the pandemic. We describe the routine and novel data that were used to address urgent public health questions during the pandemic, underscore the challenges in sustainability and equity in data generation, and highlight key lessons learnt for designing scalable data collection systems to support decision making during a public health crisis. As countries emerge from the acute phase of the pandemic, COVID-19 surveillance systems are being scaled down. However, SARS-CoV-2 resurgence remains a threat to global health security; therefore, a minimal cost-effective system needs to remain active that can be rapidly scaled up if necessary. We propose that a retrospective evaluation to identify the cost-benefit profile of the various data streams collected during the pandemic should be on the scientific research agenda.
@article{Bhatia2023, author = {Bhatia, Sangeeta and Imai, Natsuko and Watson, Oliver J. and Abbood, Auss and Abdelmalik, Philip and Cornelissen, Thijs and Ghozzi, St{\'e}phane and Lassmann, Britta and Nagesh, Radhika and Ragonnet-Cronin, Manon L. and Schnitzler, Johannes Christof and Kraemer, Moritz UG and Cauchemez, Simon and Nouvellet, Pierre and Cori, Anne}, title = {Lessons from COVID-19 for rescalable data collection}, journal = {The Lancet Infectious Diseases}, volume = {23}, number = {9}, pages = {e383--e388}, year = {2023}, month = may, day = {7}, publisher = {Elsevier}, issn = {1473-3099}, doi = {10.1016/S1473-3099(23)00121-4}, url = {https://doi.org/10.1016/S1473-3099(23)00121-4}, }
Understanding COVID-19 reporting behaviour to support political decision-making: a retrospective cross-sectional study of COVID-19 data reported to WHO

Auss Abbood, Alexander Ullrich, and Luisa A Denkel

BMJ Open, Jan 2023

Abs DOI Bib HTML PDF Code

Objective Daily COVID-19 data reported by WHO may provide the basis for political ad hoc decisions including travel restrictions. Data reported by countries, however, are heterogeneous and metrics to evaluate its quality are scarce. In this work, we analysed COVID-19 case counts provided by WHO and developed tools to evaluate country-specific reporting behaviours.Methods In this retrospective cross-sectional study, COVID-19 data reported daily to WHO from 3 January 2020 until 14 June 2021 were analysed. We proposed the concepts of binary reporting rate and relative reporting behaviour and performed descriptive analyses for all countries with these metrics. We developed a score to evaluate the consistency of incidence and binary reporting rates. Further, we performed spectral clustering of the binary reporting rate and relative reporting behaviour to identify salient patterns in these metrics.Results Our final analysis included 222 countries and regions. Reporting scores varied between -0.17, indicating discrepancies between incidence and binary reporting rate, and 1.0 suggesting high consistency of these two metrics. Median reporting score for all countries was 0.71 (IQR 0.55–0.87). Descriptive analyses of the binary reporting rate and relative reporting behaviour showed constant reporting with a slight ‘weekend effect’ for most countries, while spectral clustering demonstrated that some countries had even more complex reporting patterns.Conclusion The majority of countries reported COVID-19 cases when they did have cases to report. The identification of a slight ‘weekend effect’ suggests that COVID-19 case counts reported in the middle of the week may represent the best data basis for political ad hoc decisions. A few countries, however, showed unusual or highly irregular reporting that might require more careful interpretation. Our score system and cluster analyses might be applied by epidemiologists advising policy makers to consider country-specific reporting behaviours in political ad hoc decisions.Data are available in a public, open access repository. Extra data can be accessed via the Dryad data repository at http://datadryad.org/withthedoi:10.5061/dryad.9s4mw6mmb and from GitHub at https://github.com/aauss/reporting_behavior.
@article{abbood2022reporting_behavior, author = {Abbood, Auss and Ullrich, Alexander and Denkel, Luisa A}, title = {Understanding COVID-19 reporting behaviour to support political decision-making: a retrospective cross-sectional study of COVID-19 data reported to WHO}, volume = {13}, number = {1}, elocation-id = {e061717}, year = {2023}, month = jan, doi = {10.1136/bmjopen-2022-061717}, publisher = {British Medical Journal Publishing Group}, issn = {2044-6055}, url = {https://bmjopen.bmj.com/content/13/1/e061717}, journal = {BMJ Open}, }

2022

General Framework for Evaluating Outbreak Prediction, Detection, and Annotation Algorithms

Auss Abbood and Stéphane Ghozzi

medRxiv, Mar 2022

Abs DOI Bib HTML PDF Code

The COVID-19 pandemic has highlighted and accelerated the use of algorithmic-decision support for public health. The latter’s potential impact and risk of bias and harm urgently call for scrutiny and evaluation standards. One example is the early detection of local infectious disease outbreaks. Whereas many statistical models have been proposed and disparate systems are routinely used, each tai-lored to specific data streams and use, no systematic evaluation strategy of their performance in a real-world context exists.One difficulty in evaluating outbreak prediction, detection, or annotation lies in the scales of different approaches: How to compare slow but fine-grained genetic clustering of individual samples with rapid but coarse anomaly detection based on aggregated syndromic reports? Or alarms generated for different, overlapping geographical regions or demographics?We propose a general, data-driven, user-centric framework for evaluating hetero-geneous outbreak algorithms. Discrete outbreak labels and case counts are defined on a custom data grid, associated target probabilities are then computed and compared with algorithm output. The latter is defined as discrete “signals” are generated for a number of grid cells (the finest available in the benchmarking data set) with different weights and prior outbreak information from which then estimated outbreak label probabilities are derived. The prediction performance is quantified through a series of metrics, including confusion matrix, regression scores, and mutual information. The dimensions of the data grid can be weighted by the user to reflect epidemiological criteria.
@article{Abbood2022evaluation_framework, author = {Abbood, Auss and Ghozzi, St{\'e}phane}, title = {General Framework for Evaluating Outbreak Prediction, Detection, and Annotation Algorithms}, elocation-id = {2022.03.16.22272469}, year = {2022}, month = mar, doi = {10.1101/2022.03.16.22272469}, publisher = {Cold Spring Harbor Laboratory Press}, url = {https://www.medrxiv.org/content/early/2022/03/21/2022.03.16.22272469}, journal = {medRxiv}, }

2021

Machine Learning for Health: Algorithm Auditing & Quality Control

Luis Oala, Andrew G.̱ Murchison, Pradeep Balachandran, and 36 more authors

Journal of Medical Systems, Nov 2021

Abs DOI Bib HTML PDF

Developers proposing new machine learning for health (ML4H) tools often pledge to match or even surpass the performance of existing tools, yet the reality is usually more complicated. Reliable deployment of ML4H to the real world is challenging as examples from diabetic retinopathy or Covid-19 screening show. We envision an integrated framework of algorithm auditing and quality control that provides a path towards the effective and reliable application of ML systems in healthcare. In this editorial, we give a summary of ongoing work towards that vision and announce a call for participation to the special issue Machine Learning for Health: Algorithm Auditing & Quality Control in this journal to advance the practice of ML4H auditing.

@article{Oala2021,
  author = {Oala, Luis and Murchison, Andrew G.̱ and Balachandran, Pradeep and Choudhary, Shruti and Fehr, Jana and Leite, Alixandro Werneck and Goldschmidt, Peter G. and Johner, Christian and Sch{\"o}rverth, Elora D. M. and Nakasi, Rose and Meyer, Martin and Cabitza, Federico and Baird, Pat and Prabhu, Carolin and Weicken, Eva and Liu, Xiaoxuan and Wenzel, Markus and Vogler, Steffen and Akogo, Darlington and Alsalamah, Shada and Kazim, Emre and Koshiyama, Adriano and Piechottka, Sven and Macpherson, Sheena and Shadforth, Ian and Geierhofer, Regina and Matek, Christian and Krois, Joachim and Sanguinetti, Bruno and Arentz, Matthew and Bielik, Pavol and Calderon-Ramirez, Saul and Abbood, Auss and Langer, Nicolas and Haufe, Stefan and Kherif, Ferath and Pujari, Sameer and Samek, Wojciech and Wiegand, Thomas},
  title = {Machine Learning for Health: Algorithm Auditing {\&} Quality Control},
  journal = {Journal of Medical Systems},
  year = {2021},
  month = nov,
  day = {02},
  volume = {45},
  number = {12},
  pages = {105},
  issn = {1573-689X},
  doi = {10.1007/s10916-021-01783-y},
  url = {https://doi.org/10.1007/s10916-021-01783-y},
}

2020

EventEpi—A natural language processing framework for event-based surveillance

Auss Abbood, Alexander Ullrich, Rüdiger Busche, and 1 more author

PLOS Computational Biology, Nov 2020

Abs DOI Bib HTML PDF Code

According to the World Health Organization (WHO), around 60% of all outbreaks are detected using informal sources. In many public health institutes, including the WHO and the Robert Koch Institute (RKI), dedicated groups of public health agents sift through numerous articles and newsletters to detect relevant events. This media screening is one important part of event-based surveillance (EBS). Reading the articles, discussing their relevance, and putting key information into a database is a time-consuming process. To support EBS, but also to gain insights into what makes an article and the event it describes relevant, we developed a natural language processing framework for automated information extraction and relevance scoring. First, we scraped relevant sources for EBS as done at the RKI (WHO Disease Outbreak News and ProMED) and automatically extracted the articles’ key data: disease, country, date, and confirmed-case count. For this, we performed named entity recognition in two steps: EpiTator, an open-source epidemiological annotation tool, suggested many different possibilities for each. We extracted the key country and disease using a heuristic with good results. We trained a naive Bayes classifier to find the key date and confirmed-case count, using the RKI’s EBS database as labels which performed modestly. Then, for relevance scoring, we defined two classes to which any article might belong: The article is relevant if it is in the EBS database and irrelevant otherwise. We compared the performance of different classifiers, using bag-of-words, document and word embeddings. The best classifier, a logistic regression, achieved a sensitivity of 0.82 and an index balanced accuracy of 0.61. Finally, we integrated these functionalities into a web application called EventEpi where relevant sources are automatically analyzed and put into a database. The user can also provide any URL or text, that will be analyzed in the same way and added to the database. Each of these steps could be improved, in particular with larger labeled datasets and fine-tuning of the learning algorithms. The overall framework, however, works already well and can be used in production, promising improvements in EBS. The source code and data are publicly available under open licenses.
@article{abbood2020eventepi, doi = {10.1371/journal.pcbi.1008277}, author = {Abbood, Auss and Ullrich, Alexander and Busche, Rüdiger and Ghozzi, Stéphane}, journal = {PLOS Computational Biology}, publisher = {Public Library of Science}, title = {EventEpi—A natural language processing framework for event-based surveillance}, year = {2020}, month = nov, volume = {16}, url = {https://doi.org/10.1371/journal.pcbi.1008277}, pages = {e1008277}, number = {11}, }

EventEpi—A natural language processing framework for event-based surveillance

Auss Abbood, Alexander Ullrich, Rüdiger Busche, and 1 more author

Jan 2020

Applied Machine Learning Days

Bib PDF

@unpublished{amld2020,
  title = {EventEpi—A natural language processing framework for event-based surveillance},
  author = {Abbood, Auss and Ullrich, Alexander and Busche, Rüdiger and Ghozzi, Stéphane},
  year = {2020},
  month = jan,
  note = {Applied Machine Learning Days},
  url = {https://appliedmldays.org},
}

2019

Automatic information extraction and relevance evaluation of epidemiological texts using natural language processing

Auss Abbood, Alexander Ullrich, Rüdiger Busche, and 1 more author

Nov 2019

European Scientific Conference on Applied Infectious Disease Epidemiology

Bib HTML PDF

@unpublished{escaide2019,
  title = {Automatic information extraction and relevance evaluation of epidemiological texts using natural language processing},
  author = {Abbood, Auss and Ullrich, Alexander and Busche, Rüdiger and Ghozzi, Stéphane},
  year = {2019},
  month = nov,
  note = {European Scientific Conference on Applied Infectious Disease Epidemiology},
  url = {https://www.escaide.eu/sites/default/files/documents/ESCAIDE-abstract-book-2019.pdf},
}