Usman Anwar

I am a final year PhD student in Computational and Biological Learning lab at Cambridge University, UK. I am broadly interested in AI Safety and Alignment, with my recent focus on chain-of-thought monitorability. I am supervised by David Kruger and funded by Open Phil AI Fellowship and Vitalik Buterin Fellowship on AI Safety.

Email  /  GitHub  /  Google Scholar  /  LinkedIn  /  CV

If you want to chat or collaborate with me, or pitch me on what I should do post-PhD, please drop me an email.

A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoring

Current approaches to detecting hidden communication in AI systems rely on ad-hoc approaches such as inspecting messages for anomalies. We introduce a new framework that instead detects steganography through its behavioral effects; measuring whether a signal helps intended recipients more than outside monitors on real tasks.

Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory

We propose a simple training objective based on mutual information that prevents CoT obfuscation and maintains CoT monitorability when models are optimized against monitors. Through our theoretical analysis, we also characterize two possible failure modes for practical monitors: information gap, where the monitor cannot interpret the model’s reasoning, and elicitation error, where the monitor fails to correctly evaluate outputs for the target attribute.

Interpreting Emergent Planning in Model-Free Reinforcement Learning

We bring concept-based interpretability tools to a model-free reinforcement learning agent (DRC) and uncover that it learns internal plans reminiscent of bidirectional search. This planning circuitry both forecasts the downstream environment state and causally controls action selection, revealing how model-free policies can still leverage structured reasoning when given additional test-time compute.

Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens

We study how in-context learners built from transformers behave when faced with adversarial hijacking attacks. We find both linear and GPT-2 style transformers are surprisingly brittle, but adversarial training during pretraining or finetuning substantially improves robustness and generalizes to stronger attacks. By comparing across architectures and to classical algorithms for fitting linear models, we show that the learned in-context learning procedures differ qualitatively, as evidenced by poor attack transfer even between large models trained with identical recipes.

Foundational Challenges in Assuring Alignment and Safety of Large Language Models

This 150+ pages long agenda identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are organized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose 200+, concrete research questions.

Reward Model Ensembles Help Mitigate Overoptimization

We show that using an ensmeble of reward models can be effective in mitigating overoptimization.

Bayesian Methods for Constraint Inference in Reinforcement Learning

We develop a Bayesian approach for learning constraints which provides several advantages as it can work with partial trajectories, is applicable in both stochastic and deterministic environments and due to its ability to provide a posterior distribution enables use of active learning for accurate learning of constraints.

Inverse Constrained Reinforcement Learning

We propose a framework for learning Markovian constraints from user demonstrations in high dimensional, continuous settings. We empirically show that constraints thus learned are general and transfer well to agents with different dynamics and morphologies.

Design and source code from Leonid Keselman's website