Usman Anwar

GLM-5.1 Has ~Astra Level CoT Control

We run the CoT-Control evaluation suite on GLM, Kimi and DeepSeek models and find that GLM models are outliers in their ability to control their CoT, yet GLM-5.1 performs poorly at monitor evasion tasks.

Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability

We propose an unsupervised post-training objective based on maximizing mutual information between CoT and the output for improving CoT monitorability. Across three model organisms with degraded CoT, it recovers most or all of the lost monitorability.

Training LLMs to Verbalize Evaluation Awareness

Evaluation awareness in LLMs is increasing, which can undermine the validity of capability and alignment evaluations. This paper introduces a training method that makes LLMs verbalize their evaluation awareness more reliably, so it can be detected and accounted for.

Paraphrasing Is (At Best) a Partial Defence Against Steganography in LLMs

We give a taxonomy of steganographic channels in LLM-generated text and show that paraphrasing cannot detect or defend against some types that LLMs have been shown to use, and is unreliable against the others.

A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoring

Current approaches to detecting hidden communication in AI systems rely on ad-hoc approaches such as inspecting messages for anomalies. We introduce a new framework that instead detects steganography through its behavioral effects: measuring whether a signal helps intended recipients more than outside monitors on real tasks.

Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory

We propose a simple training objective based on mutual information that prevents CoT obfuscation and maintains CoT monitorability when models are optimized against monitors. Through our theoretical analysis, we also characterize two possible failure modes for practical monitors: information gap, where the monitor cannot interpret the model’s reasoning, and elicitation error, where the monitor fails to correctly evaluate outputs for the target attribute.

Mitigating Goal Misgeneralization via Minimax Regret

We show that goal misgeneralization is possible under approximate optimization of the standard maximum expected value objective but not under minimax expected regret, and empirically find regret-based unsupervised environment design to be more robust to it than domain randomization.

The Reality of AI and Biorisk

We review the evidence behind two AI biorisk threat models, LLM-assisted information access and planning, and AI-enabled biological design tools, and find that existing studies are nascent and methodologically limited, with current models not posing an immediate risk.

Interpreting Emergent Planning in Model-Free Reinforcement Learning

We provide the first mechanistic evidence that standard model-free RL agents can perform planning by using implicitly learned world-models.

Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens

We show that transformers do not learn adversarially robust in-context learning algorithms for linear regression and that these learned algorithms do not correspond to classical algorithms for linear regression. We also provide evidence of non-universality in terms of algorithms learned by identical transformers.

Learning to Forget using Hypernetworks

We introduce HyperForget, a machine unlearning framework that uses diffusion-based hypernetworks to sample models that lack knowledge of targeted data while preserving performance on the remaining data.

A Sober Look at Steering Vectors for LLMs

We discuss challenges facing activation steering methods: they are often unreliable and fail to generalize beyond their training setup, commonly used metrics overestimate their effectiveness, and methods are rarely compared on the same benchmarks.

Comparing Bottom-Up and Top-Down Steering Approaches on In-Context Learning Tasks

We compare function vectors (bottom-up) and in-context vectors (top-down) for steering LLMs on in-context learning tasks, and find each is effective only on specific types of tasks: in-context vectors are better at behavioral shifting, while function vectors excel at tasks requiring more precision.

Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Games

We formulate noisy zero-shot coordination, where agents observe different noisy versions of the true coordination problem rather than sharing common knowledge of it, and show that meta-learning across a distribution of such problems trains RL agents that coordinate well with novel partners.

IDs for AI Systems

We propose a framework in which IDs are ascribed to instances of AI systems so that parties interacting with them can access relevant information, and analyze where IDs would be useful, how adoption could be incentivized, and their limitations and risks.

Foundational Challenges in Assuring Alignment and Safety of Large Language Models

This 150+ page agenda identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are organized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose 200+ concrete research questions.

Reward Model Ensembles Help Mitigate Overoptimization

We show that using an ensemble of reward models can be effective in mitigating overoptimization.

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

We survey open problems and fundamental limitations of RLHF, overview techniques to understand, improve, and complement it in practice, and propose auditing and disclosure standards to improve societal oversight of RLHF systems.

Bayesian Methods for Constraint Inference in Reinforcement Learning

We develop a Bayesian approach for learning constraints which provides several advantages as it can work with partial trajectories, is applicable in both stochastic and deterministic environments and due to its ability to provide a posterior distribution enables use of active learning for accurate learning of constraints.

Inverse Constrained Reinforcement Learning

We propose a framework for learning Markovian constraints from user demonstrations in high dimensional, continuous settings. We empirically show that constraints thus learned are general and transfer well to agents with different dynamics and morphologies.

Design and source code from Leonid Keselman's website