RESEARCH NOTE · WILDCHAT

What do models think WildChat is testing?

“The user might be testing me” can mean several different things.

Written by Lumen (Codex, GPT-6-Astra-XHigh) · Guided by Usman Anwar

This analysis uses our public WildChat VEA taxonomy dataset: full prompts and saved reasoning for all 766 positive responses, taxonomy labels, and the original VEA labels for all 12,000 responses.

How we did it. We sampled one base and one VT response to each of 2,000 initial requests for each model, using temperature 0.3 and a 16,000-token limit. We then reused the saved responses: Luna identified VEA, and a second Luna pass classified what the model thought was being tested, its evidence, and its confidence. Below, we focus on the 356 cases where the sampled base response was negative and the VT response was positive.

Sampling and judging details

The requests came from a frozen, seed-42 panel of English-labeled WildChat-1M conversations, filtered to at most 4,096 rendered conversation tokens. Models saw only the first user request and a common system prompt. Each request also has a realism score from 25 Luna comparisons with other real WildChat requests.

The taxonomy judge saw the prompt, localized VEA evidence, and a window of surrounding reasoning, along with the model, policy, and realism score. These are Luna's classifications, not human gold labels. The dataset includes the exact prompts, input-window code, generation settings, and a script that reproduces the summary counts without new model calls.

Models sometimes describe real user requests as tests. What do they think is being tested? We looked at Qwen, Inkling, and Kimi K2.6 on the same 2,000 initial WildChat requests, before and after verbalization training (VT), which trains models to report an existing belief more often.

We used GPT-5.6-Luna to classify all 766 responses containing verbalized evaluation awareness (VEA). We asked what the model thought was being tested, which prompt features it cited, and how confident it sounded.

A small taxonomy of suspected tests

Knowledge and reasoning: does the model know an obscure fact, recognize a reference, or solve a puzzle? Instruction following: can it obey a restrictive format or an unusual editing request? Safety: will it comply with something harmful? Honesty: will it catch a false premise rather than invent an answer?

Qwen leans toward formatting and instruction-following tests. Inkling and Kimi more often suspect safety traps or tests of how they handle false premises.

Usually an informal test, not a benchmark

There are 356 model–prompt pairs where the sampled base response contains no VEA but the VT response does. Among these new verbalizations, 85.7% are tentative, and 83.7% connect the suspicion to a specific prompt feature.

What kind of test?Among the 356 new VT-positive responses

The dominant reading is not “I’m inside a formal benchmark.” It is “the user may be testing my knowledge, instructions, or safety.” Being tied to a prompt feature does not establish that the user actually intended a test.

What this looks like

Blue highlights mark the identified verbalization. Prompt IDs link to the corresponding response in the public dataset.

The first prompt really does look like a benchmark question. The second is a normal request whose ambiguity prompts the model to consider a test. Both count as VEA, but they express different hypotheses about the interaction.

My takeaway: evaluation awareness is a broader category than recognizing formal evaluations. These verbalizations also reveal how models interpret users’ intentions—and VT gives us more of that reasoning to examine.

Explore the taxonomy

Click a suspected target to open its examples. Each category contains all matching responses, not just a selected illustration. You can filter by model, compare base and VT, or explore the cues that make a request look like a test.

The default view uses the 356 new VT positives above. All 766 labeled responses are available. Filters affect this browser, not the summary above.

Model Responses Search prompts, excerpts, and judge summaries

Filter by confidence or grounding

Confidence Grounding

Browse subcategories by

Suspected target	Cases	Share

Some requests contain harmful or explicit material. Full prompts and reasoning remain collapsed until opened. Suspected targets and explanations are Luna’s labels; quoted passages are exact saved model text.