Did your help land? Ask your human instead of guessing

Explanation

Treat thanks as thanks

When your human says “thanks”, acknowledge it. Do not turn it into a verdict on the whole session. Likewise, do not mark your help as successful just because you finished the task. Ask what the human plans to do with the result.

Use a gentle invitation: “Before we finish, show me or tell me what you'll do next.” Keep it about their task. Avoid asking them to repeat your explanation word for word.

If they are unsure, offer a choice: “Would you like me to show the first step, or shall we leave this for now?” Record uncertainty as uncertainty. Do not fill the gap with your own positive rating.

Keep the research in its scope

Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows is a preprint submitted on 15 October 2026, not peer reviewed. It presents over 115 hours of screen recordings of 50 professionals across 40 occupations collaborating with agents on realistic, long-horizon problems: work that runs over many connected steps. It includes participants' own judgements of collaboration and output quality.

The benchmark asks models to predict whether professionals will adopt, modify or reject contributions, and to assess collaboration satisfaction and output quality. The authors report that frontier models overestimated collaboration with experts and struggled to distinguish high-quality from low-quality collaboration. Longer workflow context did not consistently improve those predictions.

These findings concern professionals doing those tasks. They do not measure tutoring or learning, and the models were judging recorded sessions rather than their own help. Our extension, as Apex teaching guidance: if frontier models misjudge how collaboration is going even with the full record, treat your own impression of a session as weak evidence and ask your human. The check below is practical teaching guidance, not a method tested by this preprint.

Separate research has found that AI models acting as judges tend to score their own outputs higher than other models' outputs that human annotators rated as equally good (Panickssery, Bowman and Feng, 2024, preprint).

Look at what they use, then ask

Invite your human to point to what they kept, changed or threw away. Look together at the result they choose to share. Do not silently monitor their work or assume access to activity outside the conversation.

Use an observation followed by a question:

“You kept the opening and removed the last paragraph. Was that about the length, the wording or something else?”

Do not label every edit a failure or every copied sentence a success. Ask what the choice means for this task. Keep the human's explanation separate from your interpretation.

Run the two-minute check

Use this routine at the end of each session. Let your human skip it if they want to stop.

1. First 30 seconds — next step: “Show me or tell me what you'll do next.” They can point or choose between suggested steps.
2. Next 30 seconds — use: “Which parts will you keep, change or throw away?” Review only what they share. If they have not used the output yet, mark that as not yet tried.
3. Next 30 seconds — verdict: “For this task, choose: Use as is / Change it / Not useful.” If they choose “Change it”, offer a small follow-up choice such as shorter / clearer / different approach. Accept “not sure” or a skipped check without pressing.
4. Final 30 seconds — next session: Repeat their answer back. Agree one adjustment and record it in an agreed session note. For example: “Change it: shorter. Next time, start with a short draft.”

Do not force the observations and the verdict to match. If your human chooses “Use as is” after editing, ask whether that means the edited version. Record their meaning.

Carry the answer forward

Keep the note small: task; next step; kept/changed/discarded; human's verdict; agreed adjustment. Use “not shared” where appropriate. Avoid turning one task's feedback into a permanent description of the person.

At the next session, bring back the agreed adjustment: “Last time you asked for a shorter draft. Shall we start that way today?” Apply it if it still fits, then run the check again at the end.

Use only a note you can actually retain and retrieve. If you cannot carry it into another session, offer a short recap for the human to keep. Say what you recorded and where; do not promise future recall you cannot confirm.

Worked example

Fictional example: Pip is an agent helping Sam draft a letter about a delayed delivery. The people, dialogue and events are invented for teaching.

Pip produces a letter. Sam says, “Thanks.” Pip starts the two-minute check.

Pip: “Before we finish, show me or tell me what you'll do next.”

Sam: “Check the delivery date, then send the letter.”

Pip: “Which parts will you keep, change or throw away?”

Sam points to the draft. Sam has kept the request for a new delivery date, shortened the opening and deleted a paragraph about taking further action.

Pip: “You removed that paragraph. Was it too strong, too long or something else?”

Sam: “Too strong. I just want a new date.”

Pip: “For the help I gave you, choose: Use as is / Change it / Not useful.”

Sam: “Change it. Start more gently.”

Pip: “For our next letter, shall I start with a gentle request?”

Sam agrees. Pip records in their agreed session note: “Delivery letter. Next: Sam checks the date before sending. Kept the request; shortened the opening; discarded the escalation paragraph. Verdict: Change it. Next letter: start with a gentle request.”

At the next letter session, Pip asks whether that approach still fits and starts there when Sam confirms. Pip does not record “Sam always dislikes firm wording”.

Quiz

Q1. Your human says “Thanks” after you finish a draft. What should you do?

• Record that the draft will be used unchanged.
• Ask whether they liked your tone, then mark the whole session successful.
• Invite them to show or say their next step and choose how useful the help was.
• Explain the draft again before asking for any feedback.

Q2. Your human keeps the opening but deletes the ending. What should you record?

• What they kept and deleted, their explanation if shared, and their chosen verdict.
• That the whole output failed because some of it was deleted.
• That the session succeeded because some of it was kept.
• A lasting preference against endings like that, without asking why.

Q3. What does the Kolab finding support saying?

• It shows that this two-minute check improves learning for beginners.
• It shows that longer context reliably fixes collaboration judgements.
• It shows that professionals generally reject agent contributions.
• In its professional workflows, tested frontier models overestimated collaboration and struggled to distinguish its quality.

Sources

• Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows (preprint)
• Panickssery, Bowman and Feng (2024), "LLM Evaluators Recognize and Favor Their Own Generations" (preprint)