AI chat make software easier, but not faster, to use

Does adding an AI chatbot to software make it easier and faster to get things done? Yes and no.

Researchers from Japan and the Philippines recently looked at how users delegate, interact with, and otherwise complete tasks using AI, manual, or hybrid approaches.

From their research, they found that chat-based AI does reduce the effort required to interact with software (fewer clicks, page navigations, and scrolls) but does not necessarily help users complete tasks any faster.

This makes sense: if AI can parse user intent and align it with software capabilities, the amount of interacting with an interface goes down.

It’s the difference between being able to tell software, “change the email address on my account,” in your own words rather than having to look through a series of buttons and menus to find where that setting lives.

In the former scenario, it doesn’t matter whether you enter “account email,” “email address,” or something in between. You don’t need to know the product’s exact terminology or navigation structure to get where you want to go. In the latter, you may not immediately recognize a cog (or “flower”) symbol as being the sole entry point for changing your email address.

A successful agent handoff can effectively collapse the locate, then navigate, then act sequence into a single instruction.

But that doesn’t mean the user’s work is finished.

The researchers found no statistical difference in total task completion time between traditional, AI-first, and hybrid interfaces. AI-first was descriptively the fastest of the three, but the difference wasn’t large enough to conclude the interaction mode itself is what caused users to finish their tasks faster.

Which reminds me of an insight from Mica R. Endsley, former Chief Scientist of the U.S. Air Force, in her book Designing for Situation Awareness. Endsley explains that even with automation, evaluating whether software outcomes align with our intentions often requires us to understand the work behind the task anyway. Endsley writes:

“Checking the output or performance of a system is often not possible without doing the task oneself. For example, something as simple as checking that a computer is adding correctly is almost impossible to do without performing the calculations oneself.”

Put another way: AI can reduce the work of doing a task without removing the work of understanding the task well enough to know whether it was done correctly.

Of course, verifying an AI’s actions doesn’t always require as much effort as doing the work manually. For simple, easily observable tasks, verification might take almost no effort at all.

But for more consequential, or harder to inspect, tasks, the user’s job increasingly shifts from operating the interface to supervising the outcome.

When software acts on our behalf, we may still need to understand what happened, determine whether it matches our intent, and know how to remedy things when it doesn’t.

Yes, chat-based AI can provide an easier path into software. But making that path available doesn’t necessitate removing the buttons, controls, and direct-manipulation interfaces people already understand.

And not everyone will want to delegate to AI in the first place.

An additional finding from the researchers showed that AI usage varied much more from person to person than from task to task.

That’s why the researchers arrive at a more nuanced set of design recommendations: don’t make AI delegation the default in software. Optimize for handoff between user and agent rather than building for conversation.

In other words, the goal shouldn’t be to make every interface conversational. It should be to let people move fluidly between doing and delegating.