Why AI Agents Still Fail at Real-World Tasks

AI agents sound great in theory. Give one a job, let it work through the steps, and come back when it’s finished. 

But in practice, you may still spend a fair amount of time checking what it did, and OSWorld 2.0 shows why.  

Its benchmark tests agents on 108 computer tasks that involve things people actually do at work: moving between apps, gathering information, responding to new instructions, and completing a task properly. 

From their results, a skilled human takes a median of about 1.6 hours to complete one, while the best-performing system currently finishes only 20.6% of them end-to-end. 

Where do AI agents lose the plot?

One example, in particular, stood out to me. During a purchase-order task, an agent noticed that new approvals had arrived in its messages. But it didn’t properly update the list it was working from. When it finished, an approved request was still missing. 

So, this wasn’t a case of the agent never seeing the information. It saw it. But somewhere between receiving the update and finishing the job, it lost track of what had changed. 

That’s a problem if the whole point of using an agent is to stop watching every step.