When it comes to human-computer interaction, it’s no longer just about typing commands or tapping screens.
The relationship between humans and machines has evolved, growing more intuitive, conversational, and emotional. Today’s interfaces aren’t just tools; they’re becoming partners in thought, creativity, and decision-making. And Microsoft showcases this in an update to its Copilot AI.
In a website post, The Redmond-giant announces a "new computer use in Microsoft Copilot Studio for UI automation."
This computer use ability enables AI agents to interact with websites and desktop apps just like how humans would.
Like clicking buttons, selecting menus, and typing into fields on the screen.
This allows agents to handle tasks even when there is no API available, by literally connecting to the system directly.
"If a person can use the app, the agent can too," said Microsoft.
And since AI is involved, this computer use can adapt to the changes that tend to happen in websites and apps, automatically.
"It adjusts in real time using built-in reasoning to fix issues on its own, so work continues without interruption. It is also built on Copilot Studio’s robust security measures and governance frameworks, to help ensure compliance with organizational and industry standards," explained Microsoft.
"With computer use in Copilot Studio, makers can build agents that automate tasks on user interfaces across both desktop and browser applications, including Edge, Chrome, and Firefox. Additionally, computer use runs on Microsoft-hosted infrastructure, meaning organizations don’t need to manage their own servers. Enterprise data stays within Microsoft Cloud boundaries and is not used to train the Frontier model. This helps your organization accelerate deployment, reduce maintenance, and lower infrastructure costs."
Uses cases include: automated data entry, market research, and invoice processing.
And because computer use agents can adapt to changes and do things automatically, these agents can also transform robotic process automation (RPA).
Due to its automation ability, computer use can overcome traditional limitations, like the fragility of UI elements, and can handle complex dynamic interfaces. This makes automation accessible to people beyond professional RPA developers.
In Copilot Studio, computer use addresses common RPA challenges by making automation smarter and more intuitive:
The goal of this computer use in Copilot Studio is to create an end-to-end agent platform designed to help organizations achieve their AI and operational goals.
"We want to empower you to streamline processes, enhance productivity, and drive innovation," ended Microsoft.