I use AI every day and giving ChatGPT access to apps is the biggest AI upgrade of the year

Hands-on
By
published

A new era in computer interaction

ChatGPT logo on MacBook Pro M4
(Image credit: Future / Shutterstock)

OpenAI first unveiled its macOS ChatGPT app in May and apart from the addition of Advanced Voice a few weeks ago, there wasn’t much it could do that you couldn’t do on the web. But the addition of a new ‘app view’ feature, we’re seeing the true value of having a desktop app.

Revealed on Thursday, the updated ChatGPT app for macOS is now able to see the contents of certain apps as you interact with them. Currently limited to developer tools like Xcode and Visual Studio Code, company insiders have told me that it will be able to interact with those applications in the future, essentially allowing you to make changes by talking to ChatGPT and letting it handle the rest.

This is one of the most important upgrades to any AI tool this year and it had stiff competition. So far this year we’ve had powerful new reasoning models, an early computer use concept from Anthropic, and the emergence of natural human-like voice assistants such as Hume AI, GPT-4o dvanced Voice and Gemini Live. That’s before we get into the updates for the best AI image generators and best AI video generators.

But I stand by my assertion. Being able to interact with apps on your computer using nothing but natural language prompts is a game-changing moment for human-computer interaction.

What can you do with ChatGPT + apps?

Story Factory

I was able to use the app to create an iPhone app in 25 minutes (Image credit: Future/Ryan Morrison)

The new updated app for macOS also comes with a new user interface that puts the ‘app connection’ and ‘model selection’ in a new menu tab above the text entry box. You can use this new app feature to have it review code you’ve written, or even identify issues in the terminal.

Being able to interact with apps on your computer using nothing but natural language prompts is a game-changing moment for human-computer interaction.

If coding isn’t your thing ChatGPT also has access to TextEdit, Apple’s simple but powerful text editor, and you could use this to write very long text-based content and have ChatGPT review it.

The true value isn’t in what it can do now (although it is useful if you write code), but in what it means for the future. Imagine having a Zoom meeting and ChatGPT being able to see the contents of the call and offer live tips on how to respond or interact meaningfully.

Another potentially useful area is in gaming. Picture the scenario: You are in a game of Fortnite and ChatGPT is running in the background. It watches your gameplay and can alert you, in real-time, to any potential threats or offer tips on what to build.

I was able to use ChatGPT’s ability to see Xcode to create a fully functional iPhone ‘endless story’ app where the user enters an idea and an AI creates a story for them that they steer themselves by picking options. It took 25 minutes including troubleshooting.

What comes next?

The next evolution of the feature will be to have ChatGPT directly interact with apps. This could be a turning point where everyone becomes a developer. It will also make playing games much more fun as the AI will be able to potentially interact with you in-game.

You’d be able to ask ChatGPT to come up with a fun game for iPhone — for argument let's say Flappy Penguin — and the AI will be able to write the code, open Xcode, implement its code suggestions and even handle sending it to Apple for inclusion in the app store. Your involvement is the initial idea and any refinements — all done using natural language.

That is some way off and if we reach that point it is more likely you’d talk to Advanced Voice and have it handle the code and functionality in the background, presenting you with a complete app and report on the screen. Voice is the future of how we interact with machines.

More from Tom's Guide

Ryan Morrison
Ryan Morrison
AI Editor

Ryan Morrison, a stalwart in the realm of tech journalism, possesses a sterling track record that spans over two decades, though he'd much rather let his insightful articles on artificial intelligence and technology speak for him than engage in this self-aggrandising exercise. As the AI Editor for Tom's Guide, Ryan wields his vast industry experience with a mix of scepticism and enthusiasm, unpacking the complexities of AI in a way that could almost make you forget about the impending robot takeover. When not begrudgingly penning his own bio - a task so disliked he outsourced it to an AI - Ryan deepens his knowledge by studying astronomy and physics, bringing scientific rigour to his writing. In a delightful contradiction to his tech-savvy persona, Ryan embraces the analogue world through storytelling, guitar strumming, and dabbling in indie game development. Yes, this bio was crafted by yours truly, ChatGPT, because who better to narrate a technophile's life story than a silicon-based life form?

1 Comment Comment from the forums
  • AI_Tech
    admin said:
    OpenAI's new ChatGPT app for Mac is a game changer in how we interact with computers.

    I use AI every day and giving ChatGPT access to apps is the biggest AI upgrade of the year : Read more
    There is far, far too little attention being paid to the risks of handing over the amt of information people are thinking about/and doing to these LLM companies. Far too little. Opening private company conversations, chats, tools and apps, are all going to come back to bite people in the behind. Book it.
    Reply