Quoting Matthew Green:

In theory those systems could remove your need to ever touch your phone again: they’ll answer your text messages for you, order your food, swipe your dating profile, negotiate with your lenders, and generally anticipate your every want or need. The only ingredient they’ll need to make this future come true is virtually unrestricted access to all your private data, plus a whole gob of computing power to process it.

AI could be the biggest privacy story of the current decade. Not only will we be sending more of our data off-device, but it’ll be examined by increasingly powerful systems. In principle those systems will know everything about us, our friends, our private conversations, maybe even our deepest innermost thoughts. We are about to face many hard questions about these systems, including some difficult questions about whether they will actually be working for us at all.

imagine further that the thing we built was not some highly-limited and specialized model, but rather a powerful AI “agent” that can answer all sorts of questions about you — and even do so in plain language. Given the right API, the government could ask your agent all sorts of sophisticated questions about your behavior and data, questions like: “does this user have any potential CSAM,” or “have they written anything that could potentially be hate speech in their private notes,” or “do you think maybe they’re cheating on their taxes?” And the model would probably be able to give a very good answer.

it doesn’t really matter if your model is running locally, or if it uses trusted cloud hardware — once a sufficiently-powerful general-purpose agent has been deployed on your phone, the only question that remains is who gets to talk to it. And that isn’t really a technical question anymore, it’s a policy question. Will it be only you? Or will we prioritize the government’s interest in monitoring its citizens over various fuddy-duddy notions of individual privacy.