The promise is AI that’s uniquely helpful because it knows you. But the risk is AI that feels more like surveillance than service.

In a recent episode of the Limitless podcast, Robby Stein, VP of Product for Google Search, explained that Google’s AI tends to field more queries that are advice-seeking or those where the user is looking for recommendations — and these types of questions are more likely to benefit from more subjective responses.

“We think there’s a huge opportunity for our AI to know you better and then be uniquely helpful because of that knowledge,” Stein said in the interview. “And one of the things we talked about at [Google’s developer conference] I/O was how the AI can get a better understanding of you through connected services like Gmail.”

Google has been integrating AI into its apps for some time, starting back when Gemini was still known as Bard. More recently, it began pulling personal data into another AI product, Gemini Deep Research. And Gemini is now infused into Google Workspace apps like Gmail, Calendar and Drive.

But as Google integrates more personal data into its AI — spanning your emails, documents, photos, location history, and browsing behavior — the line between a helpful assistant and an intrusive one becomes increasingly blurred. And unlike opt-in services, avoiding Google’s data collection may become harder as AI becomes central to its products.

Google’s pitch is that this deep personalization makes the AI far more useful. The idea is that Google’s AI technology could learn from the user’s interactions across Google’s various services, then use that understanding to make more personalized recommendations. For instance, if it learned that a user likes particular products or brands, the AI responses might favor those in its recommendations.

That, Stein said, would be “much more useful” than just showing users a more generic list of the best-selling products in a given category. “That is, I think, very much the vision — of building something that can be really knowledgeable for you, specifically.”

This idea isn’t all that different from how the “Others” in the hit Apple TV show “Pluribus” have gobbled up the world’s knowledge, including intimate details about individuals. When the system interacts with the show’s protagonist, Carol, it uses that data to personalize everything: cooking her favorite meals, adopting a familiar face to handle its communications with her, and otherwise anticipating her needs.

But Carol doesn’t find the personalized responses kind; she finds them invasive. She never consented to sharing her data with the hivemind, yet it knows her better than she’d like.

Similarly, it seems that avoiding Google’s data-gobbling ways will get increasingly difficult in the AI era, and if Google doesn’t get the balance right, the results could feel more creepy than useful.

(To be clear: Google does let you control the apps Gemini uses to make its AI more knowledgeable about you specifically — it’s under “Connected Apps” in Gemini’s settings.)

If you do share app data with Gemini, Google says it will save and use that data according to the Gemini privacy policy. And that policy reminds users that human reviewers may read some of their data and not to “enter confidential information that you wouldn’t want a reviewer to see or Google to use to improve its services.”

But as more data gets ingested into Google’s own hivemind, it’s easy to see how AI could make data privacy more of a gray area.

Google, however, believes it has a solution of sorts.

Stein says that Google will indicate when its AI responses are personalized.

“I think people want to intuitively understand when they’re being personalized — when information is made for them, versus when [it’s] something that everyone would see if they were to ask this question,” he said.

Stein noted, too, that Google could send a push notification to users when a product they had been considering after several days of online research becomes available or is on sale.

“There are all these ways that Google now, across modes, across kind of different aspects of your life, [is] being incredibly helpful to you…” he said. “And I think that’s more of how I think of the future of search than any one specific feature or single form factor.”