Technology is an aid. But sometimes to do this "aiding," it needs to be kind of invasive.
When Google introduced photo scanning technology to Android devices, it quickly sparked a significant backlash. Users accused the company of secretly installing new surveillance technology on Android phones without their consent. But this time, Google is able to reassure people.
The company stated that the technology behind it, called the 'SafetyCore,' is essentially a framework that would not scan photos or other content without users' permission.
Instead, Google explained that "SafetyCore provides on-device infrastructure for securely and privately performing classification to help users detect unwanted content. Users control SafetyCore, and SafetyCore only classifies specific content when an app requests it through an optionally enabled feature."
And this time, Google is releasing the feature to Google Messages.
According to reports, the Android app now has Sensitive Content Warnings that can automatically blur explicit images on Android.
Not only does it blur the content, but it also warns that such imagery can be harmful and provides options to either view the explicit content or block the sender.
Behind this technology, is AI.
Its image recognition ability is able to scan photos locally on users' devices, with Google assuring that no data is sent back to their servers.
Initially, the feature is disabled by default for adults but is automatically enabled for children.
Adults can opt in by going to Google Messages Settings under Protection & Safety—Manage sensitive content warnings. For children, settings can only be modified through either their account settings or Family Link, depending on their age.
If users prefer not to have this monitoring feature, they can uninstall SafetyCore.
But doing this may make Android to reinstall with a future Play Services update, meaning that the operating system may be able to reenable the feature, even when it's explicitly turned off.
The introduction of the feature is as expected, and it also just the beginning.
As Google operates a long list of consumer-grade services, like Gmail and others, it's expected that the security feature can be released to other products as well.
This is why users should decide how much AI scanning, monitoring, and analysis they’re comfortable with—and where they draw the line.
It's worth noting that despite this AI scanning is happening on-device, other updates don’t offer the same privacy protection, meaning that Google is having an advantage over third-party solutions.
Then comes the issue or privacy and secrecy.
While Google's automatic photo scanning technology scans photos for sensitive content like nudity within Google Messages, it still creates backlash.
Privacy advocates worry about surveillance due to the lack of transparency, as Google hasn't made the underlying machine learning models open source, limiting external oversight of the technology.
Then, there is the fact that users aren't really informed about the technology, which would lead to the fact that they may not fully understand how their data is handled. There are also security risks if data is transmitted to external servers, and also risk if there are vulnerabilities in the system. There are also concerns about the expansion of AI monitoring into other personal content, not just photos, which could erode user privacy over time.
SafetyCore is part of a larger trend of AI and machine learning technologies being integrated into consumer devices to help with content moderation and to protect users from harmful material. However, it highlights a broader debate around privacy and security in the age of AI, where users must trust companies like Google to manage sensitive data responsibly.
While useful for safety, AI scanning challenges the balance between privacy and convenience.
The introduction of this feature in Google Messages was expected. The real question now is what comes next. The risk is that this capability coincides with increasing pressure from legislators and security agencies around the world on secure, encrypted user content. Every time such technology is introduced, privacy advocates voice concerns.