Salesforce is suffering one of its largest outages as the company was forced to shut down many of its infrastructure.
The cause was a faulty script that happened to be introduced to its systems, when the company made some changes to its production environment. The script essentially broke access permission on some organizations, giving their employees the access to their employers' files.
"A database script deployment that inadvertently gave users broader data access than intended," explained Salesforce on its status update.
Making things worse, the permission access also granted write permissions, making it easy for malicious employees to steal or tamper on company's data.
Salesforce said the script only impacted customers of Salesforce Pardot, which is its business-to-business (B2B) marketing-focused CRM.
But since the issue was indeed problematic and can cause huge cost to customers, Salesforce is not taking it lightly. The company decided to take down all other Salesforce services, for both current and former Pardot customers.
"As a result, customers who were not affected may have also experienced service disruption, including customers using Marketing Cloud integrations," Salesforce said.
Salesforce customers in Europe and North America were the most impacted by the company shutting down access to its own service.
The issue caused quite a stir on social media, where the company has been mocked and ridiculed.
Parker Harris, Salesforce CTO and co-founder, apologized for his company's issue on Twitter:
We have had to disable access to our service to customers affected in order to help resolve the issue. We expect to be able to restore access soon as we continue to work through this issue.
— Parker Harris (@parkerharris) May 17, 2019
Salesforce said that it was slowly unblocking access for companies that were not impacted by the database script directly. It took the company at least 15 hours before finally and gradually restoring all faulty permissions.
"In parallel, we are working to restore the original permissions as quickly as possible for customers that were affected by the permissions change," Salesforce said.
Before the fix can be rolled out to all users, Salesforce suggested that organizations with a valid backup of their profiles and user permission data were still able to deploy that information directly from a Sandbox copy to the production environment on Saturday.
But if they lack a Sandbox containing production profiles and permission sets, the company said that administrators would have to manually modify the configurations to grant appropriate access to users.
Salesforce is one of the players that thrive by providing services to those who want to move their critical applications to the cloud. Cutting down costs, Salesforce allows its customers' employees to access their company's data from anywhere.
But this faulty script is an example of how cloud apps can also leave businesses crippled and open to loss in productivity when a system does go down.
Over the weekend, Salesforce held a series of conference calls to update its customers on the status of repairs on the 105 affected instances.