Start time: September 30, 2024 10:06 AM
End time: October 15, 2024 5:38 PM
Incident summary:
Since September 30th, we have identified a problem that occurred on the Blip platform, which impacted the execution of actions and the processing of commands in the building blocks. As a result, this led to potential impacts on bots' messaging and publishing of new streams, happening intermittently at specific times throughout the day. After the actions taken by our team, we no longer had the problem.
Impact analysis:
Bot stopping responding/performing actions in the builder, meaning users are unable to communicate effectively with the bot.
What caused the instabilities?
An internal Microsoft update resulted in a change in the security protocol used in the database. During a high volume of operations, a conflict occurred due to the activation of a feature that was not aligned with the new communication standard, resulting in query execution failure.
Actions to be resolved:
Palliative Actions: We increase the resilience of the environment and the application, in addition to optimizing the connection between them.
The definitive correction was carried out in phases, with the migration of the application structure to a new database, which resolved the identified problems and reestablished the environment. We have not recorded any more failures since the last one reported on October 15th at 5:38 pm.