Degraded performance on the BLiP
Incident Report for Take Blip
Postmortem

On 09/08/2021, Wednesday, we faced an unavailability on Blip that affected the operation of many Smart Contacts.

To be transparent with you, Blip user, we are writing to tell you what happened.

What happened

A failure in connection with our cache service was identified, which is used to store contact information, causing an interruption in the exchange of messages from our smart contacts.

How this issue impact you

Because of this failure, the Blip CRM, our customer base management functionality, faced problems in executing message exchanges.

In the name of Take Blip, I want to say sorry for any problems caused to you, your company and your customers.

What we do to solve it

As soon as we identified the problem at 8pm -3UTC, our team put together and acted quickly to start the treatments. The correction was immediately applied and the service was normalized at 10:36pm -3UTC.

Where we are now

The Blip CRM is working again, and our technical team is following up with the cloud provider to identify the main cause of the failure.  In addition, we have internal actions to prevent events like that again.

You can check this history and all other Blip features status on our Status Page.

We also want to say thank you for your patience and remind you that we are always here to help you in any need. Just open a request on our Support or create a new topic on Blip Forum, the exclusive space to the whole users' community.

Sincerely,

Posted Sep 21, 2021 - 16:02 GMT-03:00

Resolved
Fault identified:

Through our monitoring, alarms were identified on the platform's servers. After analysis, we identified that there was a loss of connection to the services of some servers, generating the scenario of slowness.

Impact:

Customers noticed slowness in the exchange of messages from smart contacts and in the use of the platform.

Solution:

To normalize the environment and solve the failures, our team acted and carried out the necessary interventions and after acting all services were normalized.

More information will be made available via postmortem.

Start date / time: 5:30 pm
End date / time: 08:08 pm
Posted Sep 09, 2021 - 20:34 GMT-03:00
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 09, 2021 - 20:25 GMT-03:00
Update
During analysis our platform team identified loss of connection on some servers affecting platform performance. We are still working to solve the problem permanently.
Posted Sep 09, 2021 - 19:41 GMT-03:00
Update
Impact:

Some smart contatcs may experience slow message exchanges.
Posted Sep 09, 2021 - 18:38 GMT-03:00
Investigating
We are suffering a degradation in the performance of the BLiP platform, our technical team is already working on the case.
Posted Sep 09, 2021 - 18:10 GMT-03:00
This incident affected: Desk, Channels (WhatsApp, Telegram, Messenger, BlipChat, Workplace Chat, BusinessChat, Skype, E-mail), Hosting Enterprise (Bot Builder, Bot Router), Hosting Business (Bot Builder, Bot Router), and Take Blip Platform (Portal).