Platform Instability
Incident Report for Blip
Postmortem

Dear Custumers

We are sorry for the failure that occurred in our platform, we would like to inform you that at 10:30 am, our monitoring tools identified an increase in the requisition time in the applications responsible for the traffic of messages in the smart contact.

How it affected you

As a consequence of the increase in time in the processing of these requests, many users faced slowness in the exchange of messages via Blip.

We apologize for any inconvenience this slowness has caused you, your company and your customers.

What have we done
As soon as the problem was identified, our Product team acted quickly to prevent you from being further affected. We reviewed recent updates to the API, limited items per command returned by the concurrent task scheduler, and increased Builder cache time.

What is the current scenario

Blip message traffic is already stable again and we are working on investigating the root cause to ensure that this problem does not happen again.

In addition, we thank you for your understanding and reinforce that we are available to help you in whatever you need! Just open a ticket or make a post in the Forum, where we communicate with the entire user community.

Posted Apr 16, 2021 - 21:48 GMT-03:00

Resolved
This incident has been resolved.
Posted Apr 06, 2021 - 21:00 GMT-03:00
Monitoring
Slow message exchange
Lentidão e timeout em comandos (blip e desk)
Problemas de publicação de fluxo

Fault identified:

An increase in the time of requests was identified within responsible for the traffic of messages in the intelligent contact (Bot).

Reflection for the client:

Customers will observe slowness in the exchange of messages, with registration in some periods of response by the Bot.

Correction applied:

We carried out some actions related to the database improvements.
At 3:19 pm we found that the database performance normalized.

Root Cause:

We continue to analyze together several times to identify a root cause.

Start date 05/04/2021 time: 10:39
Posted Apr 06, 2021 - 16:01 GMT-03:00
Update
Update:

We have performed some actions, with no improvement in the database overload scenario.

We are currently working on a hotfix and will update it when it is finished.

Actions: The technical team is already working and we will update in 30 minutes
Posted Apr 06, 2021 - 14:38 GMT-03:00
Identified
Slow message exchange
Slowness and timeout in commands (blip and desk)
Stream publishing issues

Fault identified:

An increase in the time of requests was identified within responsible for the traffic of messages in the intelligent contact (Bot).

Reflection for the client:

Customers will observe slowness in the exchange of messages, with registration in some periods of response by the Bot.

Actions: The technical team is already working and we will update in 30 minutes

Start date 05/04/2021 / time: 10:39
Posted Apr 06, 2021 - 10:39 GMT-03:00
This incident affected: Blip Platform (Core).