Blip Desk and Builder - [Degraded performance]

Incident Report for Blip

Postmortem

Start Time: 28/11/2024 ~15:05 (GMT-3)
End Time: 28/11/2024 ~15:45 (GMT-3)

Incident Summary: Unavailability of Blip APIs created by services teams.

Impact Analysis:
In short, the APIs developed by Blip's services teams were unavailable, causing a break in the conversational flow in smart contacts that have direct integration with them.

It is important to highlight that the scenario in question only affected the APIs built by Blip's digital cells and that our platform remained intact throughout this period, having no relation to this incident.

Mitigation Measures:
To restore the APIs, we carried out interventions in the infrastructure where they were located and after testing we confirmed the normalization of the services.

Root Cause Analysis:
In the face of the normalization of the environment, we classify the incident as resolved and now enter the root cause analysis stage.

This process will be conducted internally, as quickly as the situation requires, and will be published here on the status page as soon as possible.

If you have any questions or need additional clarification, we recommend that you contact the CSM of your account for support.

Posted Nov 28, 2024 - 16:32 GMT-03:00

Resolved

This incident has been resolved.
Posted Nov 28, 2024 - 16:31 GMT-03:00

Monitoring

After identifying that the infrastructure in which CSPS API requests (Internal APIs) travel was without flow, intervention was carried out in the environment (replicas were scaled to 20).

After testing, we identified that the APIs responded again.
Posted Nov 28, 2024 - 15:45 GMT-03:00

Identified

The issue has been identified and a fix is being implemented.
Posted Nov 28, 2024 - 15:30 GMT-03:00

Investigating

We are currently experiencing low performance in Blip Desk and Builder.

Impact:
Slowness when using Blip Desk and processing commands within Builder.

Update:
Our support team is actively working on it.
Posted Nov 28, 2024 - 15:15 GMT-03:00
This incident affected: Hosting Business (Bot Builder), Desk, and Hosting Enterprise (Bot Builder).