Blip Degradation

Incident Report for Blip

Resolved

All platform features are currently operational
Posted Jun 05, 2020 - 22:08 GMT-03:00

Update

We are continuing to monitor for any further issues.
Posted Jun 05, 2020 - 21:28 GMT-03:00

Monitoring

Our environment is 100% restored after the actions taken by the technical team. We continue monitoring.

Start date / time: 05/06/2020 - 09:00 AM and End date / time: 03:00 PM

Customers can experience:
Error messages on Portal Desk;
Access to Portal Desk failed;
Message exchange fail in WHatsApp;

Actions implemented:
Scaling / redistribution of the machines;
Posted Jun 05, 2020 - 15:47 GMT-03:00

Update

The environment was restored after the implementation of the actions mentioned above.
Posted Jun 05, 2020 - 15:42 GMT-03:00

Update

Problem: From the analysis, problems of network degradation and failures of physical infrastructure elements were identified.

Customers can experience:
Error messages on Portal Desk;
Access to Portal Desk failed;
Message exchange fail in WHatsApp;

Palliative actions: The service team continues to carry out interventions in the environment to keep the platform operational.

Actions in progress:
Working in the machine scaling / redistribution part;
In parallel, another front working on application-oriented analysis;
Posted Jun 05, 2020 - 12:53 GMT-03:00

Update

We are continuing to work on a fix for this issue.
Posted Jun 05, 2020 - 11:34 GMT-03:00

Update

We are taking some palliative actions in an attempt to keep the platform operational, while the technical team is focused on investigations and actions to resolve definitively.
Posted Jun 05, 2020 - 11:26 GMT-03:00

Update

We have identified instability in the portal and Desk. Our team continues to investigate the issue.

Os clientes podem experimentar:

Mensagens de erro no Desk;
Portal, falha no acesso ao Desk
Falha na troca de mensagens no WHatsApp
Posted Jun 05, 2020 - 10:39 GMT-03:00

Update

We have identified instability in the portal and Desk. Our team continues to investigate the issue.
Posted Jun 05, 2020 - 10:35 GMT-03:00

Update

We are continuing to work on a fix for this issue.
Posted Jun 05, 2020 - 10:33 GMT-03:00

Identified

The issue has been identified and a fix is being implemented.
Posted Jun 05, 2020 - 10:32 GMT-03:00

Update

The environment is stabilized, with some corrective actions taken by the technical team.
We have a multidisciplinary team focused on analysis for identification and actions for definitive correction.
Posted Jun 04, 2020 - 15:05 GMT-03:00

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jun 04, 2020 - 12:21 GMT-03:00

Update

We had an instability that caused a momentary slow to access the desk portal. An intervention was performed on one of the servers and the accesses, after a peak of connections, normalized and are currently stable. Our team remains mobilized to investigate the problem.
Posted Jun 04, 2020 - 11:38 GMT-03:00

Update

We have identified instability, some users may experience slowness to access portal and Desk. Our team continues to investigate the issue.
Posted Jun 04, 2020 - 11:13 GMT-03:00

Identified

We have identified instability in the portal and Desk. Our team continues to investigate the issue.
Posted Jun 04, 2020 - 11:04 GMT-03:00

Update

The environment is stabilized, with some corrective actions taken by the technical team.
We have a multidisciplinary team focused on analysis for identification and actions for definitive correction.
Posted Jun 03, 2020 - 16:24 GMT-03:00

Update

The environment is stabilized, with some corrective actions taken by the technical team.
We have a multidisciplinary team focused on analysis for identification and actions for definitive correction.
Posted Jun 03, 2020 - 16:08 GMT-03:00

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jun 03, 2020 - 15:56 GMT-03:00

Update

Some users may have lost their connection. Need to update the page and try to reconnect
Posted Jun 03, 2020 - 14:59 GMT-03:00

Update

We are managing to maintain the operational environment, while the technical team is working on definitive solution actions.
Posted Jun 03, 2020 - 14:43 GMT-03:00

Update

We are continuing to work on a fix for this issue.
Posted Jun 03, 2020 - 13:55 GMT-03:00

Update

Customers may still experience some effects:

Error messages on the Desk;
Portal, Desk access failure
failed message exchange on WHatsApp
Posted Jun 03, 2020 - 13:14 GMT-03:00

Update

We are working on a hotfix to try to resolve the issue.
Posted Jun 03, 2020 - 13:01 GMT-03:00

Identified

We continue to work to resolve the problem as soon as possible.

We are working with palliative interventions to minimize impacts.
Posted Jun 03, 2020 - 12:25 GMT-03:00

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jun 03, 2020 - 11:42 GMT-03:00

Update

We carried out some palliative measures in order to restore the platform acess. The technical team continues to investigate based on the evidence collected so that a definitive solution can be implemented.

Period:

Start date / time: 10:17 AM
End date / time: of palliative solution: 11:15 AM

Impact for customers:

Error messages on the Desk;
Portal, Desk access failure
failed message exchange on WHatsApp
Posted Jun 03, 2020 - 11:39 GMT-03:00

Identified

The issue has been identified and a fix is being implemented.
Posted Jun 03, 2020 - 11:32 GMT-03:00

Investigating

We are currently investigating this issue.
Posted Jun 03, 2020 - 10:17 GMT-03:00
This incident affected: Hosting Business (Bot Builder, Bot Router), Hosting Enterprise (Bot Builder, Bot Router), and Blip Platform (Portal).