Desk Degradation
Incident
Report for Blip
Resolved
After the actions taken, the scenario was normalized and the postmortem will be filled with more information about the root cause.
Monitoring
Identified scenario: High CPU consumption was identified in our transactional database responsible for processing the history of messages displayed on the Desk tool, in two of the nodes out of a total of 10 available / shared nodes for all clusters.
Impact for the client: Slowness in the display of messages inside the Desk and also impacts on some analytical reports.
Actions in progress:
1) Rolled back of a scheduled update made yesterday (02/23/2021) in the evening, in order to isolate the scenario, since the update in question could reflect on the identified scenario;
2) After the Rollback scenario was isolated because it did not have the expected effect, it was concluded that in fact it was not related, the drain (restart of the service) was performed on one of the nodes of the database server, as there was a process of Repair this machine. Then, we restart applications that make use of the transactional database, but only applications that readed in the database are back to normal. For control over the written part, the technical team reduced the number of consumers of writing in the transactional database for objective to decrease access to the database.
After the second action taken, we have already seen a decrease in failures and consequently also an improvement in the processing of the history of messages displayed on the Desk service. The technical team continues to act on the case.
NOTE: It is worth mentioning that the functionality of counting active / sent / received messages within the Analytics screen is not occurring in real time.
Impact for the client: Slowness in the display of messages inside the Desk and also impacts on some analytical reports.
Actions in progress:
1) Rolled back of a scheduled update made yesterday (02/23/2021) in the evening, in order to isolate the scenario, since the update in question could reflect on the identified scenario;
2) After the Rollback scenario was isolated because it did not have the expected effect, it was concluded that in fact it was not related, the drain (restart of the service) was performed on one of the nodes of the database server, as there was a process of Repair this machine. Then, we restart applications that make use of the transactional database, but only applications that readed in the database are back to normal. For control over the written part, the technical team reduced the number of consumers of writing in the transactional database for objective to decrease access to the database.
After the second action taken, we have already seen a decrease in failures and consequently also an improvement in the processing of the history of messages displayed on the Desk service. The technical team continues to act on the case.
NOTE: It is worth mentioning that the functionality of counting active / sent / received messages within the Analytics screen is not occurring in real time.
Identified
The issue has been identified and we are working to get it fixed as soon as possible.
We'll update the status as soon as we have news.
We'll update the status as soon as we have news.
Update
We are continuing to investigate this issue. We'll update the status as soon as we have news.
Investigating
We identified slowness in the message search commands for display in the Desk tool, with failure or delay in the display of messages to the attendant.
The team is working to normalize the environment
The team is working to normalize the environment
This incident affected: Desk.