BLiP Desk and Portal [Degradation]
Incident
Report for Blip
Resolved
Context: On 03/07/2023 after 10 pm we received some reports related to access to the portal and Blip Desk. At this time we started the investigation of the case, and after the team start to act we saw that this scenario was solved. In the morning of today, 03/08/2023, around 07:50 am, we identified an increase in the proportion of failures, as soon as the scenario was identified, the entire Engineering and Infrastructure team was mobilized to act in the case.
What was the impact for our customers? Portal and Blip Desk access failure
NOTE: In the morning, part of our customer base was affected, however, even during the actions in the early afternoon, the impact was extended to 100% of the customer base. Noting that the impact occurred intermittently throughout the day.
What caused the perceived scenario? It was identified that our user authentication service began to have a response time to requests, higher than the normal observed historically.
What were the contouring actions that restored the environment?
- An increase in the infrastructure of the user authentication application was carried out to try to improve the user experience;
- We returned the production environment to the version prior to the period that started the problem;
- We carried out the migration of the user authentication application database from one cloud provider to another;
- Workaround actions were taken in the applications to reduce the volume of requests in the authentication application.
Has the root cause been identified? Despite the mobilization of the technical team throughout the day, we did not arrive at a cause that caused the overload of the functionalities mentioned in the item above.
Do we have any action mapped for definitive correction?
We will have a Post Mortem meeting tomorrow, 09/03/2023, to continue the investigation of the root cause and also to survey the corrective actions.
What was the impact for our customers? Portal and Blip Desk access failure
NOTE: In the morning, part of our customer base was affected, however, even during the actions in the early afternoon, the impact was extended to 100% of the customer base. Noting that the impact occurred intermittently throughout the day.
What caused the perceived scenario? It was identified that our user authentication service began to have a response time to requests, higher than the normal observed historically.
What were the contouring actions that restored the environment?
- An increase in the infrastructure of the user authentication application was carried out to try to improve the user experience;
- We returned the production environment to the version prior to the period that started the problem;
- We carried out the migration of the user authentication application database from one cloud provider to another;
- Workaround actions were taken in the applications to reduce the volume of requests in the authentication application.
Has the root cause been identified? Despite the mobilization of the technical team throughout the day, we did not arrive at a cause that caused the overload of the functionalities mentioned in the item above.
Do we have any action mapped for definitive correction?
We will have a Post Mortem meeting tomorrow, 09/03/2023, to continue the investigation of the root cause and also to survey the corrective actions.
Monitoring
Impact:
Errors when using the Desk and Portal.
Update:
Our metrics and logs indicate that now we don't have problems at this moment. We continue to monitor the environment
Errors when using the Desk and Portal.
Update:
Our metrics and logs indicate that now we don't have problems at this moment. We continue to monitor the environment
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team continues working to see all platform impacts and restore the environment
Impact:
Errors when using the Desk and Portal.
Update:
Our team continues working to see all platform impacts and restore the environment
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team continues working to see all platform impacts and restore the environment
Impact:
Errors when using the Desk and Portal.
Update:
Our team continues working to see all platform impacts and restore the environment
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
We made a hotfix to improve application performance. Our team continues working to see all platform impacts and restore the environment
Impact:
Errors when using the Desk and Portal.
Update:
We made a hotfix to improve application performance. Our team continues working to see all platform impacts and restore the environment
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment.
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment.
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment. In our tests, the scenario is better than before
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment. In our tests, the scenario is better than before
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment.
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment.
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment. This error occurs just for clients that are trying to make login into platform. Everyone that was logged in before the start of the problem can use the platform.
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment. This error occurs just for clients that are trying to make login into platform. Everyone that was logged in before the start of the problem can use the platform.
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the enviroment
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the enviroment
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
We finish some actions to restore our environment but we continue working on it
Impact:
Errors when using the Desk and Portal.
Update:
We finish some actions to restore our environment but we continue working on it
Update
We are experiencing a degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team continue working to restore the environment
Impact:
Errors when using the Desk and Portal.
Update:
Our team continue working to restore the environment
Update
We are experiencing a partial degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team continue working to restore the environment
Impact:
Errors when using the Desk and Portal.
Update:
Our team continue working to restore the environment
Update
We are experiencing a partial degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
We started some actions to restore the environment
Impact:
Errors when using the Desk and Portal.
Update:
We started some actions to restore the environment
Update
We are experiencing a partial degradation on BLiP Desk and Portal application.
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment
Impact:
Errors when using the Desk and Portal.
Update:
Our team is working to restore the environment
Investigating
We are experiencing an outage in the BLiP Desk and Portal application.
Our technical team is already working on the case.
Impact: Desk and Portal is inaccessible by attendants
Our technical team is already working on the case.
Impact: Desk and Portal is inaccessible by attendants
This incident affected: Desk and Blip Platform (Portal).