Hi blipper! On 15/09/2021, Wednesday, we faced an unavailability on Blip that affected the operation of many Smart Contacts.
To be transparent with you, Blip user, we are writing to tell you what happened.
What happened
An automatic unscheduled date change for 2022 has been identified on one our Take Blip server, causing an interruption in the exchange of messages from our smart contacts.
How this issue impact you
Because of this failure, our Blip Portal functionalities faced problems to access and also Blip Desk faced problems to open and distribute tickets.
In the name of Take Blip, I want to say sorry for any problems caused to you, your company and your customers.
What we do to solve it
As soon as the problem was identified at 10:50 am - 3UTC, our team acted quickly to prevent our community from being more affected. We took actions to isolate the divergent server and made interventions in other servers, equalizing the date information and thus, normalizing the operation of our platform around 12:30 pm - 3UTC.
Later, around 2:20 pm - 3UTC, as a repercussion of the scenario reported above, we observed an impact on the tickets’ distribution. This happened because the agents' statuses were still dated 2022. Once again, our team acted immediately by intervening in the services responsible for distributing tickets and correcting agents' status actions with the date 2022, reestablishing the service flow normalizing the operation around 15:20h pm -3UTC.
Where we are now
The Blip Portal and Blip Desk is working again, and our technical team is following up action to map the root cause. In addition, we have internal actions to prevent events like that again as:
- Investigation to identify how the date was automatically changed on that server;
- Adjustments in databases that were registered with the year 2022 date information
You can check this history and all other Blip features status on our Status Page.
We also want to say thank you for your patience and remind you that we are always here to help you in any need. Just open a request on our Support or create a new topic on Blip Forum, the exclusive space to the whole users' community.
Sincerely,
During our monitoring, we identified a failure in the distribution of tickets in the Desk.
Impact for customers:
The main impact here is on the operation of the customer service team that uses the Desk tool since tickets are not distributed in queues.
Solution:
Some interventions were carried out in the services responsible for distributing tickets so that the scenario could normalize. We also advise agents to reconnect to the tool so that they can receive tickets again.
We continue to monitor our environment and apologize once again.
If you identify any fault, please let us know.
More information about the root cause will be made available.
Start date / time: 14:20
End date / time: 15:20
Reflection for the customer: Failure to distribute tickets within the Desk tool with impact on service.
Point identified: It was verified that the failure is due to an error that is occurring in the resource that is responsible for part of the ticket distribution in the service queues.
Action in progress: The technical team is still analyzing the problem to apply a fix.