A day in the life of a support engineer
Hi, my name is Linus Raes, and I am a support engineer at Orange Cyberdefense Belgium. As a support engineer, I offer assistance to our customers via our ‘Customer Portal’. Our Customer Portal (https://portal.eu.admin.orangecyberdefense.com/) is open for all customers with an active support contract. This is where customers can send in requests for support on existing installations. After filling out the necessary details about the product they wish to have assistance on, the ticket is routed to an engineer assigned to the Support Queue.
The knowledge matrix
Our day starts at 08:30. First, we check our assigned queue for open tickets and see if any urgent calls are open. The tickets assigned to our queue are based on a knowledge matrix that guarantees that every engineer is comfortable and the customers immediately get experts on their cases.
At 09:00, we have our daily call with the SOC team. The call is moderated by the support desk team leader, and in about 15 minutes, we make sure all tickets get the required support level. Some tickets get shuffled around to other engineers with a better knowledge overlap; other tickets are discussed to brainstorm the best way forward. An escalated ticket for a customer gets routed to our Escalation Manager, who will contact the customer to provide an update. At the same time, the technical engineer keeps his focus on solving the issue at hand.
We continue our day by replying to tickets, providing possible solutions, asking for more information, and updating vendor cases. We regularly have a WebEx with a customer in which we retrieve some logs and collect data for a strange issue related to inconsistent threat protection. Afterward, we replicate the issue in our lab environment and try out different solutions without impacting the customer’s production.
(A quick thanks to our internal team for the ultimate playground, which is our lab. At Orange Cyberdefense, all engineers have a dedicated virtual lab with their own IP space, enough memory and CPU for multiple virtual machines, access to all sorts of hardware, trial licenses from our vendors, and more.)
It’s my turn
At 12:30, we take on the ‘lunch calls’. Every day, another engineer stays available during normal lunch hours to ensure we offer continuous support to our customers in need, and today is my turn. We continue with our ongoing investigations until 13:30. Then, we take a well-deserved break in our brand new office lunchroom. The view on the Albert Canal is so soothing!
The whole site is down!
Then, it turns 14:30. The afternoon starts calmly, but at 15:00, we suddenly get a P1 call. An important customer has a major routing failure in his main data center. The whole site is down, and the pressure is on. We immediately make contact with our Escalation Manager. He helps manage and coordinate the situation, contacts the customer and the account manager to make sure everybody has the latest updates. Engineers who performed part of the initial installation are contacted to help out and investigate the root cause.
After a short call to retrieve as much information as possible, we take a deep dive into the components at heart. Initial debugging is done, and specialized tools are used to parse the logs. To make sure we don’t lose additional time, a vendor case is created with a P1 status to make sure we keep all channels open for a fast recovery.
We identify the root cause as a physical failure of an internal component and look to replace it as soon as possible. We consult our internal spare parts list and find a suitable replacement. An engineer goes to our warehouse to locate the device while we search for the matching firmware and the latest configuration file using our automated backup system available for all NOC/SOC-managed customers.
We prepare the device at our warehouse and jump in our car with the spare device. The escalation manager contacts the customer to make sure we have access to the data center where the device is located. Badges and access codes are arranged. On-site, we swap the device, and after some additional configuration, the site is back up and running. Great news!
In the meantime, the vendor case has been switched to an RMA case. A next-day delivery ensures our stock is replenished, while an administrative case makes sure all licenses are correctly transferred. All my active tickets were rerouted to other engineers while I focused on the P1 at hand. The intervention is finalized, and a P1 report is created, which includes a general breakdown of the incident and more detailed timestamps of all steps taken to resolve the problem.
At 17:30, all is well, and we can head home. Back tomorrow for new adventures.