In the high-stakes world of energy, frontline workers need more than just hard hats and coffee to keep things running smoothly. Enter AI and mixed reality — the dynamic duo set to turbocharge their efficiency. By combining smart technology with immersive, real-time support, these tools provide workers with everything they need, right in their line of sight, whether it’s data, guidance, or a virtual helping hand.
Throughout this article, we’ll explore some practical ways that these technologies are coming together to provide frontline workers with a real boost during their day-to-day activities.
Frontline Workers Need a Friend Copilot
Despite significant advances in mobile communications technology, it’s easy for frontline workers to feel disconnected. As crews travel from job to job during the course of a shift, many things can change in both the back office and the field, forcing crew members to adjust on the fly. In these situations, having access to an AI-powered chatbot (copilot) that can quickly provide answers to routine questions can be a real game-changer.
What About Field Services Software?
To put the copilot need into perspective, consider the way that crews normally deal with changes. Most of the time, they’re forced to run back out to the truck to grab their rugged laptops. Then, they tunnel into the company network and log onto various systems to come up to speed with everything that’s going on. On average, these kinds of disruptions can lead to 15–30 minutes of loss time.
While modern field service systems are making it easier for frontline workers to gain access to this information, it’s still more of a pull than a push. This is to say that workers normally have to log onto one or more systems to find answers — which is less than optimal when you’re on the go and not parked in front of a computer screen.
The Copilot User Experience
Now, let’s compare this user experience with the experience you get with modern copilots. As you can see in Figure 1, these mobile-friendly agents are always connected and have a very simple user interface. Unlike the kinds of “dumb” chatbots you may have interacted with in the past, these copilots are capable of answering complex questions about the goings-on within your organization.
For example, here are just a few of the types of questions a copilot could answer for a crew member:
- What's the status of a given work order?
- What's going on with this customer's account?
- When was the last time we worked on this installation?
- Has a permit been granted yet? And if not, who should I contact?
- Are there any nearby crews that have spare parts that are needed to complete an order?
Besides answering work-related questions in context, copilots can also be used to perform routine actions. For instance, in Figure 1, you can see where the user is notifying the copilot that they have completed all of the tasks for a work order. Based on this information, the copilot can then carry out the remaining steps to close out the work order (e.g., updating order statuses, creating labor confirmations, or filling in time sheet entries).
Of course, this is just the tip of the iceberg. The possibilities for automation of common work tasks within copilots are endless.
Understanding How Copilots Work
To put all this into perspective, it’s important to understand how copilots work — starting with recent advances in the field of Generative AI. If you’ve played around with ChatGPT, then you know it’s a tool that you can use to ask all kinds of questions. It’s able to answer those questions because the AI was trained on a wide range of data sources across the internet.
Copilots work in a similar fashion, but they’re more narrowly trained for a specific purpose. For example, the copilot shown in Figure 1 is an example of a copilot that’s tailored to work with Microsoft’s Dynamics 365 Field Service software package. As such, it knows the ins-and-outs of field service operations, but probably can’t help you with your son or daughter’s math homework.
Once a copilot has been trained up in a particular knowledge domain, it’s able to carry on intelligent conversations about topics within that domain. From here, we can enrich the user experience further by defining additional topics and actions that the copilot can perform.
For example, let’s say one of the topics is a “how-to” topic. If a user asks the copilot how to repair a grounding anchor, the copilot can pull from the various knowledge base articles and guides it was trained on to answer the question. However, let’s say that the answer is not to the user’s satisfaction. In this case, much like an automated phone system, we can direct the user to another “branch” within the copilot that can perform helpful actions like directing the user to an interactive learning system or phoning an expert.
These actions can be as simple or complex as you like, connecting to just about any system imaginable. Some common examples of actions we can incorporate into copilot experiences include the following:
- Looking up data and/or processing transactions in SAP or other relevant business systems — even ones sitting behind the corporate firewall.
- Launching audio or video conferencing sessions in apps like Microsoft Teams.
- Dynamically routing team members to subject matter experts within the organization.
While it’s unlikely that we can manage to automate everything for frontline workers, we can definitely create a virtual assistant that can manage most of the repetitive tasks that tie up users in the field.
Two-Way Communication
Another powerful feature of copilots is that they can facilitate bidirectional communication with key business systems. This is to say that copilots can be there running in the background keeping an eye out for potential changes. For example, if an order is put on hold or a crew gets rescheduled, the copilot can detect that event and raise an alert to the user(s) impacted by it.
These alerts can come in the form of email messages, text messages, mobile push notifications, and so forth. Alternatively, we can utilize Adaptive Cards technology to send more advanced messages in tools like Microsoft Teams. In this case, the notification comes with pre-defined actions that allow the user to quickly respond to the solution and get back to work.
Continuous Learning: Building a Knowledge Base
While having access to an intelligent virtual assistant is powerful enough on its own, it gets even better when you consider that these assistants can become increasingly more intelligent over time.
Besides the obvious operation guides and training manuals, copilots can also be trained on data from key business systems. This includes backend ERP and CRM systems, field service systems, and so forth. You might also be surprised to learn that we can also glean useful information from all the unstructured data that’s generated in the field.
Figure 4 below illustrates this concept, showing how we can utilize modern AI technology to glean useful information from photos and document attachments, transcripts from work order logs, and even handwritten notes/forms produced in the field. Collectively, this data can be continually fed into an intelligent search index that the copilot uses to answer questions.
Enhancing the Copilot Experience with Mixed Reality
As AI-powered copilots become a bigger part of our daily workflows, the potential to enrich these experiences with mixed reality is really exciting. With tools like Microsoft Dynamics 365 Guides and Dynamics 365 Remote Assist, you can take copilot interactions to the next level by immersing users in 3D environments and offering real-time support from virtually anywhere. This blend of AI and mixed reality isn’t just about making tasks easier — it’s about completely transforming how we learn, collaborate, and solve problems in the moment.
Step-by-Step Help with Dynamics 365 Guides
With Dynamics 365 Guides, you can build interactive training guides or installation/repair guides using the simple graphical interface shown in Figure 5 below. Here, you can define step-by-step instructions with logical branching and mixed media content including images, videos, and even interactive app experiences.
Once these guides are in place, frontline workers can access them using mixed reality headsets (Figure 6) or a smartphone, tablet, or desktop device (Figure 7).
Phone a Friend with Dynamics 365 Remote Assist
With Dynamics 365 Remote Assist, you can empower frontline workers to contact an expert that can joystick them through an installation or repair process. As you can see in Figure 8 below, these calls can be run through Microsoft Teams on most any device and the expert can draw/highlight on the screen to guide workers through the entire process.
Closing Thoughts
As AI and mixed reality continue to advance, the potential to revolutionize frontline work in the energy sector is huge. By integrating these technologies, companies can empower their teams to work smarter, faster, and more accurately — whether it’s accessing real-time data, following guided workflows, or receiving remote support. Embracing these innovations isn’t just about keeping up; it’s about leading the charge into the future of energy.