.jpg)
Microsoft Copilot Studio is the promised land of customized AI experiences built for real user needs. As Ryan Cunningham stressed in his keynote at the Power Conference 2023 in Prague on Friday, there is a trade-off between modernizing every process in the age of AI and waiting to significantly reduce risk (and aversion) for stakeholders.
With that begs the question, can we harness the power of public beta testing to generate value for users sooner? Even better, how do we reimagine IT functions as a cultural shift towards empowerment when using tools such as Microsoft Copilot Studio?
Whilst it may be progressive thinking for some, as Ben Gates exclaimed in National Treasure; here’s to the men who did what was considered wrong, to do what they knew was right.
Remind me, what is Copilot Studio?
During Microsoft Ignite 2023, Microsoft Copilot Studio was announced as a comprehensive platform for conversational AI to craft and personalize copilots in with natural language. Its seamless design process takes care of all the groundwork so we can focus on solving real user problems whether they’re internal or external, across industries, departments, or personas.
It really is like the Ben Gates of the tech world. A treasure hunter to unlock the secrets of productivity, creativity, and collaborative innovation on the way to transforming employee and user experience by
adopting first-party Copilots as they are (e.g., Microsoft Copilot for Microsoft 365 or Dynamics 365)
extending them through low-code development, plugins, connectors etc.
or building our own customized copilots just like the well-known framework of Power Virtual Agents
And what about public beta testing, what is that riddle?
Public betas are a crucial tool to test unreleased piece of software or new features with users. It is one of the latter stages of the software development lifecycle to simulate real-world experiences as much as possible. From testing Copilot stability to data accessibility and security frameworks, it is a fantastic approach for building trust with users in a collaborative feedback environment. This is how business SMEs shine, change champions are created and earlier innovation value is delivered.
The obvious reality is that software quality improves with time. However, how much time do we wait to deliver production value to consumer and a competitive edge to markets by exposing it to real-world conditions?
What does Microsoft Copilot Studio have to do with public betas?
With the fast-paced release of Copilot products this year alone, we are witnessing an unprecedented year of Microsoft announcements. Navigating that alone with the challenging details of which features are experimental, which region products are available in and when exactly is mind-boggling enough as it is. What if, once one business region has it available, the focus is on launching an MVP to the wild to get real insights ahead of a wider release?
Let’s summarize the benefits:
Unearth hidden secrets: Like cryptic clues, public betas will help uncover hidden bugs and issues when AI experiences interact with core business processes or third-party tools. There are over 1,000 Power Platform connectors to choose from after all!
Decipher user opinions: Soliciting early feedback makes users feel more connected by reigning in the tech chaos with us; instead of solving problems for them, we solve WITH them. This becomes even more important when deciding on the use of Copilot over legacy solutions we may not be ready to deprecate yet to avoid major user disruption.
Navigate uncharted territories: Public betas allow us to test new features and significant changes in a controlled environment whilst decreasing the potential for data loss. This is where we can leverage the power of managed environments to streamline our beta’s release.
Preserve historical versions: Like preserving historical artifacts, it’s crucial to maintain older versions of our product for users who aren’t ready to migrate to the beta version.
Gamify adoption with new use cases: Public betas can reveal unexpected ways users interact with Copilot, much like how Ben discovered the treasure’s true location. If we create anticipation over experimental features, we can elicit more testing volunteers who may find new ways to incorporate them in business processes. And adoption becomes an organic perk along the way.
Everything is so new, what quality proxies do we use?
There is no golden recipe to get the keys to the user success kingdom. However, we can leverage software development best practices such as:
UX interactions VS achievement of a business task: Did users get faster and more easily to an answer or the completion of a task? Whether a Power BI enthusiast or appreciate the power of Microsoft Clarity, it is important to understand the joint picture of a Copilot experience and its interaction with business systems.
Availability and frequency of outages or hallucinations: Just like a CRM system, this will only be as good as our data. Key metric examples include things like downtime or AI hallucinations which led to misleading answers. This is why enabling users to be a strong, conscientious pilot is crucial when sense checking any Copilot outputs.
Quality VS quantity of feedback raised: Whatever the ticket management system is (GitHub, Azure DevOps, Jira etc.), invest early on in categorizing feedback types on top of tracking numbers. User opinions are what matters most when initiating betas, especially in this fast-paced Copilot world, so treat it with care and respect.
How do we envision success metrics to move from beta to general release?
The reality is, most Copilot products are fairly new. It is the new treasure to hunt for, and relics to cherish. But with great power, comes great responsibility. How do we define a North Star for success, so we know when we have achieved our beta value? Again, the answer is a Venn diagram sweet spot between quantitative and qualitative indicators.
Functional clarity: Users have provided feedback in the key areas we needed to decide over new Copilot features or the roll out altogether. Whether good, bad, or ugly, we have sorted the feedback into categories which will unblock the backlog and future development. Maybe we decide to train our own model or built new connectors instead.
User trust: We have openly communicated with users throughout beta testing the purpose, feedback points and future intentions. Whether users should expect any disruption, a migration of processes or shiny new features to play with, we can’t take their trust or volunteering to test for granted. It is all built on user centricity and reciprocal trust, as part of wider adoption metrics.
Documentation completeness: Our development teams, business management and end-users have been provided with clear documentation of learnings (successes VS challenges) from solving business problems in a new AI-assisted way. A strong change management program will guide us to both structure that but also socialize it with the right audiences.
The tech labyrinth is never over
Rolling out new AI experiences, either pre-built or customized, is a journey rather than a standalone phase. As the Copilot products grow so should our plans for meeting more needs, responsibly and for deep human impact.
Whilst we were celebrating a few days ago the anniversary of ChatGPT in its early stages, we will similarly be celebrating the onset of Microsoft Copilot products allowing us to rethink how we build our business processes in a more connected digital workplace.
No matter whether we choose to be a pilot or a tester in every Copilot roll out, never forget the WHY and the WHOM FOR. Building with intention is the most important responsibility we have in our treasure hunting.