From concept to code: designing a new AI/ML-powered enterprise product.

Background

The product was envisioned as a way for system administrators to deploy, monitor, and control their company's internal and external applications in a simple way, while the platform handled the complex management behind the scenes.

Problem

Administrators were accountable for applications spread across separate products, each configured on its own, and their monitoring tools buried real issues under noisy alerts. Executives identified an unmet need in the marketplace where modern product design and automation helped customers understand complex issues at-a-glance. I was brought in to design the initial concept into a full product that was eventually delivered to General Availability.

At the start, I took time to understand a new domain and build relationships with Principal Architects and Product Managers. The diagram below is a rough visual interpretation of my initial understanding of the different sub-systems that, combined, met the goals and desired outcomes of leadership.

Redesigning instead of patching

As my relationships with stakeholders and understanding of the product direction grew, it became apparent that the desired solution wasn't feasible with the current IA structure. Saying so wasn't an easy conversation. Leadership's first reaction was to patch and extend the existing structure rather than redesign it, since a full redesign meant delay. My counterargument was about compounding cost: a patch might hold for the next feature or two, but it would eventually break down against use cases we already knew were coming, and absorbing that cost early was cheaper than paying it later, compounded. Below is the initial Information Architecture concept.

The diagram below shows where the team initially aligned on the design of the IA. After interviewing administrators familiar with the product and addressing their workflows, we were able to iterate and elevate the most important features. Network Functions (NF) were the initial building blocks used to create the ML models and AI feedback system. NF were reusable across any application, with analytics shown in many primary and secondary views, giving the administrator one centralized location to change the NF, streamlining their workload.

The IA called for first-level left navigation. This gave a lot more vertical space and was in accordance with the consistency goals the design team had across the product line. The end result brought a lot of clarity to the product and helped align product management, design, and engineering on our shared mission. We also reskinned the product in the company's new design system, working closely with the design system team on component creation in Figma.

Two founding values guided this work: (1) make analytics and configuration options pervasive at any level that made sense, and (2) trust is very hard to build, so automate incrementally with users' permission. It was likely that users would want to feel in control of their instance and solve problems manually, while also automating incrementally over time by delegating tasks as they became comfortable.

Earning trust incrementally

There was real pressure from architects and leadership to push for more aggressive, fuller automation out of the gate. I pushed back toward incremental, permission-based automation instead: administrators were going to be accountable for systems they didn't fully understand yet, and asking them to hand over control before trust was earned risked the product being rejected outright rather than adopted gradually.

I designed the Universal Userflow that helped align the team in understanding the importance of the administrator, who needs to understand and trust the context of an issue as well as make relevant decisions based on real-time insights.

There was a critical need for a feedback loop notifying the administrator of system changes or when thresholds were breached. Elevating actionable insights into a central location was a key problem. I often used a driving question to align the team on the core problem and its design solution:

"How will the administrator know the product is doing its job and showing value to them?"

A shared model of the data

Talking about AI/ML is exciting but it can get confusing, especially with a new data model in a new product. I created the diagram below to help align the team's understanding, at a point when we were working on the third tier: Analytics, Metrics, Segments, and Training Data.

ML relies on massive amounts of data. Providing a central location for a user to observe and act on insights was critical to getting the customer to use the product more often, fulfilling the need for product-led growth while helping track preferences for better ML insights in the future. NF didn't start out reusable. The earliest thinking treated each app's configuration independently, closer to how the products it was unifying had always worked, and the cost showed up almost immediately: administrators would have had to manage the same parameters separately for every app, repeating any threshold or security change everywhere it applied. Consolidating that into one reusable NF model removed the duplication and gave administrators a single place to make a change that applied everywhere it was needed. Think of NF as ingredients that make up a meal, reused across different apps and swapped in or out as the meal (or menu) evolves.

After enough design progress was made, I started to validate the primary administrator workflow and dashboard concepts: an initial screening survey, 14 remote-moderated interviews for concept testing, and seven usability tests for follow-up validation.

Findings

• 44% of customers think "noisy alerting" is the main frustration they have with their current monitoring tool
• Alerting outside their monitoring tool is the first touch point administrators have to start proactively troubleshooting end-user issues and configuration problems
• General administrator workflow was effective for administrators' primary tasks
• Users loved the reusable aspect of NF and configurable dashboards

• Reached General Availability with a unified IA across four existing products and a foundational AI/ML framework for intelligent automation
• Product funding doubled and engineering grew from 43 to 125 during development
• 85% primary task success rate on launching a new app and re-configuring existing app
• 100% of the administrators felt the tool was something they would use regularly with their teams

The two calls that took the most pushback were redesigning the IA instead of patching it, and automating incrementally with administrators' permission instead of all at once. The rest of the product ended up resting on both.

Eventually, we planned to build the Universal Notification feed and focus on end-user personalization. It shipped after my involvement in the product wound down, built on the foundation laid here.