Deciding what to test first in a utility's grid equipment tracker

A Lean UX workshop helping a large utility decide what to fix first in the tool that tracks its grid equipment

My role

Lead product designer

Employer: World Wide Technology
Client: A large utility
When: June and July 2022
Format: Five days online, eleven people

Coached one of the two small teams in a five-day Lean UX workshop, the one working on outages.
Drew the wireframes for both ideas the teams chose, and narrated the storyboards, illustrated with a library of workshop assets I had helped build at World Wide Technology.
Co-wrote the 83-page workshop book that the utility took away: the ranked hypotheses, the personas, the journey maps, the storyboards and the wireframes.

20 things everyone assumed, written down and rated for how much each mattered and how risky it was
19→10 hypotheses ranked down to a top ten, most with their passing bar written before any test
Half of the top ten were about getting the tool to talk to other systems, not about screens
2 ideas sketched as storyboards and wireframes: Data Stewardship and the Incident Impact Diagram

Summary

A large utility keeps a list of the equipment that helps run its power grid in an in-house tool, and reliability rules say that list has to be right. Adding and updating records had gotten so clumsy that people copied the data out and worked on it somewhere else, so the official list kept falling behind.

Our job wasn't to redesign the tool. It was to sit down with the people who own it and use it, work out which fixes were worth building at all, and leave them with a way of doing that on their own next time.

Who was involved

Stakeholders The managers with a real say in where the tool goes.
Grid specialists Subject experts who know the grid.
Everyday users People who use the tool every day to keep the equipment list.
Two small teams The eleven people in the room split in two; I coached the one working on outages.

Wireframe of the incident diagram on a laptop frame: a vertical network topology on the left with the failed application and server marked in red, and on the right the selected asset's record (the application, its line of business, MCP, compliance requirement and incident history), beside an ownership list of four roles, each with a checkbox, and an empty Bridge Team column. Wireframe 1.7, page 56 of the workshop book. The idea behind it came with its own test: someone could find an outage and explain its full damage in under five minutes. The utility got the test; I don't have any result from it. — **What broke, and who to call, in one view.** Clicking the broken piece opens it beside the network map: the application, the part of the business it serves, the compliance rule attached to it, what has gone wrong with it before, and the four people who own it, each with a checkbox to pull them onto the emergency conference call. Hypothesis #2 put a number on what that was worth: half an hour off a typical outage fix.

Context

The utility's in-house equipment tracker keeps a list of every network device, computer, piece of software and database that helps run its power grid. Reliability rules for the grid say that list has to be right.

The tracker did the basics. But as the records grew more detailed, people started avoiding the tool and working on copies elsewhere. Every copy left the official list further behind, and people trusted the data, and the tool, less. That put grid safety and the utility's ability to meet those rules at risk.

Research

Before the workshop, we sat down with the executive paying for it and the subject experts to agree, in one sentence, on the problem we were solving. We also wrote down 20 things everyone assumed about the business and its users, and rated each one for how much it mattered and how risky it would be to be wrong.

In the workshop, each team built a persona, a made-up but typical user the whole room agreed on: Sean, a telecom engineer who keeps the equipment records up to date, and Jason, an analyst who sorts out outages as they happen. Each team drew a journey map of one persona's week. The same pattern kept showing up: relief when a job got finished, and worry whenever they weren't sure the data was right.

Assumption map

Research first

Each dot is one belief, placed by how much it mattered and how much we actually knew. The top right corner is where being wrong would hurt most and we knew least, so that is where the research went first. Two beliefs landed there.

Held up
Partly held
Didn't hold
Never tested

ID	Assumption	Certainty	Importance	Status
A1	People copy equipment data out of the tracker to work on it elsewhere because adding and updating records in the tool is too clumsy.	High	Critical	ExploratoryParticipants described this in the workshop, and it framed the whole problem, but nobody measured it.
A2	People who keep the equipment list trust it less whenever they can't be sure a record is right.	Medium	High	ExploratoryBoth journey maps showed relief when a job got done and worry whenever the data might be wrong, drawn from what participants said.
A3	An outage diagram showing every affected service and connected device would cut half an hour off a typical outage fix for analysts like Jason.	Low	Critical	Not testedIt passed if someone could explain an outage's full damage in under five minutes; the utility got the test, and we have no result.
A4	Letting analysts attach equipment records from the tracker to Incident Command messages would make outage communication faster and more complete.	Low	Critical	Not testedIt passed if someone could create and send outage details within five minutes in a usability test; no result came back to us.
A5	Showing an accuracy score on each equipment record would help analysts act sooner after an outage report arrives.	Low	High	Not testedIts time target was never filled in and still reads X mins.
A6	Keeping the list of applications open while a record opens beside it would fix Sean's worst moment, going through records one tab at a time.	Medium	High	Not testedThe idea came from Sean's journey map and was sketched as wireframes, but nobody measured it.
A7	Most of the fixes worth building for the tracker would be changes to its screens.	Medium	High	Didn't holdHalf of the top ten hypotheses were about getting the tracker to talk to other systems, like the trouble-ticket system, not about screens.

Journey map for Sean, a Telecom Engineer persona, with an emotion curve crossing five numbered moments that matter. Green peaks mark verified statuses and meetings; red troughs mark repeated asset reviews and unexpected markups, each annotated with a quote and a business impact. Page 13 of the workshop book, drawn by the telecom engineer's team from what participants told us about their own work. Sean is a persona, a made-up but typical engineer the room agreed on, not someone we interviewed, and the ups and downs come from what people said, not from measurements. — **Where trust runs thin.** Sean's week, with the five moments that mattered most and what each cost the business.

Design

The teams turned the assumptions into 19 hypotheses, best guesses written so they could be tested, each with the cheapest experiment that could check it and, for most, what would count as a pass. Both teams' hypotheses were ranked by how risky and how valuable each seemed, and the top ten were kept. Hypothesis #2, for example, promised to cut half an hour off a typical outage fix, and it passed if someone could explain an outage's full damage in under five minutes.

A real network outage on the last morning pulled several people away, so we moved the sketching session (a Design Studio) a week. I turned those sketches into wireframes for two of the ideas, Data Stewardship, for keeping the records tidy, and the Incident Impact Diagram, for seeing what an outage touches, and narrated each as a storyboard. The storyboards are drawn from a library of workshop assets I had helped build at World Wide Technology, so a team can tell a user's story without a week of illustration.

Storyboard panels 2.4 through 2.6 of Jason's incident diagram narrative: he identifies a failed network switch from the impact diagram, traces connected devices upstream and downstream, and tags them into an impact report. Page 45 of the workshop book. Sketches from Jason's team, cleaned up into storyboards. It's the story of how Jason would use it. Nobody had tried it yet. — **Jason's outage as a story.** He spots a broken switch on the diagram, follows the devices connected to it, and adds each one to an outage report.

Wireframe one step further into the incident diagram: the selected switch, SW Data Center 01, lists its child connections (one marked Impacted, one Inactive), and its parent connections (RTR_Datacenter_01 marked Not Impacted, RTR_Datacenter_02 Inactive), with an Attach action on the node and one owner's initials now showing in the Bridge Team column. Wireframe 1.9, page 58 of the workshop book. We handed this idea over when our part ended, and I don't know what was built from it. — **The report builds itself.** Every device above and below the broken switch gets a label, and one click adds it to the outage report.

Wireframe of the data stewardship concept: the Applications list on the left with an asset data tool selected, and a details panel open beside it, tabbed General, BIA, Ownership, DR, Network, Infrastructure, Certification and Feedback, showing 229 deployed production servers under Infrastructure. Wireframe 1.2, page 50 of the workshop book, cleaned up from the sketches of the telecom engineer's team. We think it fixes Sean's worst moment because of what her journey map showed, but nobody measured it. — **The details panel stays open.** The list stays put while the record opens beside it, so you can work through one application's 229 servers without losing your place.

Outcomes

The utility left with a ranked list of ten hypotheses to test, most with their passing bar written in advance, and storyboards and wireframes to run the riskiest experiments with. Ranking the list also showed that the quiet assumption, that most fixes would be screen changes, didn't hold. It was good to learn that before anyone spent money building screens.

We didn't measure results, and I don't have any from the testing the utility did after my part ended.

Two personas, Sean the telecom engineer and Jason the outage analyst.
A one-sentence problem statement agreed beforehand with the paying executive and subject experts.