A research program for a pulse oximeter app

A pulse oximeter clips onto a fingertip and reads how much oxygen your blood is carrying, a number called SpO2. This one was a consumer device, made to work with an app on the user's own phone, iPhone or Android. In December 2020, World Wide Technology proposed to design and build the app with the device maker, and I wrote the proposal's pages on the research to test it.

The proposal planned the app in two phases. The first was a general wellness product, under the FDA's policy for low-risk products that support a healthy lifestyle. The second would take the app to the FDA for clearance as a medical device, a 510(k) premarket submission. So the proposal had every round of testing written up in the design history file, the record a reviewer reads, from the start.

Two months earlier, I had written the Human Factors skill area for World Wide Technology's UX career ladder. This was the program where I planned that work for a medical device.

Building the UX practice→

A usability test is only as good as the people in it. The app was for people living with lung disease, so the panel had to be them. The criteria named two diseases, pulmonary fibrosis and COPD (chronic obstructive pulmonary disease), each diagnosed more than six months before.

Draft 3.0 asks for 80 people to keep at least 60 over 20 weeks. Each disease gets 30 to 40, each half on iPhone and half on Android, and each of those half men and half women. The ranges leave room for people to drop out: the low ends add up to 60.

Everyone had to measure their blood oxygen with an oximeter at least every other day, use the internet on their phone at least monthly, already use a health app, see a doctor at least once a year, read and speak English, and be 18 or older. The screener also asked for a mix of ages and races.

Some people know too much about a product like this to test it fairly. The screener turns away anyone who works, or has someone in their household who works, in health insurance, as a physician or nurse, for a healthcare device maker, in market research, or in website or app design or development. It also turns away the self-employed in those fields, and anyone who sat on a health research panel in the last six months.

The last question asks people to rate seven oximeter makers from 1 to 5. Anyone who rates the client a 1 or a 2 is screened out, since a tester who dislikes the brand before starting could color every answer about its app.

Fidelity is how close a test is to the finished thing. The schedule starts low and climbs. Weeks 3 to 13 hold ten short remote rounds of 15 to 20 minutes. The unmoderated ones, where people test on their own, use clickable prototypes to check our human factors design assumptions. The moderated ones, one on one with a researcher, use the latest app release, and from week 9 the oximeter too.

Then come three one-hour sessions with the app and the oximeter together, end to end: 48 people on their own in week 15, then 16 with a moderator in week 17 and 16 more in week 19. That is all 80, the formal study of the app at the end.

We split the panel into five groups of 16, with 2 from every box of the grid, and the groups take turns, each coming back for a second short round five or six weeks after its first. So no round's testers would have seen the round just before it. After each round, the proposal had the UX team bring its findings to the product owners and update the design history file.

The December 2020 proposal named asthma and COPD, 40 people each, ages 18 to 65. By draft 1.0 in January, pulmonary fibrosis had replaced asthma, and the age cap came off: adults of any age, with a mix asked for.

From draft 1.0 to draft 3.0, ten days later, the plan grew from 16 weeks to 20. The short rounds went from six of each kind with 3 to 5 people to five of each kind with up to 16. The bar for taking part rose from measuring blood oxygen weekly to at least every other day. The oldest iOS and Android versions allowed went back about three years, from iOS 14 to iOS 11 and from Android 10 or 11 to Android 7 or 8. The fixed quotas of 40, 20 and 10 became ranges. And the healthcare exclusion narrowed from anyone in a provider's office, billing included, to physicians and nurses.

The program was proposed and did not run. There are no findings, and nothing in it was measured.

What the files show is the design. The panel is split into eight equal boxes and sized to survive dropouts, so a reviewer could check it box by box. The screener names whom it turns away. And the schedule doesn't put the app in front of the panel for the formal study until people have tested it with the oximeter in hand.