The Benefits Of Cognitive Inclusion In UX Research

Articles on Smashing Magazine — For Web Designers And Developers...

In the summer of 2024, I became co-chair of a working group of expert researchers who came together to determine how best to perform accessibility testing with people with cognitive disabilities. This was work I did for Fable, where I am currently VP of Innovation.

Cognitive disability is an umbrella term for several disabilities that impact how people process information, and it usually affects memory, focus, and/or learning. It is the most prevalent disability in the U.S. (13.9% via CDC), and cognitive disability is increasing rapidly (Yale study).

We set four goals for ourselves to learn how to work with this audience:

1. How should we recruit and screen participants?
2. What are best practices for research with cognitive participants?
3. Do these methods work in a real study?
4. Documenting what we learned so that we could share it.

We created a screener to recruit people who self-identified as having challenges with memory, focus, and learning. We also reviewed published studies that involved cognitive testers to learn best practices for working with them.

Next, we tested these best practices with an initial group of 25 testers in a pilot study. We fine-tuned our approach iteratively and created a guide to running user interviews with cognitive testers and a survey that could quantify their experiences using digital products. Finally, we documented what we learned.

After our pilot study with this new group of testers finished, I felt that they would uncover more usability insights than the general population (gen pop) user research participants I’d worked with in the past. I set out to validate this hunch.

The Cognitive Usability Study

I decided to run a joint study with Fable’s partners at the University of California, Irvine, in collaboration with Syed Fatiul Huq and with help from Fable researchers Pranav Pidathala, Ali Brown, and Michael Fagan to see if my hypothesis about finding more insights with cognitive testers proved true or not.

I generated three websites for the study using an AI prototyping tool. I wanted three different types of sites with different user goals and content so I could test a variety of tasks in the study.

Table 1: Websites And Tasks Tested

We used a single screener with questions about memory, focus, and learning, and screened participants into two groups based on whether they self-identified as having cognitive challenges or not.

Cognitive disability includes neurodiversity. Neurodivergent is an umbrella term used to describe people whose brains process information and learn differently. It is most commonly used for people who have learning disabilities (e.g., Dyslexia), ADHD, and Autism.

We ran 30 user interviews, 10 per website, with an even 5/5 split between cognitive and gen pop participants for each website. In each session, a participant completed all the tasks for one website during an online user interview facilitated by one of the researchers involved in the study.

All participants completed an Accessible Usability Scale (AUS) survey at the end of their session. This is a free, Creative Commons-licensed 10-question survey to evaluate the usability of websites and mobile apps.

Data Analysis Approach

I reviewed all the study recordings and transcripts and made note of every time a participant raised a concern, question, difficulty, or asked a question about how something worked. I counted all of these as issues. I also noted where a participant missed something that was part of a task, even if they didn’t notice it themselves. I also noted every suggestion for improvement made by participants.

Examples of issues found included:

• Photo is too tall and requires a lot of scrolling to get to content (noted by participant).
• I get no feedback when I like or dislike a book (noted by participant).
• Participant missed the required P.O. Box checkbox the first time (observed by me).

Examples of suggestions included:

• I would like to see a protein comparison in a table.
• The “More information” tab should be moved up higher.
• I would like more information on how the recommendation list is created.

Issues and suggestions were counted once per participant, even if they mentioned the same thing twice, but there are, of course, repeat issues and suggestions across the different participants. It is expected in UX research with multiple participants that you’ll find similar issues with each participant, and that is a signal that an issue is a universal challenge.

Findings Of The Cognitive Usability Study

Across the three websites tested:

• Cognitive participants identified 197 issues.
• Gen pop participants identified 113 issues.
• Cognitive participants made 93 suggestions.
• Gen pop participants made 54 suggestions.
• Cognitive participants surfaced more issues related to content, buttons, icons, visual elements, and media than gen pop participants.

The results aligned with my instincts: participants with cognitive disabilities identified 1.8 times more issues and made 1.8 times more suggestions than gen pop participants.

Let’s dive deeper into the data for each website. Note that an AUS score ranges from 0 to 100, with higher numbers representing better usability than lower numbers.

Table 2: Strong Snacks

This site had the simplest design and content of all websites tested in the study and accordingly had the lowest overall issues and the highest median AUS scores. The data aligns with what you’d expect from an easy-to-use and simple website.

On this website, cognitive participants found 3.4 more issues and made 2.2 more suggestions on average. Their average score of the overall experience was 13.7 points lower than that of the gen pop participants.

Table 3: Turning Pages

This was the website with the most varied functionality and the most tasks to complete (4), so it’s not surprising that participants found the most issues.

Here, cognitive participants found 6 more issues and made 3.2 more suggestions on average. They also scored the overall experience 17.2 points lower than gen pop participants on average.

Table 4: Crown & Comb

This website was intentionally designed to be complex, and task 3, finding the bridal package, was meant to be extremely difficult to complete.

On this last website, cognitive participants on average found 7 more issues and made 2.4 more suggestions. Their average score for the overall experience was 14.3 points higher than the gen pop participants.

Something interesting happened with the AUS scores for cognitive and gen pop participants in Tables 3 and 4. Cognitive participants scored Crown & Comb higher than Turning Pages, but gen pop scored the opposite — higher for Turning Pages and lower for Crown & Comb. If I had to guess why, I suspect finding more issues on Turning Pages impacted the cognitive participants’ perceptions of usability more than the gen pop participants’.

The other major difference between the sites, outlined in Table 5 below, was that cognitive participants found many more issues with buttons and links on Turning Pages and more issues with icons and visual elements on Crown & Comb. This suggests to me that the interactions being challenging on Turning Pages were a more significant challenge than issues with visual elements.

Qualitative Findings

When it comes to the more qualitative findings, I looked at trends in the types of issues found by both groups of participants.

Cognitive participants:

• Were more likely to flag issues with icons or visual elements.
• Surfaced problems with content more frequently.
• Gave richer qualitative commentary, often explaining why something was hard to find or confusing.

Gen pop participants:

• Were less likely to flag conceptual or comprehension barriers.
• Gave shorter feedback, often stopping once the task was complete.

Table 5: Number Of Issues By Category

When I grouped issues by category, the following issues surfaced more often with cognitive participants: content, buttons and links (affordances and function), icons or visual elements, and media (video, animations). They nearly tied with gen pop participants on navigation issues (45 vs 46).

Let’s look at the commentary provided by one cognitive participant versus one gen pop participant in the Crown & Comb sessions. The cognitive participant gave an AUS score of 38, and the gen pop participant gave an AUS score of 27.5. I chose to compare these two participants because they both gave the lowest scores within their group.

Notice the differences in how they described the overall experience in the quotes below. The gen pop participant explained it was frustrating and not engaging. The cognitive participant felt drained and less able to focus. I interpreted the experience as having a more profound impact on the cognitive participant’s overall wellbeing.

Gen pop participant quote

“As soon as you have a name of a treatment and a little explanation and like the duration and the price, as soon as you click onto that, it should be that you can interact with that service straight away. And I feel like if you’re seeing a service repeated on a page multiple times and you’re still not able to select it, it’s really, really frustrating. This feels not particularly engaging.”

Cognitive participant quote

“For example, like, the mental energy aspect of it, like, sometimes there’s, like, okay, cookies, and then ads, pop-ups, or maybe the website or service has too many options to look through, and maybe I just want something that I already know. I have to go through a lot of stuff. It makes me, like, feel drained and less able to focus.”

In summary, across all 3 websites we tested, participants with cognitive accessibility needs identified 197 usability issues, compared with 113 identified by gen pop participants.

Cognitive participants made 93 suggestions for improving the user experience, compared with 54 suggestions by gen pop participants.

When I compared issues and suggestions across both groups of participants, it turned out that the cognitive participants found 1.8 times more issues and made 1.8 times more suggestions than gen pop participants.

Cognitive participants surfaced more issues related to content, buttons, icons, visual elements, and media than gen pop participants.

How Cognitive Participants Benefit UX Research

In working with cognitive participants for the last few years, I’ve seen how they surface cognitive load issues consistently. These issues don’t just impact people with cognitive disabilities such as neurodivergence; they also impact:

• Gen Z who lives in a world of short videos optimized for attention-grabbing and struggles to focus on long-form and written content.
• Seniors who naturally experience cognitive decline as they age and have difficulty with complex interactions, especially online.
• Adults with jobs and families who are constantly busy, overloaded with information, making their attention and focus difficult to grab.

What would I have missed if I hadn’t included cognitive participants, and how might that have impacted the business outcomes for these websites?

Strong Snacks

On the Strong Snacks website, the cognitive participants surfaced:

• They would trust the content more if there were links to the sources of information, such as scientific journals.
• The need for more context in headlines to understand what the blog is about.
• Lack of clarity of the label “Add-ons.”
• Layout concerns where recipes for snacks interrupted the main article flow instead of being placed in a sidebar with a distinct design.
• How ads and animations can distract some users from reading the content.

These are improvements that would give all users more trust in the content while also making it easier to read and skim for key content. The research findings point towards design best practices, such as not having continuous animation and using layout to draw attention to different types of content that a senior designer might also point out.

Turning Pages

Without cognitive participants, we might have missed the more subtle but important issues with confusing interactions, such as how the “Add to book bag” button worked. They were also confused about where reviews and recommendations came from. Both of these issues could decrease a user’s trust in the website.

All participants surfaced that the book-matching feature was hard to find, but the deeper problem the cognitive participants emphasized is that the site’s interactions don’t consistently behave in ways that they can predict and understand, decreasing their confidence.

Anyone who wants to buy a book could benefit from a clear understanding of how to add books to a cart and complete the checkout quickly and with no ambiguity. Compounded over hundreds or thousands of users, a lack of clarity in a purchase flow will lead to lost revenue.

Crown & Comb

The Crown & Comb website in particular highlighted the benefits of having cognitive participants who raised:

• Concern around why a service would be “subject to stylist consultation.”
• Uncertainty with services that had similar labels but may or may not be the same service.
• The importance of choosing a date being early in the flow for booking appointments.
• Lack of clarity about when or how they would pay for services.

These issues likely also affect gen pop participants, but they are more likely to muddle through a task with incomplete information. However, that can lead to losing customers to a better experience if a competitor pops up. Loyalty is often tied to experiences, not just brands, and having a poor experience means your customer retention can be weaker.

The study showed that finding a bridal package was hard for everyone, but the cognitive group showed how that became an accessibility barrier. When you combine:

• too much ambiguity,
• too many decisions,
• too little user feedback, and
• too much effort to find something,

You create a high enough cognitive load that some people will not be able to complete the task. In my opinion, this is where usability issues start to become accessibility barriers — when they increase cognitive load so much that it becomes overwhelming for some users.

Key Takeaways

• Include people with cognitive disabilities in user research, not just accessibility research.
  They can surface general usability issues related to content, buttons and links, icons or visual elements, and media while also helping you understand how your product functions in terms of cognitive load.
• Cognitive issues are both usability and accessibility issues.
  Tasks that rely heavily on memory, focus, and decision-making can move along a scale from difficult to impossible for some users to complete. That’s where usability challenges become accessibility barriers.
• Track more than task completion.
  Ask users how they feel, how a task affects their energy, how distractions impact their ability to focus, and how easy or hard a task was for them.
• Start small and build your cognitive inclusive research practice over time.
  Even a few sessions with people who have cognitive access needs can help you better understand how to manage cognitive load for all users.

Start Incorporating Cognitive Insights Now

The percentage of people aged 65 and older in America is projected to increase from 17% to 25%. By 2060, 1 in 4 Americans will be an older adult (U.S. Census). This is where everyone starts to experience cognitive decline. As the aging and cognitive population segment expands, companies will need to build for these more complex user needs.

People with cognitive access needs are a natural starting point because they will find the types of usability issues that UX teams are used to. This could make cognitive an easier entry point for inclusive research. Getting insights from assistive technology users is still very important, but many teams don’t know how to start doing that.

Cognitive accessibility is a powerful on-ramp into broader accessibility research and testing. By focusing first on cognitive load, clarity, and predictability, we build research foundations that make future work on accessibility with screen readers, screen magnifiers, and alternative navigation users more approachable.

“2 sessions with cognitive users feel like 200 because of the volume of insights we get.”

—UX Manager at Bell Media

In this small exploratory study, participants with cognitive disabilities identified 1.8 times more issues and made 1.8 times more suggestions than gen pop participants. I’ve seen this type of impact in research conducted by Fable customers’ websites that aren’t AI-generated, too.

Cognitive inclusion in UX research is not optional, and it’s not just about accessibility. It’s how UX teams can make their research more efficient, create clearer content, simpler flows, and ship better products for everyone.

Study Limitations

This study had a relatively small sample size, so the findings are more qualitative than quantitatively validated. Testing was also done on two different platforms. Cognitive participant sessions were run using Fable Engage, and gen pop sessions were run on UserFeel. Different platforms with unique participant panels can affect the quality of insights and comfort levels with user research participation.

Disclosure: I work for Fable and chose to use our platform because it was more affordable than paying for access to another research platform, allowing me to include more participants in the study at a lower cost.

Different researchers facilitated the user interviews, which can also affect findings, but all sessions used the same task structure and discussion guide template, and all were completed online. Even though the sessions were facilitated by different researchers, the issue and suggestion counts were all done by me to ensure consistency across all websites and participants.

Resources

I’ve compiled a few useful resources as you begin your cognitive inclusion journey.

• W3C supplemental guidance on cognitive accessibility and more detailed guidance from the cognitive accessibility task force
• MDN overview of how the core web content accessibility guidelines map to cognitive accessibility supports
• Case study: Fable’s cognitive accessibility pilot
• 5 simple fixes that make digital spaces calmer—for neurodivergent and all users
• Neurodiversity and UX: Essential Resources for Cognitive Accessibility

More at Articles on Smashing Magazine — For Web Designers And Developers