User testing is essential to good design practice. But it's not infallible, and treating test results as the final word on whether a design works is a fundamental mistake. As designers, we need to recognise testing as one critical input among many, and understand its inherent limitations.
This might sound controversial, but I've learned this lesson the hard way: you can't take user testing at face value without applying your own critical judgment.
Here's the uncomfortable truth: every time you test something, you're not testing something else.
When we design user tests, we naturally focus on the specific journeys or features we want to validate. But this focus creates tunnel vision. The questions we ask direct participants' attention to certain elements while rendering everything else invisible. Users concentrate on completing the tasks we've assigned, which means anything we haven't explicitly asked about goes unexamined, even if it's right there on the screen.
This isn't a failure of the research method. It's an inherent limitation of directed attention.
Photo by Edryc James P. Binoya on Unsplash
"Every time you test something, you're not testing something else."
Design education emphasises the importance of user testing and where it fits in the process. What it doesn't teach is how to critically evaluate and use those results. There's often an implicit message that disagreeing with test findings means you're being defensive or ego-driven, that "data-backed" decisions are inherently superior to design judgment.
But research without clear intention is just theatre. If you're conducting testing to make stakeholders happy, to check a box, or to cover every possible question in a single session, you're not designing research to answer what you, as a designer, actually need to know.
The goal should be focused: validate the specific design decisions that need validation. Additional insights are welcome, but not at the expense of losing sight of your core questions.
We recently launched several redesigned customer journeys that had performed well in user testing. Once live, we were flooded with negative feedback. While some was change aversion, much of it revealed genuine problems our testing had completely missed.
Photo by Tanja Tepavac on Unsplash
We were testing changes we'd made to specific parts of several pages. Our tasks focused on validating those changes. What we failed to consider, and therefore didn't test, were the features we'd removed from those same pages. Since none of our tasks related to the removed functionality, participants never noticed their absence. They were focused entirely on what we asked them to do.
We also missed feedback on fundamental design decisions: information hierarchy, layout, visual density, spacing between elements. These weren't the focus of our testing questions, so participants didn't comment on them. But when thousands of real customers encountered these pages daily, the problems became immediately apparent.
The tunnel vision wasn't malicious. It was structural. Our questions created a spotlight that illuminated some things while leaving others in shadow.
Working in financial services adds another critical dimension. Money is inherently emotional. Testing with fictional amounts in a risk-free environment doesn't activate the same psychological responses that real customers experience when managing their actual finances.
When there's no real money on the line, or when the scenario is too far removed from reality (asking someone to pay $10,000 when they've never paid more than $2,000), participants can't fully relate to the experience. Combined with the cognitive bias to agree with researchers, you may get answers that feel valid in the moment but don't reflect real-world behaviour.
There are other gaps too:
The question we asked was: "Does this page have all the information you need?" The answer was yes, the information was there. But we never asked: "What if this information wasn't here? Would you know where to find it? Is seeing it immediately worth the space it occupies?"
You can only get answers to questions you actually ask.
It's dangerously easy to accept test results when something doesn't feel right simply because it's a comfortable position. Your decision is "backed by data," which shields you from confrontation and provides political cover.
But experienced design intuition is valuable. If something isn't sitting right, there's usually a reason, even if the data says it's fine. The trick is learning to interrogate both your intuition and your data with equal rigor.
Photo by Andrea De Santis on Unsplash
Many of our releases were technology-driven: proving new infrastructure worked, ensuring compatibility with legacy systems, demonstrating smooth transitions between tech stacks. These are legitimate technical goals, but we weren't leading with customer experience.
Some of these journeys had been designed by an agency before our team existed, which limited our input. We raised concerns pre-launch, but couldn't stop the releases. What we could do was put monitoring mechanisms in place and develop rapid response plans.
When the negative feedback arrived, we were ready. We responded quickly, demonstrating that experience design isn't just "drawing pictures". It's understanding customers, anticipating problems, and knowing how to react when issues surface.
Our design team is relatively new in an organisation that's been through multiple technological transformations. After more than two years, many people still don't understand what we do, how to engage with us, or what value we bring.
Being prepared for the post-launch feedback and responding effectively gave us something crucial: demonstrated value. It showed the organisation that release decisions can't be purely technical, and that experience design has real, measurable impact.
That's how you earn a seat at the table. Not by demanding it, but by proving your discipline contributes to outcomes that matter.
When reviewing test results, challenge yourself with these questions:
User testing is invaluable, but it creates tunnel vision by design. The questions we ask determine what we can possibly learn. The scenarios we create can never fully replicate emotional reality. The tasks we assign direct attention toward some things and away from others.
This doesn't mean testing is broken, it means testing requires critical interpretation.
Challenge your findings. Trust your design judgment while interrogating it honestly. Recognise that testing captures a narrow slice of a complex reality. Most importantly, ensure that customer experience drives your releases and not just technical milestones or the comfort of having "data-backed" decisions.
The data should inform your decisions, not make them for you. And sometimes, the most important questions are the ones you forgot to ask.