A team I was on ran a bunch of experiments, which was great. We learned a lot, and we had some ideas about what might work better. But sometimes we relied too much on data to make decisions, and when the data was incomplete or inconclusive, we were stuck. We lacked confidence.

I was in a meeting with this team where they wanted to make a decision about whether or not to ship an experimental variant that, in theory, would be better than the existing experience — but unfortunately the data collected during the experiment was incomplete and unreliable.

We had no quantifiable measure by which we could make this decision, and it seemed like we would have to run the experiment again — or proceed without much emperical support, though everyone was quite nervous about this proposition. Hmm.

I’ve often heard it said that “to properly evaluate design, you must know the objectives of the organization” that produced it to know if “it worked.” To evaluate a company’s onboarding flow or marketing campaign, you must know the metrics they are using to define success and the internal corporate strategy that the flow or campaign seeks to impact. Basically — how do you know it worked if you don’t know what it’s trying to do. Makes sense. This is not an unreasonable premise, but must come with caveats.

And while I don’t entirely disagree — if design is how it works — then you must be able to define what something that “works” looks like. Design should always advance an organization’s strategy and goals. However, I challenge the idea that this context is a requirement to evaluate the work in any capacity. In fact, with experience, I believe that we can evaluate the strength of a design to a great degree a priori — often enough to make a good decision, with sufficient confidence, about next steps or outcomes. This is an important distinction to make.

I’m not alone in this thinking, either — notice how all of Dieter Rams’ 10 Principles of Good Design assume an objective viewpoint.

As we develop our practice of design — as we learn to talk about design more definitively, and as we go through the process of measuring the impact of our work — we develop an instinct (or taste?) that allows us to (with some certainty) hypothesize about what might or might not work — and I believe that for a design team to move quickly enough, this instinct must be trusted. If we wait for complete, empirical data that confirms each and every of our assumptions, we’ve already lost the race.

There are also universal principles that can help us here — color theory, composition, hierarchy, typography, etc — which in most cases can be used to evaluate design independent of its intended purpose. See Rams’ list above as another great starting place. Great design satisfies both criteria — the aesthetic and the functional — or put another way, the universal and the local. Great design should demonstrate a mastery of the fundamentals and provide value to our business or organization. 

Of course, the level of confidence we need to proceed will differ depending on the project, but the takeaway is to constantly ask ourselves if we have enough confidence to reasonably believe we’re improving our product or experience. Are we making it better? Often, improving the fundamentals will lead to an improvement in the product experience, and maybe even create an impact towards our goals.

As a product team — we should know what good looks like. We should know what “good enough” looks like. We should probably even know what great looks like. We should talk to our users often — at least enough to know what they need, what they want, and what they expect — but users should be just input of many to our definition of great. We should be willing to make decisions based upon that shared understanding — and those instincts which have been developed over years of observation. It’s not a guess, it’s a judgment based on experience.

We should absolutely leverage data whenever it’s available, and we should use it to inform our decisions wherever appropriate — but we should also develop the ability to make decisions based on our experience or instinct, especially when it’s clear. We need to be comfortable moving forward with 50, 60 or 70% confidence rather than shipping too late as we approach 100%. And importantly, knowing how much confidence we need to proceed is a skill that must be learned and practiced.

In the meeting described above, I unmuted my mic and said “I’m new to this team, but looking at these two options, it’s clear to me that option B is a vast improvement — it’s well-designed and much clearer, better organized and poses little risk but a lot of potential upside… why don’t we ship it?” And with some nods and alignment — I think we will¹.