Does Optimization Leave Room for Creativity?

Someone asked me this recently, and I’ve been thinking more about it.

The assumption behind the question is optimization is like a machine that takes in a problem and returns one answer. The answer. But where does human judgment fit? What about context, nuance, experience?

After many years seeing the application of mathematical optimization to various operational problems (eg in retail, manufacturing, supply chain, workforce planning, etc) I’ve concluded that optimization isn’t always the right word for how this technology is applied because it doesn’t necessarily reduce your options. It can sometimes be used to define the space of good options, and then gives you lots of room inside that space in which to work. So the creative work can still happen, but typically within the defined space.

Here are three stories that illustrate what I mean.

Story 1: 1000+ valid answers

We recently worked with a European professional sports league on their annual season scheduling process. The rulebook is extensive, with dozens of hard constraints governing which teams can play each other and when, which rounds are blacked out for European competition, how home and away stretches must be distributed across the season, and more. 

The league has a tradition of selecting the season schedule through a public random draw. One schedule is chosen by lottery from a pool of candidates. The randomness is intentional, as a way of maintaining trust with clubs, broadcasters, and fans. This way, nobody can accuse anyone of putting a thumb on the scale.

The first time we attacked this problem as a monolithic MIP (mixed integer program) we got a really good, but single schedule. They were already impressed. But they wanted 999 more. We didn’t know if that was even possible. 

So we took a totally different approach. It took a few different tries but we eventually came upon a novel approach that mixed MIP with CP (constraint programming). It actually generated several thousand valid schedules.

So then we started thinking about how to reduce it to only the 1000 they wanted. We realized we have a lot of creative freedom here. Because deciding which candidates to include in the draw pool is itself a creative and strategic act. For example, do you maximize diversity, ensuring the pool spans the full range of what’s structurally possible? Do you cluster by characteristics, so you can show the league that some schedules favor certain qualities over others? Do you try to highlight the tradeoffs that exist within the compliant space for a person to manually choose from?

The optimizer made all of that possible by doing the hard work of ruling out the combinations that would never have worked. Everything inside the feasible space is yours to work with.

Story 2: Exploring the spectrum between fast and cheap

Earlier in my career, working with a telecommunications company on workforce and work order planning, we ran into a version of this question in a different form.

The client didn’t want one plan. They wanted to explore. Specifically, they wanted to understand the tradeoff between completing work sooner versus completing it cheaper. And they wanted that understanding to be visual, interactive, and intuitive enough that business leaders (not just analysts) could engage with it.

So we built a system that generated a spectrum of plans, ranging from the fastest possible completion of all work orders on one end to the lowest possible cost on the other, with options in between. Each point on the spectrum was a fully optimized plan, meaning the best possible outcome given a particular weighting of the two objectives.

The business leaders could move along that spectrum and see immediately what they were trading away. “If we want to finish two weeks earlier, here’s what it costs us. If we’re willing to wait an extra month, here’s what we save.” The decision about where to land on that spectrum was entirely theirs but at least it was informed, visual, and backed up by real numbers.

So in this case it’s not the computer making the decision, but more that it’s making the decision legible, so the humans who have to live with it can make it well.

Story 3: A plan that starts smart and stays human

A third example, also from earlier in my career, highlights the challenge of getting manual planners (who can be amazingly good but sometimes resistant to change) to let a software application participate at all. 

A client needed to build and adjust schedules throughout the day, and they had strong opinions about how they wanted to do it. Sure, we tried to get inside their heads during the consulting project to understand how they thought about the problem so that we could replicate their thought process. But sometimes even they didn’t know why they did things a certain way. Or perhaps they didn’t want to share. 

So we built a system where the optimizer produced a high-quality starting point, meaning a plan that already respected most of the rules and constraints – and then handed it to the planner. The planner could click on any element of the gantt chart, drag it to a new position, and release it. On release, two things could happen: the system would attempt to reshuffle surrounding elements to accommodate the change, or it would generate a warning explaining which rules the new arrangement would violate (eg due to travel time the technician would have to be paid overtime).

The planner was never blocked. They could override the system if they chose to. But they were never flying blind either. Every manual change came with an immediate, specific accounting of its consequences.

What resulted wasn’t an optimized plan in the traditional sense. It was a human plan, built with optimization as a collaborator. The planner’s judgment and contextual knowledge shaped the final output. The optimizer contributed structural intelligence and guardrails. Neither one could have produced that result alone.

What these three stories have in common

In each case, the value of the optimization wasn’t that it made the decision. It’s that it changed what kind of decision the humans were making.

In the sports scheduling case, the decision moved from “which of thousands of possible schedules should we construct manually?” to “which of 90,000 already-valid schedules should we draw from?”

In the telecoms case, the decision moved from “how do we balance speed and cost?” to “which on this spectrum of optimized options do we want to go with?”

In the scheduling interaction case, the decision moved from “build a plan from scratch and hope it’s compliant” to “adjust this intelligent starting point and understand the tradeoffs immediately.” Mathematicians even have a name for why this is possible. It has an unfortunately negative-sounding name: degeneracy. It means the case in which there are multiple equally good solutions, or the creative space. 

So the point here is that the creativity, judgment and contextual wisdom that experienced humans bring to hard operational problems doesn’t go away, but rather, gets to operate in a less noisy environment.

That’s what I find most exciting about this work. I mean, don’t get me wrong, the math is genuinely cool, but the way it changes what’s possible for the people who have to make the decisions is even cooler. 

 

Nevra Ledwon is Managing Director of SimpleRose, a provider of mathematical optimization technology and services. SimpleRose helps organizations apply advanced optimization to their hardest planning and scheduling problems.