What is optimization?

I did not really know what optimization meant when I first encountered it.

Before I started working as a data scientist, I mostly associated the field with machine learning and deep learning: collect data, train a model, and predict some future quantity.

Questions like these felt familiar:

What will the price of a commodity be tomorrow?

Or:

What is the probability that the central bank will cut rates at its next meeting?

For the first problem, we might use historical prices, supply and demand, inventories, and macroeconomic indicators. For the second, we might use inflation, employment, market rates, and central-bank communication. Both look like classic data-science problems.

Then I started working on actual business problems, and a different class of questions appeared. They were the questions that came after the forecast.

So how much of the commodity should we buy now?

If prices are likely to rise, how much earlier should we buy?

Given the probability of a rate cut, how much duration should the portfolio carry?

With limited budget and production capacity, which plant should make how much?

These questions are not asking, “What will happen?” They are asking, “So what should we do?”

The first is primarily a problem of prediction. The second is a problem of decision. Optimization turned out to be one of the main mathematical ways to handle that second problem.

This does not mean machine learning and optimization live in separate worlds. Training a machine-learning model is itself often an optimization problem in which we minimize a loss function. But the distinction that mattered to me at the time was the direction of the question.

Machine learning had taught me to ask, “What is likely to happen?” The optimization problems I met at work asked, “Given all these real constraints, which choice is best?”

My first encounter with optimization was not especially friendly

At first, the field felt completely unfamiliar. I asked colleagues:

“What exactly is optimization?”

I searched online. Eventually I found Convex Optimization by Stephen Boyd and Lieven Vandenberghe, a book that appears almost everywhere in the field.

Then I opened it.

Convex set. Convex function. Lagrangian. Duality.

The unfamiliar vocabulary arrived with impressive efficiency. I had only wanted to understand what optimization was, but somehow I had wandered into the middle of a mathematics textbook.

“Was this really what I was trying to learn?”

After working through practical problems and slowly becoming more comfortable with the notation, I realized something important: the idea of optimization was not unfamiliar at all.

We optimize things in ordinary life all the time. We simply do not call them objective functions, constraints, and decision variables.

Let’s plan dinner with friends

Suppose four friends in New York City agree to meet for dinner on Friday. We have not picked a restaurant yet, but that does not mean every place in the city is an option. The restaurants, prices, ratings, and travel times below are illustrative, not actual listings or route estimates.

The date and time are fixed: Friday at 7 p.m.

A restaurant that is closed on Fridays is out, however good the food is. So is one that closes at 6 p.m. We already have our first constraint.

Price matters too

Now consider the budget. For this casual dinner, let’s allow about $30 per person before tax and tip—enough for a simple meal, perhaps shared plates and a beer if the menu allows. Much more than that, and we are planning a different kind of evening.

Imagine everyone agrees on a casual dinner, and someone books a $120-per-person steakhouse instead. The steak may be excellent.

That friend would probably get roasted before anyone even left home.

So we have an implicit condition:

Expected cost per person before tax and tip ≤ $30

That is our second constraint.

Location may matter most of all

Suppose the four friends live on the Upper West Side, in Williamsburg in Brooklyn, in Lower Manhattan, and in Midtown East. The friend in Williamsburg says:

“I know an amazing place around the corner. Let’s meet there.”

The food may be amazing. The convenience is less evenly distributed. One person gets a ten-minute walk; someone else gets a long subway ride, a transfer, and another walk. The recommendation starts to look a little self-serving.

Of course, I cannot pick a restaurant five minutes from my own home and declare, “This is basically central Manhattan,” either. My friends are unlikely to accept the geography.

Fairness has a way of entering the group chat. Everyone notices who gets to walk around the corner—and who spends the evening on the subway. We want a location that does not leave anyone with an unreasonable trip. Door-to-door travel time matters more than distance on a map.

That is already an optimization problem. One possible rule is:

Minimize the total travel time of all four friends.

If fairness matters more, we might choose a different rule:

Minimize the longest individual travel time.

The two rules sound similar, but they do not necessarily select the same place. This brings us to one of the central questions in optimization:

What exactly do we mean by “good”?

The quantity we want to minimize or maximize is called the objective function.

Decision variables, objective functions, and constraints

Our dinner problem now has several conditions. A restaurant must:

• be open at 7 p.m. on Friday;
• cost no more than $30 per person before tax and tip;
• keep each person’s door-to-door trip within an agreed limit, say 45 minutes; and
• have a table available for four.

These are the constraints.

Among the restaurants that satisfy them, we need a rule for deciding which one is best. “Minimize total travel time” is one possible objective function. “Maximize a combined score for taste, price, and travel time” is another.

The thing we actually choose—whether to visit restaurant A or restaurant B—is represented by a decision variable.

An optimization problem therefore begins with three surprisingly simple questions:

1. What can we decide?
2. What do we want to make as good as possible?
3. Which conditions must be respected?

More formally, optimization means finding, among the choices that satisfy the constraints, a solution that minimizes or maximizes the objective function.

Why not just meet at the exact midpoint?

At this point, someone may ask:

“Why not calculate the exact midpoint of where everyone lives?”

We could. We could translate each address into coordinates and find a geometric center.

There is just one small problem: there may not be a restaurant there.

Depending on where people live, that point could land in the East River, inside an office building, or on a block with nowhere to sit down and eat. Subway routes do not follow straight lines, either. A mathematically perfect midpoint does not help much when the actual goal is to eat dinner.

So we usually start with a shortlist, then check which options are feasible:

• Restaurant A in Midtown
• Restaurant B near Union Square
• Restaurant C on the Lower East Side
• Restaurant D in Downtown Brooklyn

Then we compare their prices, travel times, food, and opening hours.

This is what makes real optimization interesting. Not every imaginable choice is available. There is a feasible set, there are practical constraints, and we must find the best choice inside it.

Now add taste

So far, we have focused on travel and price. But we have left out an important reason for choosing a restaurant: the food.

Suppose Restaurant B near Union Square minimizes total travel time. It wins on convenience. Unfortunately, its rating is 2.1 out of 5.

Restaurant D in Downtown Brooklyn adds twenty minutes to one friend’s trip, still within our travel-time limit, but the food is excellent. Suddenly we are negotiating:

“Would the better food be worth another twenty minutes on the train?”

The problem is now more interesting. We want to:

• reduce travel time;
• reduce price; and
• get better food.

In shorthand: Travel time ↓, Price ↓, Food quality ↑.

The perfect restaurant rarely exists. Good restaurants may be expensive. Cheap and good restaurants may be far away. Nearby restaurants may be disappointing.

We have to make a trade-off.

Would we travel ten minutes longer for better food? Pay $10 more for a much better meal, if we agree to raise the budget? Ask one friend to travel a little farther if it improves the group’s overall experience?

Optimization gives us a way to make those trade-offs explicit instead of leaving them entirely to vague intuition. It forces us to say what matters and how much it matters.

So what is linear programming?

One of the first terms we encounter in optimization is linear programming. Here, programming does not mean writing Python. It is an older use of the word that is closer to planning how scarce resources should be allocated.

The word linear means that the objective and constraints are linear expressions of the decision variables. That sounds more technical than it is. To keep the notation simple, narrow the shortlist to A, B, and C. If their overall satisfaction scores are $s_A$, $s_B$, and $s_C$, we might write:

$$\text{maximize}\quad s_Ax_A + s_Bx_B + s_Cx_C$$

If their prices are $p_A$, $p_B$, and $p_C$, all measured in dollars per person before tax and tip, and our budget is $30, we could add:

$$p_Ax_A + p_Bx_B + p_Cx_C \le 30$$

Each variable is multiplied by a fixed coefficient and the terms are added together. There are no terms such as $x_A^2$ or $x_Ax_B$. With continuous decision variables, a linear objective, and linear constraints, this is a linear program. Adding integer restrictions gives an integer linear program.

Continuous variables are natural when quantities can be divided. A company can buy 3.5 tonnes from supplier A and 6.5 tonnes from supplier B, or allocate production across several plants. In problems like those, an ordinary linear program may describe reality quite well.

Dinner is different. If we allow $0 \le x_A,x_B,x_C \le 1$ and require $x_A+x_B+x_C=1$, a linear-programming solver might return an answer like this:

Choose 30% of Restaurant A and 70% of Restaurant B.

Mathematically, that can be a perfectly sensible solution. Practically, we are trying to eat one dinner. Unless the plan is to have an appetizer at A and move to B for the main course, the answer is awkward.

The objective and constraints can still be linear, but the reality that we must choose a whole restaurant requires the variables to be 0 or 1. The relationship between linear and integer programming can therefore be summarized simply:

If a quantity can be divided, linear programming may be enough. If a choice cannot be split, we need an integer restriction.

The reservation problem makes that restriction concrete.

What if the best restaurant has no table?

After all that thought, suppose we finally identify the perfect restaurant. The price is right. The location is good. The food is excellent.

We call, and it is fully booked.

Now the model must include not only price and distance, but whether the restaurant can actually be selected. This is a simple way to see the idea behind integer programming.

Let a variable represent whether each restaurant is selected:

$$x_A = \begin{cases} 1 & \text{if restaurant A is selected} \\ 0 & \text{otherwise} \end{cases}$$

Define $x_B$ and $x_C$ in the same way. Each is a binary decision:

$$x_A, x_B, x_C \in \{0,1\}$$

We need exactly one restaurant, so:

$$x_A + x_B + x_C = 1$$

If Restaurant A is fully booked, it is unavailable:

$$x_A = 0$$

No matter how good its food or location may be, it is no longer in the feasible set.

In integer programming, some or all decision variables are constrained to take integer values. When all decision variables can take only 0 or 1, this is binary, or 0–1, integer programming.

The terminology sounds formidable. The basic idea is not:

Do we choose this restaurant? Yes = 1. No = 0.

The same structure appears in many business decisions:

• Do we operate this factory?
• Do we invest in this project?
• Do we use this distribution center?
• Do we assign this vehicle to this route?

Structurally, these decisions are not so different from choosing where to have dinner.

Is there always one optimal solution?

The word “optimal” can make it sound as though a single perfect answer is hidden somewhere in the universe. That is not always true.

If Restaurants A and B have exactly the same objective value, both can be optimal. The problem becomes more complex when we care about several objectives at once. The cheapest, tastiest, and closest restaurants may all be different.

Ideas such as multi-objective optimization and Pareto optimality help us reason about these competing goals. We will leave those for a later article.

For now, one idea is enough:

Optimization is not necessarily a machine for producing one magical answer. It is a way to define what matters and find the best available choice under that definition.

Prediction and optimization belong in the same pipeline

Return to the data-science questions from the beginning. A model may estimate tomorrow’s commodity price or the probability of a central-bank rate cut.

But a company rarely stops there. The consequential questions come next:

How much of the commodity should we buy today?

How much inventory should we hold against the risk of higher prices?

How should the portfolio’s interest-rate exposure change given the probability of a cut?

Prediction provides information about the world. Optimization can use that information to decide what to do. In practical systems, they often form one pipeline:

Data
  ↓
Prediction / Estimation
  ↓
Optimization
  ↓
Decision

A forecast becomes an input to an optimization model, and the optimization model proposes an action. That is why machine learning and optimization should not be treated as disconnected subjects.

Optimization is the mathematics of choice

It took me longer than expected to understand optimization because I encountered the intimidating vocabulary first.

Linear Programming. Integer Programming. Convex Optimization. Lagrangian. Duality.

At first, these terms made optimization look like a highly specialized and forbidding corner of mathematics. At a deeper level, it certainly can be difficult. But the problems it tries to solve are remarkably ordinary.

We want things, and reality imposes limits. Money is finite. Time is finite. People, factory capacity, storage, vehicles, and budgets are all finite. We still have to choose.

Where should friends meet for dinner? How much should a company buy from each supplier? Which plant should produce how much? In what order should delivery vehicles visit customers? How much capital should be allocated to each asset?

These problems look unrelated, but their underlying structure is similar.

There is something to decide, something to improve, and a set of conditions that must be respected.

In formal language, this is mathematical optimization. Intuitively, I prefer to think of it this way:

Optimization is the mathematics of finding the best choice in a constrained world.

Once I saw it that way, the equations in Convex Optimization began to look different. The equations were not the starting point. They were a language for expressing, clearly and precisely, the choices we make every day.

Next: when no restaurant is perfect for everyone — Pareto optimization

Our dinner plan gives us an intuitive starting point for optimization. But one question remains:

If travel time, price, taste, and fairness cannot all be improved at once, what should count as optimal?

The next article will use that question to introduce Pareto optimization. We will compare whether one restaurant is better than another on every criterion and identify the point at which improving one objective necessarily makes another worse.

• What is Pareto dominance?
• Why can there be more than one Pareto-optimal solution?
• How does the Pareto frontier display the available trade-offs?
• Whose preferences determine the final choice among those candidates?

We probably will not find a restaurant that makes everyone perfectly happy. We should, however, be able to say much more clearly whose satisfaction must be sacrificed to reduce someone else’s travel time.

The problem is still Friday dinner. This time, instead of asking only “Which restaurant is best?”, we will ask a more realistic question:

When several objectives conflict, which choices deserve to remain serious candidates?