Lagrange Multipliers, multivariable and vector calculus lesson

Picture a hiking trail drawn on a hilly map. You want the highest point you reach while staying on the trail, which is usually not the top of the hill. At that point the trail runs along a contour line instead of crossing it, and that simple picture is the whole idea behind Lagrange multipliers.

Key terms

Constrained extrema

Definition Let $f$ be defined on a set containing $S$ , where $S$ is the set of points that satisfy one or more equations, the constraints. The function $f$ has a local maximum on $S$ at a point $\mathbf{a}$ of $S$ if there is a ball $B$ centered at $\mathbf{a}$ with $f(\mathbf{x}) \le f(\mathbf{a})$ for every $\mathbf{x}$ in $S \cap B$ , and an absolute maximum on $S$ if $f(\mathbf{x}) \le f(\mathbf{a})$ for every $\mathbf{x}$ in $S$ . Minima are defined with the inequalities reversed. Only points of $S$ are compared.

In plain words You are allowed to move only along the constraint. A constrained maximum is the best you can do there, even if $f$ is larger elsewhere.

Example Maximize $f(x, y) = xy$ on the line $x + y = 10$ . Substituting $y = 10 - x$ gives $10x - x^2$ , which is largest at $x = 5$ . The constrained maximum is $f(5, 5) = 25$ , although $xy$ has no maximum at all on the whole plane. Substitution works here because the constraint is easy to solve, which is often not the case.

Parallel gradients

Definition Let $f$ and $g$ have continuous first partial derivatives on an open set $U$ in $\mathbb{R}^n$ , and let $S = \{\mathbf{x} \in U : g(\mathbf{x}) = c\}$ . If $f$ restricted to $S$ has a local extremum at $\mathbf{a}$ and $\nabla g(\mathbf{a}) \ne \mathbf{0}$ , then there is a number $\lambda$ , called a Lagrange multiplier, with

\nabla f(\mathbf{a}) = \lambda \nabla g(\mathbf{a}).

So candidates are found by solving the $n + 1$ equations $\nabla f = \lambda \nabla g$ and $g = c$ for the $n + 1$ unknowns $x_1, \dots, x_n, \lambda$ . The multiplier may be zero, which happens when $\nabla f(\mathbf{a}) = \mathbf{0}$ .

Here is why the condition holds. Take any $C^1$ curve $\mathbf{r}(t)$ in $S$ with $\mathbf{r}(0) = \mathbf{a}$ . Differentiating $g(\mathbf{r}(t)) = c$ at $t = 0$ gives $\nabla g(\mathbf{a}) \cdot \mathbf{r}'(0) = 0$ , and since $f(\mathbf{r}(t))$ has a local extremum at $t = 0$ , also $\nabla f(\mathbf{a}) \cdot \mathbf{r}'(0) = 0$ . Because $\nabla g(\mathbf{a}) \ne \mathbf{0}$ , the implicit function theorem shows that such curves reach every vector orthogonal to $\nabla g(\mathbf{a})$ as a velocity $\mathbf{r}'(0)$ . So $\nabla f(\mathbf{a})$ is orthogonal to everything that $\nabla g(\mathbf{a})$ is orthogonal to, and that forces it to be a multiple of $\nabla g(\mathbf{a})$ .

In plain words At a constrained high or low point, moving along the constraint does not change $f$ to first order. So $\nabla f$ has no component along the constraint and must point straight across it, in the same direction as $\nabla g$ or the opposite one.

Example Find the extreme values of $f(x, y) = x + 2y$ on the circle $x^2 + y^2 = 5$ . Here $\nabla g = \langle 2x, 2y \rangle$ is nonzero on the circle, and the system is

\begin{gathered} 1 = 2\lambda x, \qquad 2 = 2\lambda y, \\[0.4em] x^2 + y^2 = 5. \end{gathered}

The first equation shows $\lambda \ne 0$ and $x \ne 0$ , so dividing is allowed and $y = 2x$ . Then $5x^2 = 5$ , which gives the points $(1, 2)$ and $(-1, -2)$ with values $5$ and $-5$ . The circle is closed and bounded and $f$ is continuous, so the maximum is $5$ and the minimum is $-5$ .

Several equality constraints

Definition Let $f, g_1, \dots, g_m$ with $m < n$ have continuous first partial derivatives on an open set $U$ in $\mathbb{R}^n$ , and let $S$ be the set of points of $U$ with $g_1 = c_1, \dots, g_m = c_m$ . If $f$ restricted to $S$ has a local extremum at $\mathbf{a}$ and the vectors $\nabla g_1(\mathbf{a}), \dots, \nabla g_m(\mathbf{a})$ are linearly independent, then there are numbers $\lambda_1, \dots, \lambda_m$ with

\begin{aligned} \nabla f(\mathbf{a}) &= \lambda_1 \nabla g_1(\mathbf{a}) + \dots \\[0.4em] &\quad + \lambda_m \nabla g_m(\mathbf{a}). \end{aligned}

Together with the $m$ constraints this gives $n + m$ equations for $n + m$ unknowns.

In plain words With two constraints in space you move along a curve, the intersection of two surfaces. At an extremum, $\nabla f$ must lie in the plane spanned by the two constraint gradients.

Example Find the highest and lowest points on the ellipse where the cylinder $x^2 + y^2 = 2$ meets the plane $x + y + z = 1$ , so $f = z$ . The system is

\begin{aligned} \langle 0, 0, 1 \rangle &= \lambda \langle 2x, 2y, 0 \rangle + \mu \langle 1, 1, 1 \rangle, \\[0.4em] x^2 + y^2 &= 2, \qquad x + y + z = 1. \end{aligned}

The third component gives $\mu = 1$ , and then $2\lambda x = -1$ and $2\lambda y = -1$ . So $\lambda \ne 0$ and $x = y$ , hence $x = y = \pm 1$ . The points are $(1, 1, -1)$ and $(-1, -1, 3)$ . The ellipse is closed and bounded, so the highest value of $z$ is $3$ and the lowest is $-1$ .

Independent constraint gradients

Definition A point $\mathbf{a}$ of the constraint set is regular when $\nabla g_1(\mathbf{a}), \dots, \nabla g_m(\mathbf{a})$ are linearly independent. For one constraint this means $\nabla g(\mathbf{a}) \ne \mathbf{0}$ . For two constraints in $\mathbb{R}^3$ it means

\nabla g_1(\mathbf{a}) \times \nabla g_2(\mathbf{a}) \ne \mathbf{0}.

By the implicit function theorem, near a regular point the constraint set is smooth of dimension $n - m$ (a curve, a surface or a higher dimensional analogue), and its tangent directions are exactly the vectors orthogonal to every $\nabla g_i(\mathbf{a})$ . This is the hypothesis that makes the multiplier theorem true.

In plain words Independence makes sure the constraints really cut the space down by one dimension each, so the set you move on has a well defined tangent plane or tangent line. Always check it before trusting the multiplier equations.

Example In the cylinder and plane problem,

\begin{aligned} &\langle 2x, 2y, 0 \rangle \times \langle 1, 1, 1 \rangle \\[0.4em] &\quad = \langle 2y, -2x, 2x - 2y \rangle. \end{aligned}

This is zero only when $x = y = 0$ , and no such point lies on the cylinder $x^2 + y^2 = 2$ . So every point of the ellipse is regular and the two candidates found above are the only possible extrema.

Singular constraint points

Definition A point of the constraint set where the constraint gradients are linearly dependent, for one constraint a point where $\nabla g = \mathbf{0}$ , is a singular point of the constraint. The multiplier theorem says nothing there, so an extremum may occur at a singular point without any $\lambda$ satisfying the Lagrange equations. Singular points must be added to the list of candidates, together with any points where $f$ or $g$ fails to have continuous partial derivatives.

In plain words Where the constraint curve has a cusp or crosses itself, the picture of a smooth trail breaks down. Check those points by hand.

Example Minimize $f(x, y) = x$ on the cusp curve $y^2 = x^3$ , so $g = y^2 - x^3$ . Both functions have continuous partial derivatives everywhere, so the only extra candidates are singular points. The equations $1 = -3\lambda x^2$ and $0 = 2\lambda y$ force $\lambda \ne 0$ , so $y = 0$ , then $x = 0$ from the curve, and the first equation becomes $1 = 0$ . There are no Lagrange solutions at all. Yet every point of the curve has $x^3 = y^2 \ge 0$ , hence $x \ge 0$ , so the minimum is $0$ , reached only at the origin, where $\nabla g = \langle -3x^2, 2y \rangle = \langle 0, 0 \rangle$ . The origin is a singular point, and the extremum sits exactly there. There is no maximum, since $x$ is unbounded on the curve.

Feasible boundaries

Definition The feasible set is the set of all points that satisfy every constraint. If some constraints are inequalities, such as $x \ge 0$ or $G(x, y) \le c$ , the feasible set can have interior points, smooth boundary pieces and points where those pieces meet. The full candidate list then contains the critical points of $f$ in the interior, the Lagrange points on each smooth boundary piece given by an equation with nonzero gradient, and the endpoints, corners and singular points of the boundary.

In plain words Lagrange multipliers handle the smooth parts of an edge. The inside of the region and the places where edges meet still need their own checks.

Example Find the extreme values of $f(x, y) = x^2 + 2y^2$ on the disk $x^2 + y^2 \le 1$ . Inside, $\nabla f = \langle 2x, 4y \rangle$ is zero only at $(0, 0)$ , where $f = 0$ . On the circle $g = x^2 + y^2 = 1$ , with $\nabla g \ne \mathbf{0}$ , the equations are

2x = 2\lambda x, \qquad 4y = 2\lambda y.

If $x \ne 0$ then $\lambda = 1$ , so $y = 0$ and $x = \pm 1$ , with $f = 1$ . If $x = 0$ then $y = \pm 1$ , with $f = 2$ . Keeping the case $x = 0$ instead of dividing by $x$ is what finds these points. The maximum is $2$ at $(0, \pm 1)$ and the minimum is $0$ at the origin.

Candidate verification

Definition The Lagrange conditions are necessary, not sufficient. A point that satisfies them may be a maximum, a minimum or neither. To finish a problem, collect all candidates (Lagrange points, singular points, interior critical points and boundary endpoints or corners) and then justify which is which. If the feasible set is closed and bounded and $f$ is continuous there, the extreme value theorem guarantees an absolute maximum and minimum, and comparing the values of $f$ at the candidates identifies them. If the feasible set is not compact, existence must be argued separately, for instance with an inequality or the behavior of $f$ far away.

In plain words The method hands you suspects, not verdicts. Compactness lets you decide by comparing numbers, and without it you need a reason.

Example Look at $f(x, y) = x + y$ on the hyperbola $xy = 1$ . The equations $1 = \lambda y$ and $1 = \lambda x$ give $x = y$ , so the candidates are $(1, 1)$ with $f = 2$ and $(-1, -1)$ with $f = -2$ . The hyperbola is not bounded. On the branch with $x > 0$ the values are $x + \tfrac{1}{x} \ge 2$ , and on the branch with $x < 0$ they are at most $-2$ and go to $-\infty$ . So $(1, 1)$ is a local minimum on the constraint and $(-1, -1)$ a local maximum, and $f$ has no absolute maximum or minimum. The local maximum value is even smaller than the local minimum value.

Check yourself

Find the extreme values of $f(x, y) = xy$ on the ellipse $x^2 + 4y^2 = 8$ .

Show the answer

The constraint gradient $\langle 2x, 8y \rangle$ is nonzero on the ellipse. The equations are $y = 2\lambda x$ and $x = 8\lambda y$ . If $y = 0$ then $x = 0$ , which is not on the ellipse, so $y \ne 0$ and $y = 16\lambda^2 y$ gives $\lambda = \pm \tfrac{1}{4}$ , that is $y = \pm \tfrac{x}{2}$ . Then $2x^2 = 8$ , so $x = \pm 2$ . The maximum is $2$ at $(2, 1)$ and $(-2, -1)$ , and the minimum is $-2$ at $(2, -1)$ and $(-2, 1)$ , since the ellipse is closed and bounded.

Find the minimum of $x^2 + y^2$ on the hyperbola $xy = 1$ and justify that it is a minimum.

Show the answer

The constraint gradient $\langle y, x \rangle$ is never zero on the hyperbola. The equations $2x = \lambda y$ and $2y = \lambda x$ give $2x^2 = \lambda xy = 2y^2$ , so $x^2 = y^2$ , and with $xy = 1$ this means $x = y = \pm 1$ . Both points give the value $2$ . The hyperbola is not bounded, so comparing values is not enough. The inequality $(x - y)^2 \ge 0$ gives $x^2 + y^2 \ge 2xy = 2$ for every point of the hyperbola, so $2$ is the minimum. There is no maximum.

Find the extreme values of $f(x, y) = 2x + y$ on the quarter circle $x^2 + y^2 = 1$ with $x \ge 0$ and $y \ge 0$ .

Show the answer

On the open arc the constraint gradient $\langle 2x, 2y \rangle$ is nonzero, and $\langle 2, 1 \rangle = \lambda \langle 2x, 2y \rangle$ gives $x = 2y$ . Of the two solutions on the circle only the one with $x, y > 0$ lies on the arc, so the point is $\left(\tfrac{2}{\sqrt{5}}, \tfrac{1}{\sqrt{5}}\right)$ with $f = \sqrt{5}$ . The arc has endpoints $(1, 0)$ and $(0, 1)$ , with $f = 2$ and $f = 1$ . The arc is closed and bounded, so the maximum is $\sqrt{5}$ and the minimum is $1$ at the endpoint $(0, 1)$ , a point the multiplier equations alone would miss.

Lagrange Multipliers

Key terms

Constrained extrema

Parallel gradients

Several equality constraints

Independent constraint gradients

Singular constraint points

Feasible boundaries

Candidate verification

Check yourself

How this lesson connects