Derivatives, Slopes, and Gradient Descent

Calculus can initially feel like a collection of complicated formulas. But at the heart of many machine learning algorithms is one simple idea:

A derivative tells us how a function is changing at a particular point.

Once we understand derivatives, slopes, gradients, and optimization become much easier to connect.

1. What Is a Derivative?

Suppose we have a function:

y=f(x)

A derivative tells us how much the output $y$ changes when the input $x$ changes by a very small amount.

In other words:

The derivative measures the instantaneous rate of change of a function.

We write the derivative as:

f'(x)

or:

\frac{dy}{dx}

A simple example

Consider:

f(x)=x^2

At $x=2$ :

f(2)=4

Now slightly increase $x$ to $2.01$ :

f(2.01)=4.0401

The change in $x$ is:

\Delta x=0.01

The change in $y$ is:

\Delta y=4.0401-4=0.0401

Therefore, the approximate rate of change is:

\frac{\Delta y}{\Delta x} = \frac{0.0401}{0.01} = 4.01

As the change in $x$ becomes smaller and smaller, this value approaches $4$ .

Therefore:

f'(2)=4

This means that at $x=2$ , the function is changing at a rate of $4$ .

2. Where Does the Derivative Formula Come From?

To understand the derivative, first think about the slope between two points.

Suppose we have two points on the curve:

P=(a,f(a))

and:

Q=(a+h,f(a+h))

The slope between these two points is:

\frac{f(a+h)-f(a)}{(a+h)-a}

which simplifies to:

\frac{f(a+h)-f(a)}{h}

This is called the difference quotient.

But this only gives us the slope between two points.

What if we want the slope at exactly $x=a$ ?

We move point $Q$ closer and closer to $P$ by making $h$ approach zero:

h\rightarrow0

The slope between the two points then approaches the slope of the curve at exactly $a$ .

Therefore, the derivative is defined as:

\boxed{ f'(a)= \lim_{h\rightarrow0} \frac{f(a+h)-f(a)}{h} }

This is the mathematical foundation of the derivative.

3. The Derivative Is the Slope of a Curve

For a straight line, the slope is constant.

For example:

y=2x+1

has a slope of $2$ everywhere.

But a curve does not have the same slope everywhere.

Consider:

f(x)=x^2

Its derivative is:

f'(x)=2x

Therefore:

f'(1)=2

f'(2)=4

f'(-1)=-2

The slope changes depending on where we are on the curve.

Geometrically, the derivative gives us the slope of the tangent line at a particular point.

So we can think of a derivative in three equivalent ways:

Rate of change
Slope of the tangent line
Instantaneous change

4. What Does the Sign of a Derivative Tell Us?

The derivative also tells us which direction a function is moving.

Positive derivative

If:

f'(x)>0

the function is increasing.

Negative derivative

If:

f'(x)<0

the function is decreasing.

Zero derivative

If:

f'(x)=0

the function is momentarily flat.

This last case becomes particularly important when we study optimization.

5. From Derivatives to Optimization

Optimization is about finding the best possible value.

In machine learning, we commonly want to minimize a loss function.

Suppose our loss is:

L(w)

where $w$ represents a model parameter.

Our goal is:

\text{minimize }L(w)

Imagine the loss function as a landscape:

Loss
 ^
 |        /\
 |       /  \
 |      /    \
 |_____/      \____
              ↑
          minimum

We want to find the bottom of the valley.

This is where gradient descent comes in.

6. What Is Gradient Descent?

Gradient descent is an optimization algorithm that repeatedly moves toward a direction that reduces the function’s value.

For a single parameter $w$ , the update rule is:

\boxed{ w_{\text{new}} = w_{\text{old}} - \alpha L'(w) }

where:

$w$ is the parameter
$L'(w)$ is the derivative of the loss
$\alpha$ is the learning rate

The derivative tells us the direction in which the loss is increasing.

Therefore, we move in the opposite direction to decrease the loss.

That is why we subtract the derivative.

7. Why Do We Subtract the Derivative?

Suppose:

L'(w)>0

The loss is increasing as $w$ increases.

So we want to move toward smaller values of $w$ .

The update:

w_{\text{new}} = w_{\text{old}} - \alpha L'(w)

moves $w$ to the left.

On the other hand, if:

L'(w)<0

the loss is decreasing as $w$ increases.

Subtracting a negative value causes $w$ to increase:

w_{\text{new}} = w_{\text{old}} - \alpha(-|L'(w)|)

So we move to the right.

In both cases, we are moving downhill.

8. What Is the Learning Rate?

The learning rate, represented by $\alpha$ , controls how large each step is.

w_{\text{new}} = w_{\text{old}} - \alpha L'(w)

If $\alpha$ is too small:

The algorithm takes very small steps and learning can be extremely slow.

If $\alpha$ is too large:

The algorithm may jump over the minimum and fail to converge.

A good learning rate allows us to gradually move toward the minimum.

9. Local Minimum

A local minimum is a point that is lower than the points immediately surrounding it.

Think of it as the bottom of a small valley:

       \       /
        \     /
         \___/
           ↑
      local minimum

For example:

f(x)=x^2

has a minimum at:

x=0

because:

f(0)=0

while:

f(-1)=1

and:

f(1)=1

The derivative is:

f'(x)=2x

Therefore:

f'(0)=0

At the bottom of the valley, the slope is flat.

10. Local Maximum

A local maximum is the opposite of a local minimum.

It is a point that is higher than the points immediately surrounding it.

Think of it as the top of a hill:

         /\
        /  \
       /    \
      /      \
          ↑
     local maximum

For example:

f(x)=-x^2

has a maximum at:

x=0

Its derivative is:

f'(x)=-2x

Therefore:

f'(0)=0

Again, the slope is zero at the top.

11. Does Derivative = 0 Always Mean a Minimum?

No.

This is an important point.

A derivative of zero only tells us that the function is flat at that point.

Consider:

f(x)=x^3

Its derivative is:

f'(x)=3x^2

At $x=0$ :

f'(0)=0

But $x=0$ is neither a local minimum nor a local maximum.

The function simply becomes flat momentarily and continues increasing.

Therefore:

A zero derivative is a candidate for a minimum or maximum, but it does not guarantee one.

12. From Derivative to Gradient

So far, we have considered functions with one variable.

Machine-learning models usually have many parameters:

w_1,w_2,\ldots,w_n

Our loss function might therefore look like:

L(w_1,w_2,\ldots,w_n)

We calculate the derivative with respect to each parameter.

These partial derivatives are collected into a vector called the gradient:

\boxed{ \nabla L= \begin{bmatrix} \frac{\partial L}{\partial w_1}\\ \frac{\partial L}{\partial w_2}\\ \vdots\\ \frac{\partial L}{\partial w_n} \end{bmatrix} }

The gradient points in the direction of steepest increase of the function.

Therefore, the negative gradient points toward the direction of steepest decrease.

Gradient descent uses this idea:

\boxed{ \mathbf{w}_{\text{new}} = \mathbf{w}_{\text{old}} - \alpha\nabla L }

13. The Big Picture

All these concepts are connected:

\boxed{\text{Derivative}}

Tells us how a function is changing.

\boxed{\text{Slope}}

Tells us the direction and rate of change at a point.

\boxed{\text{Gradient}}

Extends this idea to functions with multiple variables.

\boxed{\text{Negative Gradient}}

Points toward decreasing values.

\boxed{\text{Gradient Descent}}

Repeatedly moves in the direction that decreases the loss.

\boxed{\text{Local Minimum}}

A point where the function is lower than its nearby points.

Final Mental Model

Imagine a ball sitting somewhere on a mountain landscape.

The gradient tells you which direction is uphill.

If your goal is to minimize the height, you go in the opposite direction.

You take a step:

\text{new position} = \text{old position} - \text{learning rate}\times\text{gradient}

Then you calculate the gradient again and take another step.

You repeat this process until you approach a minimum.

That simple idea — measure the slope, move downhill, and repeat — is one of the fundamental mathematical ideas behind how machine-learning models learn.

Share this article