Soft-thresholding Explained with Worked Examples

The soft-thresholding function shrinks towards 0 by , and sets it to 0 if . It is the solution of .

Write 2x or 2*x to multiply, ^ for powers, and ; between simultaneous equations. You can also type factor, expand, derivative of, integrate … from a to b, limit … as x->0 or 15% of 80.

Try an example

Before you start

This page is easier if you are comfortable with:

What Soft-thresholding (LASSO) means

Here . The minimiser of is found by cases: for the derivative gives if ; similarly for ; otherwise the minimum is at 0 where has a corner. This is the idea of a modulus function penalty, whose corner makes exact zeros possible, unlike the squared penalty of ridge regression, which only shrinks.

With : , , . In LASSO regression (an -penalised fit, a variation of least squares), the coefficients are soft-thresholded, so small ones become exactly zero and the model is sparse. Its derivative is not defined at the corner, so the solution is found by the cases above rather than setting the derivative to zero.

The operator is the proximal operator of the norm and is the basic step of algorithms such as ISTA, which combine a gradient step with soft-thresholding. It is related to the bias–variance trade-off in the bias–variance decomposition; compare the triangle inequality for the norm used.

What is it used for?

Soft-thresholding shrinks every value towards zero and removes the small ones entirely, which gives sparse solutions.

How to use soft-thresholding (lasso)

  1. Choose the threshold and the value .
  2. If , the result is 0.
  3. If , the result is ; if , it is .
  4. Apply to each coefficient separately.

Graph

S₁(z) is 0 for |z| ≤ 1 and moves other values 1 closer to 0S1(3) = 2S1(0.4) = 0-4-3-2-101234-3-2-10123S₁(z) is 0 for |z| ≤ 1 and moves other values 1 closer to 0zS1(z)soft-threshold S1(z)y = zGraph with x from -4 to 4 and y from -3 to 3. Shows the curve soft-threshold S1(z); the curve y = z; point S1(3) = 2 at (3, 2); point S1(0.4) = 0 at (0.4, 0).

The output is flat at 0 near the origin and runs parallel to the line y = z outside.

Worked examples

These examples are worked by hand, step by step, so you can compare your own working.

Example 1: shrink a large value

Find .

Worked example
Answer
22
  1. |3| > 1

Example 2: a negative value

Find .

Worked example
Answer
−1.5-1.5
  1. |−2.5| > 1

Example 3: a small value

Find .

Worked example
Answer
00
  1. |0.4| ≤ 1

Special cases

A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.

z = 5, λ = 2

S = 5 − 2

3.

z = −4, λ = 1

S = −(4 − 1)

−3.

z = 1.5, λ = 2

|z| ≤ λ

0.

z = 3, λ = 3

boundary

0.

Word problems

S_λ(z) = sign(z)(|z| − λ)₊: shrink towards 0 by λ, and set to 0 if |z| ≤ λ.

z = 5, λ = 2

Shrink.

  1. 5 − 2
  2. Check: ✓3

z = −4, λ = 1

Keep the sign.

  1. −(4 − 1)
  2. Check: ✓-3

z = 1.5, λ = 2 Higher

Inside the threshold.

  1. 0
  2. Check: ✓0

z = 3, λ = 3 Higher

Exactly on the threshold.

  1. 3 − 3
  2. Check: ✓0

Common mistakes

Subtracting λ from negative z

−4 − 1 = −5 ✗

−4 + 1 = −3 ✓

Toward 0.

Not zeroing small values

negative ✗

0 ✓

Positive part.

Same as hard thresholding

same ✗

soft shrinks all ✓

Different.

Shrinking by λ² or 2λ

wrong ✗

λ ✓

Definition.

Exam tips

  1. |z| ≤ λ ⇒ 0.
  2. Otherwise shrink by λ.
  3. Keep the sign.
  4. Lasso uses it.
  5. Draw the graph.

Practice questions

Try each one on paper, then open the answer. Use the solver above to see the full working.

1. S₃(7) Foundation

4

2. S₂(−6) Foundation

-4

3. S₄(3) Foundation

0

4. S₁(−0.5) Higher

0

5. S₀.₅(2) Higher

1.5

6. S₅(−5) Higher

0

Practice generator

Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.

Solve: …

Key terms

Soft thresholding
Shrinkage operator.
Threshold (λ)
Cut-off.
Lasso
L1 regression.
Sparsity
Many zeros.
Positive part
(t)₊ = max(t, 0).
Shrinkage
Moving toward 0.

Frequently asked questions

What is soft thresholding?

S_λ(z) = sign(z)(|z| − λ)₊.

What does it do to small values?

Sets them to 0.

Where is it used?

In lasso regression.

How does it differ from ridge?

It gives exact zeros; see ridge regression.

Which page covers gradient descent?

gradient descent.

Which page covers bias and variance?

bias–variance decomposition.

More in Numerical methods and optimisation and the formula sheets.