z = 5, λ = 2
S = 5 − 2
3.
The soft-thresholding function shrinks towards 0 by , and sets it to 0 if . It is the solution of .
This page is easier if you are comfortable with:
Here . The minimiser of is found by cases: for the derivative gives if ; similarly for ; otherwise the minimum is at 0 where has a corner. This is the idea of a modulus function penalty, whose corner makes exact zeros possible, unlike the squared penalty of ridge regression, which only shrinks.
With : , , . In LASSO regression (an -penalised fit, a variation of least squares), the coefficients are soft-thresholded, so small ones become exactly zero and the model is sparse. Its derivative is not defined at the corner, so the solution is found by the cases above rather than setting the derivative to zero.
The operator is the proximal operator of the norm and is the basic step of algorithms such as ISTA, which combine a gradient step with soft-thresholding. It is related to the bias–variance trade-off in the bias–variance decomposition; compare the triangle inequality for the norm used.
Soft-thresholding shrinks every value towards zero and removes the small ones entirely, which gives sparse solutions.
The output is flat at 0 near the origin and runs parallel to the line y = z outside.
These examples are worked by hand, step by step, so you can compare your own working.
Find .
Find .
Find .
A few situations come up again and again. Knowing them saves time and helps you spot answers that cannot be right.
S = 5 − 2
3.
S = −(4 − 1)
−3.
|z| ≤ λ
0.
boundary
0.
S_λ(z) = sign(z)(|z| − λ)₊: shrink towards 0 by λ, and set to 0 if |z| ≤ λ.
Shrink.
Keep the sign.
Inside the threshold.
Exactly on the threshold.
−4 − 1 = −5 ✗
−4 + 1 = −3 ✓
Toward 0.
negative ✗
0 ✓
Positive part.
same ✗
soft shrinks all ✓
Different.
wrong ✗
λ ✓
Definition.
Try each one on paper, then open the answer. Use the solver above to see the full working.
4
-4
0
0
1.5
0
Unlimited fresh questions with instant marking. Choose a level, work it out on paper, then type your answer.
Solve: …
S_λ(z) = sign(z)(|z| − λ)₊.
Sets them to 0.
In lasso regression.
It gives exact zeros; see ridge regression.
gradient descent.
bias–variance decomposition.
More in Numerical methods and optimisation and the formula sheets.