As in the hint on the web site, I’d like you to define f(t) = | A.(cos(t), sin(t) )|.  Since the Euclidean norm | * | involves a square root it’s convenient to define f2(t) = f(t)^2

f2[t_] := (Cos[t] + 2 Sin[t])^2 + Sin[t]^2

The operator norm of A is the maximum value of the function plotted below, about 2.4.

Plot[Sqrt[f2[t]], {t, 0, 2 Pi}, PlotRange ->{0, 2.5}]

Following the book we get the operator norm squared of A is the maximum eigenvalue of A^T A.  I suggested you not do the calculus to get the maximum of f2, since it get’s messy, but here it is with Mathematica:

Simplify[f2'[t]]

I use a trig identity that f2(t) = 2 sqrt(2) sin(2t + pi/4).  To be honest, I found the phase shift of pi/4 experimentally by playing with the next cell

Simplify[f2'[t] - 4 Sqrt[2] Sin[2t + Pi/4]]

So the critical points of f are the values of t such that 2t + Pi/4 = n Pi, or t = (n-1/4)Pi/2 for all integers n, or shifting n we get t = (n + 3/4)Pi/2.  The two smallest critical points of f2 are t = 3Pi/8 (n = 0) and t = 7 Pi/8 (n = 1).
There are some half angle formulas that allow us to evaluate f2(3 Pi/8) and f2(7Pi/8) exactly, but let’s let the computer do it.

Simplify[f[3 Pi/8]]

Simplify[f[7 Pi/8]]

Plot[{Sqrt[f2[t]], Sqrt[3 + 2Sqrt[2]], Sqrt[3 - 2Sqrt[2]]}, {t, 0, 2 Pi}, PlotRange ->{0, 2.5}]

We can get a decimal approximation for the operator norm of A by replacing the integer “3” with the floating point number “3.”

Sqrt[3.+2Sqrt[2]]

Note:  The image of the unit circle under A is an ellipse with semi-major axis 3+2 (Sqrt[2]) and semi-minor axis 3-2 (Sqrt[2]) .  That semi-major axis is the operator norm of A.  You can look at the Mathematica notebook A times Ball that is linked to on our homework page.