Let $f(x,y)=x^{2}+4xy+y^{2}.$
(a) Show that the origin is a critical point of $f.$
(b) Show that $f>0$ at every point of both axes except the origin.
(c) Show, with Definition 11.2, that the origin is nevertheless a saddle point.
Hint
For (c), look along the two diagonals $y=x$ and $y=-x.$ You need points with $f>0$ and points with $f<0$ in every disc around the origin.
(a) Critical. $f'_{1}=2x+4y$ and $f'_{2}=4x+2y,$ both $0$ at $(0,0).$
(b) The axes. $f(x,0)=x^{2}>0$ for $x\ne0$ and $f(0,y)=y^{2}>0$ for $y\ne0.$ Along both axes the origin looks like a minimum.
(c) The diagonals. On $y=x{:}$ $f(x,x)=x^{2}+4x^{2}+x^{2}=6x^{2}>0$ for $x\ne0.$ On $y=-x{:}$ $$ \begin{aligned} f(x,-x)&=x^{2}-4x^{2}+x^{2}\\ &=-2x^{2}<0 \end{aligned} $$ for $x\ne0.$ Both lines pass through every disc around the origin, so every such disc contains points where $f$ is larger than $f(0,0)=0$ and points where it is smaller: a saddle point.
What it shows. Two directions (the axes) agreed on "minimum", and a third direction disagreed. That is why the test of the next section needs $B=f''_{12}$ as well as $A$ and $C.$
The origin is critical; $f>0$ on the axes away from it; but $f(x,-x)=-2x^{2}<0,$ so it is a saddle point.