IIT-JEE 2010 · Paper 1, Mathematics question 38

Cosine rule with $C=\frac\pi6$: $a^2+b^2-c^2=\sqrt3ab$.

Factorise. $a^2-c^2=(x^2+x+1-2x-1)(x^2+x+1+2x+1)=x(x-1)(x+1)(x+2)$, and $b^2=(x-1)^2(x+1)^2$, so
$$a^2+b^2-c^2=(x-1)(x+1)\left[x(x+2)+(x-1)(x+1)\right]=(x-1)(x+1)(2x^2+2x-1).$$
Also $\sqrt3ab=\sqrt3(x^2+x+1)(x-1)(x+1)$.

Solve. $b\ne0$, so divide by $(x-1)(x+1)$:
$$2x^2+2x-1=\sqrt3(x^2+x+1)\implies x^2+x=\frac{1+\sqrt3}{2-\sqrt3}=5+3\sqrt3.$$
The roots are $x=1+\sqrt3$ and $x=-(2+\sqrt3)$.

Valid triangle. For $x=-(2+\sqrt3)$, $c=2x+1<0$, impossible. For $x=1+\sqrt3$ all sides are positive and satisfy the triangle inequality.

(B)