JEE Advanced 2023 · Paper 2, Mathematics question 14

Setting up the angles. Let $A>B>C$, with $A$ obtuse and $A-C=\frac\pi2$. Then
$$A=C+\tfrac\pi2,\qquad B=\tfrac\pi2-2C.$$

AP condition. $2b=a+c$ with $a=2\sin A$ etc. ($R=1$):
$$2\cos2C=\cos C+\sin C\implies\cos C-\sin C=\tfrac12.$$
Squaring: $\sin2C=\frac34$, so $\cos C+\sin C=\frac{\sqrt7}2$ and
$$\cos2C=(\cos C-\sin C)(\cos C+\sin C)=\frac{\sqrt7}4.$$

Area.
$$a=2R^2\sin A\sin B\sin C=2\cos C\cos2C\sin C=\sin2C\cos2C=\frac34\cdot\frac{\sqrt7}4=\frac{3\sqrt7}{16}.$$

$$(64a)^2=(12\sqrt7)^2=1008.$$