JEE Main 2015 · 11 Apr · Shift 1, Mathematics question 85

$\vec{DB}=\vec{AB}-\vec{AD}$, so $\vec{DB}\cdot\vec{AB}=a^2-\vec{AB}\cdot\vec{AD}$.

The keyed answer, $\frac12(a^2-b^2+c^2)$, comes out when $c$ is the diagonal $|\vec{DB}|$: then $c^2=a^2+b^2-2\vec{AB}\cdot\vec{AD}$, and
$$\vec{DB}\cdot\vec{AB}=a^2-\frac{a^2+b^2-c^2}2=\frac12(a^2-b^2+c^2).$$
With $c=|\vec{AC}|$ as printed, the value would be $\frac12(3a^2+b^2-c^2)$, which is not an option.