Find the $\lim_{x\to\infty}\frac{(1+1/x)^{x^2}}{e^x}$

Find the $$\lim_{x\to\infty}\frac{\left(1+\dfrac{1}{x}\right)^{x^2}}{e^x}.$$

I know that the result is equal to $e^{-1/2}$, but I am curious why I can't substitute one of the $x$ outside result in $\left[\left(1 + \dfrac{1}{x}\right)^x \right]^x$, and the inner part will result in $e$ and, therefore, the whole thing will become $e^x$, which after division will become $1$ instead of $e^{-1/2}$.