The problem is to find a constant $c > 0$ that holds for all $n \in \mathbb{N}$.
Now, what I have tried is the following:
I wrote $\{x^2\} = x^2 - \lfloor x^2 \rfloor$ and split the integral into two integrals:$$ \int_0^n \{x^2\} \, dx = \int_0^n x^2 \, dx - \int_0^n \lfloor x^2 \rfloor \, dx.$$
The integral of $x^2$ gave me $\frac{n^3}{3}$.
For the integral of $\lfloor x^2 \rfloor$, I converted it into a sum:$$ \int_0^n \lfloor x^2 \rfloor \, dx = \sum_{i=0}^{n^2-1} \int_{\sqrt{i}}^{\sqrt{i+1}} i \, dx.$$
For each interval $\left[\sqrt{i}, \sqrt{i+1}\right)$, we have $\lfloor x^2 \rfloor = i$.
Further simplifications lead me to the result for the integral of $\{x^2\}$ from $x = 0$ to $x = n$:$$ \int_0^n \{x^2\} \, dx = \sum_{i=1}^{n^2} \sqrt{i} - \frac{2}{3}n^3.$$
But how can I progress from here?