Let $x,y,a,b$ be the column vectors $(n,1)$ , $C(n,n)$ be the matrix, and
$$\phi(x,y)=\left\Vert x-a\right\Vert _{2}^{2}+\left\Vert y-b\right\Vert _{2}^{2}+\left\Vert x^{T}Cy\right\Vert _{2}^{2}$$
be the minimized vector function of two variables $x,y$ . The function can be rewritten as
$$\phi(x,y)=(x-a)^{T}(x-a)+(y-b)^{T}(y-b)+x^{T}Cyy^{T}C^{T}x.$$
Using the symmetry
$$x^{T}Cyy^{T}C^{T}x=y^{T}C^{T}xx^{T}Cy,$$
extremes are determined from conditions
\begin{align*}\frac{\partial\phi(x,y)}{\partial x} & =2(x-a)+2Cyy^{T}C^{T}x=0, \\\frac{\partial\phi(x,y)}{\partial y} & =2(y-b)+2C^{T}xx^{T}Cy=0.\end{align*}
Then, we obtain quadratic forms for $x,y$
\begin{align*}x & =(I+Cyy^{T}C^{T})^{-1}a,\\y & =(I+C^{T}xx^{T}C)^{-1}b.\end{align*}
Is there any method how to get the analytic solution?
Thank you very much for your help.