Thank for looking into the matter and communicating also in the GEANT4 forum. I also tried to get deeper insight.
Maybe it is worthwile to explain where we are using this function and stumbled over its behaviour. When transporting an input event with GEANT, we have to rotate the event (all particle momenta within the event) from the coordinate system of the event generator (UrQMD in our case), which usually defines the z axis as beam axis, to the experiment coordinate system, accounting for a finite beam divergence. For this, we apply TVector3::RotateUz to each particle momentum, with the beam direction as argument, before we put them onto the GEANT stack.
Now, the implementation of the function uses two rotations. In polar coordinates, the direction vector NewUzVector can be written as u = (sin(theta)cos(phi), sin(theta)sin(phi), cos(theta)), with 0 <= theta <= pi, and -pi < phi < pi. This vector can be rotated into the unit vector in z by first rotating around the z axis by -phi, which brings u into the x-z plane, and then rotating around the (new) y axis by theta.
What we want is the inverse transformation, so first rotating by -theta around y, and then by phi around z. The implementation follows this as @ferhue has explained in the GEANT4 forum. So, it appears mathematically correct.
But is does not behave as we expect, as can be seen in the special case of e.g., u2 = 0, i.e., u is in the x-z plane. Looking into the formulae, one gets for the transformation matrix in the limit of theta->0
A = (
1 0 0
0 1 0
0 0 1
)
i.e. identity for ux > 0 (phi = 0), but
A = (
-1 0 0
0 -1 0
0 0 1
)
for x <0 (phi = pi). So here, the vector is rotated by pi in x-y.
In other words, the transformation has a discontinuity at theta=0. But rotations should be continuous transformations, should they not?
Probably, the limes theta-> 0 is mathematically not legal, since for theta = 0, phi is not defined. But the fact remains that one gets very distinct transformations for infinitesimally small ux depending on the sign of ux.
What one surely expects or wants to have in the case of u being in the x-z plane is just a rotation by theta around the y axis. The function RotateUz does not provide this. To me, it seems an artifact caused by restricting theta to positive values; so, if ux < 0, the rotation by pi in x-y is applied to get theta positive.
One remark still: This is a purely mathematical function, so arguments like “rotations in x-y do not change the physics” should not apply.