You are not alone with this issue, but it usually in games where you see this phenomenon.
Site have a discussion about this:
https://social.technet.microsoft.com/Forums/en-US/7d6e1bb6-902a-4e0a-a660-9eaaa11e26b6/oversensitive...
https://superuser.com/questions/849918/erratic-mouse-movement-in-3d-games-over-rdp-with-remotefx
However, you can try something on your own machine. In ARCHICAD, the mouse can go crazy if the Remote Desktop Window isn't full screen, simply because the calibration is for a different resolution. The same goes for the Fly mode.
Try switching to full screen, if you haven't already done it, because orbiting, panning and zooming should work flawless only Fly mode is mixed up.
Szabolcs Mikó
Education Program Manager