Okay, I see you asking about what is specifically wrong with the controls. What's wrong is that there are too many input ways that the player needs to keep track of. A better way of handling the controls would be: mouse for ALL looking around, WASD for moving around, spacebar and Z for upwards and downwards movement (that last part worked okay). That means that the camera looks towards the mouse, and A & D strafes, W & S moves back and forth, spacebar and Z moves up and down. This is how the majority of first-person games do controls.
The camera should move quickly with the mouse, not slowly like it does in the current version. And inverted mouse will not work at all when the camera moves with the mouse, AFAIK.
An alternative to this would be to discard Z and spacebar altogether, and to have the character just move in the direction of the camera with W, instead of moving along an axis. This is a nice way of doing it, because it's simpler and requires less buttons. However, it also takes some of the freedom away. For example, you can't look down and go upwards at the same time, you need to be looking in the direction you want to go.
But yeah, choose one of these two control schemes, and it should improve the experience a lot.
I liked the different personalities of the color guys a lot! Great way to create some variety.