If you use a matrix, there's a subtle difference between
objects and the camera: with objects, you want to rotate, then
translate, which is easy to set up in the columns of one matrix;
yet with the camera, you want to translate (move the world),
then rotate (look in it). This is much easier if you prepare
*two* matrices storing translation and rotation, and multiply
them like so:
// rotate
var R=new THREE.Matrix4().identity();
... set rotation part of R ...
// translate
var T=new THREE.Matrix4().identity();
... set translation part of T ...
camera.matrix.multiplyMatrices(T,R);
Note you get really weird behavior if you multiply the matrices
in the other order.