Stability and transition of three-dimensional rotating boundary layers