We present JAPE, a flexible prototyping system to support the design of a new advanced collaborative virtual environment. We describe the utilization of different hard‐ and software components to quickly build a flexible, yet powerful test bed for application and algorithm development. These components include a 3‐D rendering toolkit, live video acquisition, speech transmission, and the control of tracking and interaction devices. To facilitate the simultaneous design of applications and algorithms that take advantage of unique features of new collaborative virtual environments, we provide the developer with a flexible prototyping toolkit which emulates the functionality of the final system. The applicability of JAPE is demonstrated with several prototype applications and algorithms.