We present the concept of 3D video particles, a versatile represen- tation for 3D video and real-time visual effects. By generalizing 2D video pixels towards 3D irregular point samples we combine the simplicity of conventional 2D video processing with the power of more complex polygonal representations for 3D video. The heart of the video particle representation is a hierarchical space- subdivision data structure. Once initialized, this data structure can progressively be updated by so-called “smart” streaming. A smart 3D video stream consists of operators such as particle inserts, updates or deletes accounting for input changes. Our approach works with any real-time 3D reconstruction method and scales smoothly from view dependence to view independence. The video renderer implements a variety of advanced rendering operators including re-shading, and numerous 3D visual effects and makes use of state-of-the-art vertex processing hardware. 3D video parti- cles are primarily designed for 3D telepresence, augmented reality, location-based entertainment, and real-time virtual studio applica- tions.