The depth images from RGB-D cameras contain a substantial amount of artifacts such as holes and flickering. Moreover, for fast moving objects in successive frames, we perceive ghosting artifacts on the depth images. Hence, the poor quality of the depth images limits them to be used in various applications. Here, we propose a gradient based spatial and temporal method of depth enhancement (gSMOOTH) using least median of squares, which deals with these artifacts. For each depth pixel over a sequence of frames, we look for invalid or unstable or drastically changed depth values and use our approach to replace those values with stable and more feasible depth values. Our approach removes the ghosting artifacts and flickering, and attenuates the amount of temporal noise significantly in real time. We conduct experiments with our own- and reference datasets and evaluate our method against reference methods. Experimental results show improvements for both static and dynamic scenes.