Due to its high dimentionality, deformable object manipulation is a challenging problem in robotics. In this paper, we present a deep neural network based controller to servocontrol the position and shape of deformable objects with unknown deformation properties. In particular, a multi-layer neural network is used to map between the robotic end-effector’s movement and the object’s deformation measurement using an online learning strategy. In addition, we introduce a novel feature to describe deformable objects’ deformation used in visual-servoing. This feature is directly extracted from the 3D point cloud rather from the 2D image as in previous work. In addition, we perform simultaneous tracking and reconstruction for the deformable object to resolve the partial observation problem during the deformable object manipulation. We validate the performance of our algorithm and controller on a set of deformable object manipulation tasks and demonstrate that our method can achieve effective and accurate servo-control for general deformable objects with a wide variety of goal settings. Experiment videos are available at https://sites.google.com/view/mso-deep.