We propose a differentiable cloth simulator that can be embedded as a layer in deep neural networks. This approach provides an effective, robust framework for modeling cloth dynamics, self-collisions, and contacts. Due to the extremely high dimensionality of the dynamical system in modeling cloth, traditional gradient computation for collision response can become impractical due to prohibitively high computational costs. To address this problem, we propose to compute the gradient directly using QR decomposition of a much smaller matrix. We show experimentally that our method can speed up backpropagation by two orders of magnitude, and apply our approach to a number of inverse problems for cloth modeling, including parameter estimation and motion control.