This paper addresses the problem of efficient and effective restoration of text images, by formulating the problem as inferring the surface from a sparse and noisy point set in a 3D structure tensor space. Given a set of noisy data correspondence in corrupted images, the proposed method extracts good matches and rejects the noisy elements. The methodology is unconventional, since, unlike most other methods, it optimizes certain scalar, objective functions. Also, as the proposed approach does not involve initialization, or any iterative search in the parameter space, it is free from the problems of identifying only local optima or having poor convergence properties. Subject to the general restoration of natural images, the removal and restoration of corrupted regions is performed by 3D tensor voting based on a fuzzy median filter. In essence, the input set of matches is first transformed into a sparse 3D point set so that 3D tensor kernels can then be used to vote for the most salient surface that captures all inliers inherent in the input. Lastly the density estimation for detecting the center modes is performed as well as a clustering algorithm for segmenting the values according to the color components in the restored image. Experimental results are presented which show that the proposed approach is efficient and robust in terms of restoring and segmenting corrupted text images.