Digital compositing is an important topic in the field of computer graphics and image processing. Many works have sought a good method to prevent composited images from being distinguishable from real images. The paper proposes a new environment matting algorithm to model the appearance of transparent object under different backgrounds. In the proposed approach, the frequency-domain analysis is adopted to compute the relationship between the area of foreground object and background image. Furthermore, the Kaczmarz method is applied to compute the caustics from refraction by transparent objects. The experimental results show that the algorithm effectively improves the quality of composited image and has a high PSNR. In addition, the composited results are also demonstrated by a video sequence involved of the motion, rotation and scaling of the transparent object.