To remove shadows from foreground objects, an algorithm for background subtraction based on Scale-invariant model of local pattern (SILTP) was proposed. The algorithm was designed by fusing three levels of information from the frame,region and pixel levels. The frame level detects sudden,global changes between frames by the global mean gray value. At the region level,the SILTP operator was utilized to get the texture images. Then the texture histograms were used to model background, so as to quickly find contours of moving objects in the frame sequence. The pixel-level obtains object contours accurately using similar-ViBe algorithm. Exhaustive experimental evaluations on complex scenes show efficiency of the proposed method.