As for target and background changes, this paper proposes a target tracking algorithm with scale and orientation adaptive mean shift tracking with corrected background-weighted histogram algorithms and fast compression tracking algorithms in collaboration based on image classification. According to the difference of the image changes, the algorithm classifies the images into two categories, global changes and target local interest area changes. Global changes caused by lighting, background similar and background blur, it uses BW-SOAMS algorithm for target tracking. Local area of interest changes caused by size, rotate and occlusion, it uses CT algorithm for target tracking. Firstly, images are done preprocessing and classification, and then the appropriate track drift problems are caused by changes in the environment. By experiments, the algorithm has improved significantly in precision and efficiency.