In this paper, we propose an advanced high-level cache management policy based on the processor access information, named as L1VPAI (L1 plus Victim cache with Processor Access Information). The L1VPAI is a cache replacement policy that uses the frequency of the specific cache line. In this policy, conflicted lines in the L1 cache are placed selectively in the victim cache or the level 2 (L2) cache based on previous memory access patterns. In this manner, the L1VPAI policy can make the frequently used address of cache locations reside longer in the high-level caches. We simulate our policy with RSIM, the event-driven simulator, and analyze the simulation results. The simulation results show that the average execution time of the L1VPAI outperforms the simple victim cache (L1V) by up to 6.44%. Moreover, performance gain is expected to increase, in the case of multiprocessor systems.