Thoughts - You do mean you set an Override Focus target, and not the camera target? Because if you haven't set an override focus target, the DOF will remain at a fixed range and not "keep that object in focus" (the camera target just sets where the camera is looking, DOF will remain at a fixed distance).
Other thoughts - is there motion blur is well (in case the out of focus is motion blur and not DOF)? Do you mean every frame is out of focus throughout the camera fly around, or just some? Are you rendering the whole animation, or just a single frame?