[buffer_cache] Fix Xenoblade Chronicles series (#4066)

This fixes a bug introduced in the Xenoblade Chronicles series where some small elements would occasionally flicker on the screen.

Reviewed-on: https://git.eden-emu.dev/eden-emu/eden/pulls/4066
Reviewed-by: crueter <crueter@eden-emu.dev>
Reviewed-by: Lizzie <lizzie@eden-emu.dev>

src/video_core/buffer_cache/buffer_cache.h

@@ -1644,8 +1644,6 @@ bool BufferCache<P>::SynchronizeBuffer(Buffer& buffer, DAddr device_addr, u32 si
             start = (std::max)(start, gend);
         });
         push(start, end);
-        ClearDownload(addr, range_size);
-        gpu_modified_ranges.Subtract(addr, range_size);
     });
     if (upload_copies.empty()) {
         return true;