How to avoid stale overlay files in mix releases – Questions / Help

I noticed a weird behavior in one of the CI/CD pipeline I’m working on: I removed a file inside rel/overlays, but the CI included it inside the mix release regardless.

I’ve looked into it and found out that the release stored inside _build/$env/rel/$name has all the stale files copied from old overlays and I can’t get rid of those files even when I run mix release --overwrite --force, I’ll paste the steps to reproduce this issue in an empty project:

mix new my_project
cd my_project

# I'm just adding a release to the mix.exs
cat << EOF > mix.exs
defmodule MyProject.MixProject do
  use Mix.Project
  def project do
    [
      app: :my_project,
      version: "0.1.0",
      elixir: "~> 1.15",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      releases: [
        demo: [
          include_executables_for: [:unix],
          applications: [runtime_tools: :permanent]
        ]
      ]
    ]
  end
  def application, do: [ extra_applications: [:logger] ]
  defp deps, do: []
end
EOF

mkdir -p rel/overlays

# create two files in `rel/overlays`
touch rel/overlays/file1 rel/overlays/file2
mix release
ls _build/dev/rel/demo # both file1 and file2 are there

rm rel/overlays/file2
mix release --overwrite --force
ls _build/dev/rel/demo # `file2` is still there

Obviously if I remove _build/dev/rel the next release will only include file1, as expected. The problem is that our pipeline caches both deps and _build by default, so the only solution was to remove the whole cache and rebuild everything from scratch.

I’m trying to find a solution to avoid this issue in the future, my first idea was to cache only _build/$MIX_ENV/lib which I guess would work but that cache would not include dialyzer’s PLTs.
My second guess was to run rm -rf _build/$MIX_ENV/rel before running mix release --overwrite which I think would also work, but at this point I’m wondering if I’m missing something.

So my questions are: how do you cache your _build in your CI? How do you avoid stale files in you releases?

I use robocopy for something similar so maybe you could rsync the overlays folder?