You can’t use Replace Color for this. You’re not dealing with color here, you’re dealing with luminosity. There is no color in black, gray, or white. They are the total absence of color.
In digital imaging, color and luminosity are two strictly separate components. The term “color” has a narrower meaning than in casual everyday speech, it just refers to the chroma component.