I think the the appropriation can still be said to be a problem in that the people appropriating music might not even be racist, there is nothing inherently wrong with playing or selling another cultures music.
But by doing so they are "aiding and abetting" a racist society by enabling racists to listen to music they desire, from people they desire. If they choose not to it makes it harder for racists to be racist.
I think examples of harm, would be how like in the 20th century African American musicians had a much more difficult time selling their music, but the same music would garner a much larger audience if played by white musicians.
Its not simply white people performing black music, that is wrong, but rather that the white people filled a demand for black music that could have been supplied by the original culture were the market not racist. Its like, no direct racism, but making racism easier? If those white people had said "No I will not play this music", then the market would either have had to listen to black muscians or not get the music at all.
But by doing so they are "aiding and abetting" a racist society by enabling racists to listen to music they desire, from people they desire. If they choose not to it makes it harder for racists to be racist.