I understand. These terms are imperfect and drift... I've heard this called the "euphemism treadmill". Even 'white' as a race has meant different things across time. I personally like person of color or Black & Indigenous persons of color (POC or BIPOC) as directly opposing non-white because it doesn't obviously center white as the default. To me, it feel like solidarity amongst groups that aren't currently seen as the default and are othered. Regardless, I can understand your point because at the core of it what we're saying is 'non-white'.
I also agree that eradicating poverty (& closing the wealth gap) would be a great step forward towards racial justice in the U.S. even in the absence of racial distinctions. I've just started reading more about this strategy and am slowly becoming convinced.
However, I think even with the elimination of poverty, there would still be a mentality of "white as default" that would need to change. Lack of money is not the only aspect that would prevents equality (e.g. even a wealthy Black person can be racially profiled). Even in my ideal society, there would still be reason to actively name groups that have been marginalized and create/hold space for them. I could be wrong, but I just can't imagine reaching true racial equality (fair less justice) without naming who is ahead and behind (... hopefully one day, was ahead and behind).