People of color have been historically barred from sports groups, people of color have a long history of being discriminated against in the U.S.. There's baggage there. Forming an all-white team is not at all the same as forming an all-black team, because in the latter case it's an attempt to empower the minority. And if the double standard bothers you, well then, blame history.
I was agreeing with "well what's wrong with it - there are all-black teams too" side until I read your post. I never thought about it that way.