If we're going with the definition of invasive species as the non-native organism whose sudden introduction suddenly upsets the balance of an established ecosystem... definitely white folks. After all, we brought all the other invasive species over, and do cool stuff like strip mine, clear cut, and dam rivers.
But "invasive species" usually just means "organism that we don't like, at least not anymore." Heck, coyotes are considered invasive by a lot of folks for no better reason than that they eat animals we have an interest in eating. They didn't cross an ocean or hitch-hike with us or anything.
Humans hate change in the natural world, which is perfectly understandable. We're top dog for now but we realize that any little shift in the makeup of the globe could upset that little throne. 99.9% of all species that we think have existed are extinct now. Natural selection selects in a rather brutal manner. The Bradford pear, fire ant, and wild hog are just winners like us and we resent them for it. The black rhino and a lot of megafauna are losers but we want to string them along.
We're weird.