Can I remove a board user from my workspace even though I didn't add them?

Absolutely! You are able to remove any board user if you have edit access to the workspace or board that the user is part of.