Can I remove a guest from my workspace even though I didn't add them?

Absolutely! You are able to remove any guest if you have edit access to the workspace or board that the guest is part of.