When you click “Remove as Friend,” it should take you to another pop-up that will confirm if you want to remove the player as a friend.
When you click on a friends name, you have three options; remove friend, join server and go back to menu. I know many people, including myself, who have accidentally clicked remove as friend instead of join server. I think that a confirm remove friend button will prevent this accident from happening.