Friend list in Game

In the game, on my friend list. There are a lot of friends of mines have one "little people" next to their names. Most of my friends have 2 little people next to their names. Now, i want to clean my friend list by deleting someone that I would not talk to. But I cant click on those players with 1 people next to their names. Is there a way to delete them, it's kinda annoying, you cant delete them, but they took the spots on my friend list. Please help.