When we ask the server to remove a contact we need to push the requester and responder ids to `remove_contacts` so that when the UI updates, the correct contacts will disappear from the list.

Co-Authored-By: Antonio Scandurra <me@as-cii.com>