Swarm select next speaker should be using the latest handoff message and check for the last speaker

[ekzhu]

I believe you just identified a bug in Swarm. Rather than using the last message to identify the handoff target, we should be using the last handoff message and check if the handoff has happened already. The currently implementation didn't consider this scenario.

Originally posted by @ekzhu in #4146 (comment)