Instead of disconnecting them have them in a spectator mode until the game returns to a lobby. Probably also needs a spectators limit.
Actually probably shouldn't be a spectator mode, should only show the game state. To prevent people from connecting as spectator and getting extra info.