I found a new tool makesense who is trying to do the same thing that you're already doing. Probably with some new thoughts in mind. I recently asked the author how that tool is doing differently: SkalskiP/make-sense#23

The author responded that we need multiple clicks while labeling in imglab. Though I didn't understand it well as I can control most of the things with