The DG module of Graphisoft has an old deprecated C interface and a newer C++ interface.
Using the old C interface you have to define callbacks and handle each event. It seems you're struggling with that...
I recommend you to use the C++ interface. It's much more easier, just simply inherit your class from DG::ModalDialog base class. If you want to handle an event (for example button click), then override the corresponding virtual function. There is a very simple Modal dialog example at our blog. I recommend you to start with that.
There is also a little more complex example at our blog, the Archicad Maze Generator dialog.