The API documentation is out-of-date and still shows the old 3-argument version.
We are working on it, and hope we can publish a new one in few months.
You can read the following in the documentation's "New features of the API" topic:
Resource chain is gone (also ACAPI_UseOwnResModule() and ACAPI_ResetResModule() along with it)
OS X internally maintained a stack of open resource files called the resource chain. Whenever you want to load a resource, the system searched downwards in this stack to find the resource in any of the open resource files below the current one. We removed that implicit mechanism; you'll have to specify each and every time the resource module explicitly (usually with ACAPI_GetOwnResModule()).
So when you are updating your Add-On from 17 to 18, first you should delete ACAPI_UseOwnResModule() and ACAPI_ResetResModule() calls, and after that fill the C style DG function calls new resource module parameters with API_GetOwnResModule().
DG_DLL_EXPORT short CCALL DGModalDialog (GSResModule dialResModule, short resId, GSResModule dialIconResModule, const DGDialCallBack dCallBack, DGUserData userData);
DGModalDialog (ACAPI_GetOwnResModule (), MyDialogResId, ACAPI_GetOwnResModule (), MyDialog_Handler, NULL);