Dave,
Here are some recommendations I have:
A. Examine the polygon count and the number of element interactions, as discussed here:
http://www.archicadwiki.com/3dSpeed
(Polygon Counting Tool for 16 can be found
here)
B. Are 3rd party or custom objects beings used in the project? What about add-ons? The next time the error occurs, before restating the computer, try excluding 3rd party objects or elements created by add-ons in the 3D view to see if it will generate. Also, check the file sizes of objects in the embedded library. I have seen some cases where simple looking objects are 40-60 MB in size and account for more than half of a .PLN file's size and create unwanted drag on the file's performance. Eliminate or replace these objects with a better alternative.
C. Open the Windows Resource Monitor via the Task Manager and check in the Memory tab to see a listing of data about memory usage. In particular take note of the "Commit (KB)" readings and the corresponding "Commit Charge" graph to notice how the virtual memory is performing. If this is maxed out then the error in ArchiCAD is correct.
The fact that a restart on your computer will clear out the problem for a time makes me think that there is a hungry application or service which is eating away at the memory on the system. I have had experiences where some fairly common applications (MS Outlook, Firefox, Safari etc.) have been the cause of scarce resources.
Since you are in my region of support I will contact you directly as well to troubleshoot directly.
Best regards,