Definitely sounds like a hardware problem.
I have a p4-2.0Ghz system with 1 Gig of RAM, and a 32Mb Graphics card, and i've never seen it crash on 3D.
The fact that it is killing windows entirely would indicate that it is a system-wide problem, not just ArchiCAD.
Try running 3DMark on it(use version 2001SE):
www.futuremark.com
If it works at all you can stress test your machine using it.
The symptoms you are experiencing sound a lot like Power supply issues, so try swapping that out for another one.
Also, if you can, check your CPU and GPU temperatures.
Josh Osborne - Central Innovation
HP Zbook Studio G4 - Windows 10 Pro, Intel i7 7820HQ, 32Gb RAM, Quadro M1200