Libraries are fundamentally different from attributes (though I would like to have libraries of attributes but that is another discussion). The parts do not reside in the project, they are only references to the separate files, and so there is no meaning to purging them - any parts that aren't used aren't there (in the file).
The only performance issue related to this is when you load or reload the libraries. This can get quite tedious over a slow network with large, unbundled (non-PLA) libraries.
You can create a project specific library by saving your PLN as a PLA. This includes all the used parts into a single file with the project. The only problem with this arises if you have used library parts that call other components (like door leaves or hardware) using a variable parameter.
The only (reasonable) way to be sure is to save and open the PLA (using its included parts) and generate a full 3D model. If you get warnings such as "Can't find <macro name> at <line number> of <library part>" then you are suffering this fate. The good news is that you can usually add the few bits manually. The bad news is that doesn't help if you now want to change the door panel to one that hasn't been included in the library.
HTH.
