Of course it all depends on just how complex a jumble of elements are that you are trying to select.
But some things that may help.
As you mentioned just SHIFT+click to add new elements - don't drag a selection box.
Let go of everything and drag a new larger selection box.
Don't forget the selection area does not have to be a box - it can be an irregular polygon.
Also there is the left to right = select all partly inside the selection area, and right to left = select only entirely within selection area (I might have that around the wrong way) - this may help to select just particular elements.
Barry.
One of the forum moderators.
Versions 6.5 to 27
i7-10700 @ 2.9Ghz, 32GB ram, GeForce RTX 2060 (6GB), Windows 10
Lenovo Thinkpad - i7-1270P 2.20 GHz, 32GB RAM, Nvidia T550, Windows 11