You're right. I can't see any compression options, even when choosing the option to save individual frames as JPEG. This dialog doesn't look like it has been updated in many years, the last time I looked at it...
Try saving the individual frames as PNG or TIFF files - and while testing, don't save "All Frames" but rather a frame range - enough to generate a second or two of final video. Open a few frames to confirm that they are crisp. Depending on your platform and the software you have available - use whatever video editor you have (old QuickTime Pro, iMovie/Final Cut, Microsoft's current thing, shareware, etc) to convert the frames into a video file. In your video program, you can then have control over the compression level.
One of the forum moderators
AC 28 USA and earlier • macOS Sonoma 14.7.1, MacBook Pro M2 Max 12CPU/30GPU cores, 32GB