

Well, after a decent amount of research and tweaking, there is a solution to this problem. If you're lucky, it'll be Dolby ProLogic if you're not, it will be plain old 2.0 stereo.
PLEX HOME THEATER MAC NO SOUND TV
If you're watching media that uses, for example, an AAC-encoded multi-channel soundtrack (most of the Apple HD trailers are like this) then it will be like the Apple TV situation above - your Mac can't send the AAC stream out the audio port, as it's only a two-channel device, so QuickTime player (or VLC or.) mixes it down to stereo and outputs this. You will have real surround sound from this setup. AC3 is the codec that Dolby Digital uses, so if you've already got a Dolby Digital soundtrack, and your optical port is configured properly (as a digital passthrough), then you may get the AC3 stream output through the optical port, and your surround receiver decodes it. Under a certain set of circumstances, your Mac can output a surround stream from the optical output that a surround receiver can decode as proper surround - this is if the media file you're using already contains an AC3 encoded soundtrack. Either way, you're hearing Dolby ProLogic, not Dolby Digital. Your surround receiver probably can't decode AAC, and at any rate, the Apple TV won't send it as AAC, it decodes it, mixes it back up as a stereo soundtrack (using Dolby ProLogic) and outputs that. Other times, and more often on the HD content, it is actually a real 5.1 surround soundtrack, but it's in AAC format. This isn't "true" surround sound, it's surround information matrix-encoded into a regular two-channel audio stream, and done extremely cleverly.

It sometimes uses a stereo soundtrack that uses Dolby ProLogic to do surround sound. Apple supply media to the Apple TV with one of two different options for the soundtrack. Well, the truth lies somewhere in between these two extremes. "But Wait!" you say, "Yes it can, Apple even advertise surround sound as a feature of the Apple TV!" or "I can play a DVD and I'm hearing surround sound." Your Mac can not do real surround sound from its built-in optical audio port in fact, not even your Apple TV can. I have also modified the title of the hint to more accurately reflect what it's about.] Just take the intro to the hint with a grain of salt, as it's not the whole truth. I've chosen to leave the hint online, as the hint (and moreso, the comments) contain a wealth of useful information.
PLEX HOME THEATER MAC NO SOUND HOW TO
However, what the author then goes on to describe explains how to convert certain AAC-surround-encoded files to AC3 mode for true surround playback. As noted in the comments by the hint's author, the introduction is misleading - your Mac can do real surround sound, assuming the source has an AC3 soundtrack. [ robg says: The following hint is presented as it was submitted.
