Plex Media Center for OS X Leopard

It’s the topic of the hour, so I guess I should add a few more words.

Apple has supported GPU decoding of H.264 video since the release of the first Macbooks with 9400M GPUs. These feature NVIDIA’s PureVideo 3 chipset, for hardware-accelerated decoding of H.264 video. (I believe the PureVideo spec also supports VC1, aka Microsoft’s version of H.264, which is partially compatible).

Apple exposed this functionality through Quicktime initially. The Quicktime X player will use the hardware automatically on a supported chipset (first the 9400M, and now GT320M and GT330M) when a supported video is played. They also allowed the use of the hardware accelerator when the new QTKit Objective-C API was used in playback mode.

(Note that “a supported video” in this case appears to be limited to Main and some High-profile H.264 video. Additionally, when using QTKit  / QT X, a .mp4 container was required.)

Fast-forward to 10.6.2, and Apple broke out the hardware acceleration library into a system framework called VideoDecodeAcceleration.framework. (VDA for short). They’ve now provided development headers, and the framework is usable on 10.6.3.

What this means is that, instead of passing a movie file to Quicktime and having it render and display the movie using its own player, we can now feed the hardware encoded packets directly from ffmpeg, and receive decoded frames back, which we can do whatever we like with. From the end-user point of view, any compliant H.264 video in any container (.mp4, mpeg-ts, .mkv) can be accelerated transparently, without needing to use Quicktime.

So what does this mean for Plex? As I’ve alluded to before, the team and I are working on a new Core Video-based player to eventually replace the Plex video player. I’ve been able to integrate VDA into the player, and have hardware-accelerated playback on my 9400M-based Mini. There are some impressive performance gains to be had, and it will be invaluable for notebook users in terms of power consumption.

We’re currently investigating the feasibility of backporting the API to Plex, however there are some technical and design issues which make this non-trivial. Hopefully we’ll have some good news soon.

Technical Note TN2267: Video Decode Acceleration Framework Reference

Stay tuned.

There’s been a lot of noise on the forums about hardware accelerated decoding of video in Plex lately, largely due to the release of Snow Leopard, which enables hardware decoding of Quicktime videos, and the Broadcom Crystal HD mini-PCIe card, which also supports H.264 and MPEG-2 decoding.

Obviously the question is whether these and other performance-enhancing technologies will make it to Plex. The answer depends very much on what these technologies bring to the table, so it’s probably worth spending some time on explaining them.

Quicktime X

The new version of Quicktime uses NVIDIA’s VP3 (Pure Video 3) API for hardware acceleration of H.264 video. This is currently only available on the 9400M GPUs in Macbooks and Mac Minis. The actual hardware is a separate decoder chip on the board, not directly using the GPU itself. This differs from OpenCL, which aims to use the GPU as a general-purpose CPU, and is not well suited to video decoding.

Unfortunately the API is only available when using QTKit, Apple’s new Cocoa framework for Quicktime. This means the support is limited to formats QuickTime X supports, which for our purposes isn’t particularly useful. Installing Perian helps by allowing MP4/MOV with AC3 audio to play, but MKV support even with Perian is pretty poor. Surround sound isn’t very well supported either, although future versions of Plex will have the capability to take multichannel sound being played through Quicktime, re-encode it to AC3 and send it to your receiver, in much the same way as the AC3 encoder works now. This may change if Apple broadens the API to allow acceleration of raw H.264 streams.

The other difficulty with QTKit is that rendering is only possible to a QTMovieLayer or a QTMovieView. This makes it incompatible with the current video player in Plex, so QT X support needs to wait for our new player, and is not an ideal solution for all video.

Crystal HD

This is a H.264/MPEG-2 decoder card made by Broadcom. It can be installed in the mini-PCIe slot of an older (non-NVIDIA 9400M) Macbook or Mini, replacing the WiFi card, or in a desktop using a mini-PCIe to PCIe adaptor. XBMC supports this and has made an OS X driver. Unfortunately the driver does not yet support Snow Leopard or 64 bit kernels. Once the XBMC team makes an improved driver available, we’ll look at implementing support in Plex.

64 bit FFmpeg / ffmpeg-mt

Compiling ffmpeg for 64 bit (the AV decoding library Plex uses) gives a small but measurable performance increase. Additionally, the ffmpeg developers are always improving the H.264 decoder, and we benefit from that as we upgrade Plex.

An experimental branch of ffmpeg called ffmpeg-mt is being developed, which uses multithreading to speed up video decode. In our development builds, this has also resulted in improved video playback performance.

Core Video

The dvdplayer code in XBMC isn’t ideal from design point of view anyway. Its threading is poor as a legacy from the XBox, and often results in slow performance in general, let alone playing video.

It doesn’t use timestamps for A/V sync, instead relying on audio playback speed to sync video. This is why we have so many problems with poor AV sync and drifting sync.

Apple released a new video framework in 10.4 called Core Video. Essentially this is a high-performance thread attached to a timer, which fires when the active video device is ready for a frame. It provides a clock which players can use to sync video and audio, and the advantage of outputting frames as the hardware is ready means we get smooth playback of 24p video on 50/60Hz TVs.

We’re developing a player based on this technology, which not only gives higher performance, as we only render when the display needs it, but A/V sync is much improved, as we can match DTS/PTS timestamps in the video to the audio and display hardware.

Finally…

We’re certainly not sitting still on the issue of better performance, but we’re aware for most people, their videos play fine, and the overall user experience is what needs work. We’re moving forward on all fronts, and hope to have some exciting things to show you soon.

I’ve been beavering away in the background on support for an exciting new Plex feature (Coming Soon™), but in the meantime the chaps over at FFmpeg (www.ffmpeg.org) have released v0.5 of the library (which Plex uses to decode most of the media you all enjoy). FFmpeg releases are few and far between, but the guys at XBMC have already worked hard getting the new version up and running. Plex of course benefits by inheriting that hard work. We also benefit from all the work the FFmpeg team have put in optimising and improving the various codecs the library supports.

I’ve also added the new version of libfaad2 (the library we use for AAC decoding) and added support for LATM-AAC (a common muxing format for AAC when used in broadcast DTV). The more astute of you will note that we now support both PAFF H.264 video, and LATM-AAC audio, which are the formats NZ (and Norway) broadcast their digital HDTV in. I’ll let you ponder that while you enjoy the screenshots…

The new stuff should be in an upcoming Plex release, but those of you who want to live on the edge can grab a binary from http://bluemandrill.com/plex/Plex-new-ffmpeg.zip. Source is available at http://github.com/elan/plex/tree/ryan-new-ffmpeg. Enjoy!

There’s been a few reports since we released Plex/7 that the volume of movies is much quieter than usual, and quieter than other apps playing the same movie. Elan and I have been tossing around a few theories about why, and what the answer is.

There are a few ways to solve it. The first is simply to “make the volume go to 11.” Unfortunately its not quite that easy, as you can’t amplify a digital signal above 100% without introducing clipping and distortion.

A solution common in the recording industry is to use Dynamic Range Compression (DRC), which amplifies the quiet sections of audio to match the louder sections. This is an acceptable solution for some, but I prefer to avoid it because you lose audio quality as you decrease the dynamic range available.

The third way is volume normalisation (what iTunes does when you choose Sound Check). Lets say you have a movie with an average volume of 50% of maximum, and peaks during action sequences which reach 70% of maximum. We can safely amplify this audio track by 30% without introducing clipping, and with no loss in dynamic range.

This is exactly what the next release of Plex will do, tracking the current signal in real time and amplifying it as much as possible without clipping. I’ve allowed a maximum amplication of 10dB so that you don’t blow up your speakers with silence.

Enjoy, and as always, polite and detailed feedback is welcome.

-Ryan

It looks like the new audio code has gone down well in most quarters, at least for people who were previously unable to use their 5.1 receivers with Plex. Thanks to the forum users for all the help debugging that issue.

Unfortunately you can’t make an omelette without breaking a few eggs, and two eggs I broke were the volume control (de-amplifier) and playback of some unusually-formatted music in paplayer (our music player). The volume control was purely because I personally don’t use it, and therefore it didn’t get the testing it deserved. My profuse apologies. The music playback speed bugs were more an issue of not having the files to test, but again, I apologise profusely.

I’ve read the forum reports of bugs, and have been able to reproduce and fix both. As a nice side effect, I’ve removed the resampling code from paplayer. We do a single conversion from whatever input format you use to 32 bit floating point (the same conversion CoreAudio would do internally) and maintain the audio in that format. We no longer do any sample rate conversion, which means if you want to play 24-bit 96khz (or higher!) audio, just set your hardware to that format in Audio MIDI Setup, and Plex will deliver it bit-perfect for you. The same is true for CD audio or other 16 bit 44.1/48Khz audio.

The new release will be available once it’s been through our dedicated testing team. Enjoy, and as always, polite and detailed success/failure reports here or in the forums are appreciated.

Well, they don't grow in trees...

I’ve just checked the new CoreAudio-based audio system into Plex’s unstable 0.7 branch. Elan’s going to roll a new build very shortly, so you should all be able to take it for a spin very soon.

There are two major benefits to the code: the first is better detection of audio hardware, so you shouldn’t have to muck around in the audio settings as much trying to get 5.1 sound to work. If you have a digital output built into your mac, plug it in, select the digital device, and play movies in surround!

The second of course is native AC3 output using CoreAudio. If you have a receiver which hasn’t previously been able to play surround sound from Plex, it almost certainly will now. If you’ve been using a USB or Firewire audio device, you may need to use the old system, which is available using the override switch in the audio settings page. Lip sync is also better because we can now accurately check the hardware latency of different devices.

The AC3 encoder is still present, and will work as previously.

This is the product of many late nights and weekends, and gracious help from Elan, and the PortAudio team, Phil Burke in particular. Thanks also to all the users on the forum who helped with testing and insight.

Enjoy!

Screenshot of the new audio settings showing the CoreAudio autodetection

I’m Ryan, one of the Plex Developers. This is a part time project for me (My day job is a junior doctor in NZ’s public health system) but I usually manage to find time to add a feature or fix a bug here and there.

Currently I’m working on improving our digital audio support, but am planning on integrating EyeTV support, better DVD playback, and giving Elan a hand on a couple of other projects which we think you’ll like…

In the meantime, I’ll post updates on audio support and other things I’m working on here.

Finally, in the spirit of Plex, here’s my personal Plex mascot (Plexscot?) Ted.

-Ryan