Picture-in-picture multitasking for iPad in iOS 9: Explained

iOS 9 takes multitasking from the background and puts it right up front on the iPad.

Apple calls the new features they're introducing in iOS 9 "multitasking", but the iPad has always multitasked. Over the years, it's gone from system multitasking to background tasks to background refresh. What it hasn't done until now, however, is run two apps on the same screen at the same time. That's where Picture-in-Picture (PiP) comes in.

Note: iOS 9 is currently in beta and governed by a non-disclosure agreement (NDA) that doesn't allow for screenshots or video. All the material contained in our iOS 9: Explained series is from previous, now public versions of iOS, from iOS 9 features shown off during the WWDC 2015 keynote, and from our coverage of the event, including our iOS 9 first look.

Multi-app multitasking basics

Picture-in-Picture isn't a new concept. It's existed in television and computer displays for years. It involves one video running on top of another video. So, for example, having the channel showing the baseball game on your TV, full screen, and the channel playing the football game in a smaller window in one of the corners.

The idea of traditional PiP is that you could watch more than one thing at a time. With PiP on the iPad, the idea is that you can watch a video while doing something else at the same time.

Because Picture-in-Picture needs to run the video on top of another app, it requires the resources of an iPad Air 2, iPad Air, iPad mini 3, or iPad mini 2.

Picture-in-Picture is only one of three ways iOS 9 offers multi-app multitasking on iPad. The others are Slide Over and Split View. For more on those, see my previous explainer:

• Slide Over and Split View multitasking for iPad in iOS 9: Explained

Where Slide Over and Split View apps bring with them the concepts of "primary" and "secondary" apps, Picture-in-Picture uses an established concept—"background media".

A brief history of video playback

The original iPhone, first shown off in 2007, had an iPod app to play iTunes video, a YouTube app to play YouTube videos, and Safari to play some forms of web video. In other words, video has been built into iOS from the start.

System, like phone calls and music, has always been able to continue playing in the background on iOS. It wasn't until 2010 and iOS 4, however, that application programming interfaces (API) were introduced that allowed App Store apps to handle Voice over IP calls and streaming audio—apps like Skype and Pandora—in the background.

In late 2010, Apple introduced AirPlay as a way of pushing video (and audio) content from an iPhone, iPod touch, or iPad to an Apple TV. In the media player implementation—Apple added other implementations over time—the video would keep playing even if the originating device went to sleep or changed apps.

In 2011, iOS 5 split the iPod app into discrete Video and Music apps to match the iPad and iPod touch. Music kept backgrounding in the divorce.

More recently, Media Player, the original method used to play back video on iOS going back to the first iPhone SDK in 2008, was superseded by a new Framework in iOS 8: AVKit. In iOS 9, Media Player's player controller and player view controller are being deprecated in favor of AVKit's unified player view controller.

Likewise, the two other ways to play video, AVFoundation, which lets developers build their own players, and WebKit, which plays HTML5 web video, are getting new capabilities in iOS 9.

How PiP works

Previously, when using the built-in player, a player layer was created to hold a video. If you left the app running the video, however, the video would stop running.

Now, with iOS 9, there's also a PiP layer. When enabled, video frames will stop being directed to the player layer and start being directed to the PiP layer.

The transition is seamless—as soon as the switch is registered, the app ceases to get the video and PiP begins. Then, if you switch screens, switch apps, or exit to the Home screen, the video keeps playing. It's been decoupled from the app and, with very few exceptions, can just keep playing no matter what else you do.

(The major exception is—you can't play more than one video at once. So, if you have Videos.app playing in PnP, and then you start a YouTube video in Safari, the PiP video will pause as soon as the YouTube video begins.)

For apps using the standard media player, the PiP button now appears on the right side of the control bar, similar to how the AirPlay button used to. Tap it, and the video goes from the in-app player layer to the PiP player.

You can also enter PiP mode automatically when you leave an app. For example, if you start a video and then hit the Home button or tap on a Mail notification, the video will go to the PiP player as you exit. (You can turn that off in Settings if you prefer not to have automatic PiP.)

The PiP layer presents with three buttons and a progress bar. The first lets you leave PiP and go back to the in-app player layer. The second lets you pause or play the video. The third lets you close the PiP and stop watching the video.

FaceTime can also use PiP so that you can engage in a video conference call while switching between or using other apps, like checking your calendar or taking notes.

The PiP defaults to quarter-width (with ample transparent border) in the bottom left corner, but you can reposition it by touching and swiping it into another corner. It'll always snap to one of the four corners, however, so you can't just position it anywhere. iOS will also nudge the PiP above or below important interface elements, like navigation and tab bars, and even the Dock on the Home screen, so it doesn't obstruct anything.

You can also pinch-to-zoom the video from quarter-width to up to half-width (again with ample transparent borders) and back. There's no snapping here. You can adjust it to pretty much any size you like within that range.

Because the iPad has a 4:3 aspect ratio and video is 16:9, even in quarter-width mode the PiP only takes up about a sixth of the screen.

If the PiP gets in your way, but you don't want to kill it completely, you can tuck it off to the side of the screen for a while. The audio keeps playing, so you can keep listening, and the edge of the video remains visible as a tab at the side of the screen—with an arrow overlay—so you can pull it back out at any time.

You can even swipe the tab from bottom to top and side to side, and change its size, if it gets in your way.

So, for example:

1. You're in the Videos app watching the season finale of The Flash, full screen.
2. You decide you want to make sure the season finale of Arrow is available to watch next, so you tap the Picture-in-Picture button, and The Flash shrinks down to a small, quarter-width window and docks at the bottom right of your screen.
3. Now that you can use the rest of the videos app again, you tap to change shows and begin scrolling to find the episode you want.
4. Because the PiP is covering the bottom of the list, however, you swipe it over to the left side of the screen and, while you're at it, pinch-to-zoom it up to half-width.
5. Just then you hear the magic words—"Supersonic punch!" and, not wanting to miss it, you tap the full-screen button and go back to watching without distraction.
6. A few minutes later a push notification banner roles down. You tap it, switch to Messages, and The Flash switches back to PiP mode and follows along with you.
7. The message is from your co-worker. She needs some info. So, you pull Notes in Slide Over, pin it down into Split Screen, and start looking. And the whole while The Flash just keeps playing for you while you work.
8. There's a lot of text to go through, so you swipe the PiP off the side of the screen for a moment, but keep listening, as you copy and paste it.
9. Done, you pull the PiP back out, exchange a few more messages, and then tap the full-screen button to go back to Videos app. Just as the Flash starts running up the building...

Developers and Picture-in-Picture

For the standard player view controller, Apple does a lot of the heavy lifting for PiP, but the button won't appear automagically.

• To enable PiP, developers need to enable Audio, AirPlay, and Picture-in-Picture in background modes, and to set the audio category by adding AVFoundation and some code to the app delegate. Once that's done, the PiP button will show up.
• Developers also need to add some code to the player view controller that lets people leave PiP, return to the proper place in the app to restore video playback to the standard layer.
• PiP can also be disabled for apps that don't want to support it.

For custom players, there's more work that needs to be done.

• Developers first need to check if the device the video is running on can support PiP. If so, they can create a PiP controller.
• A PiP button needs to be added to the custom player controls, so viewers can choose to enter the mode.
• The in-app video player will need to be disabled while PiP video is playing, but without killing the PiP video.
• If a developer wants to support automatic PiP on exit, they need to make sure their player is in full-window.
• If the viewer taps the full-screen button, they need to be taken back to the right place in the app, and the in-app video player needs to restore playback from that point on.
• If the device supports PiP but using it isn't possible at the moment, the developer needs to disable the button and, ideally, show the viewer that the feature is currently unavailable.

For web video, if a developer is using the standard controls for HMTL5 video, they'll get PiP video "for free". Like standard PiP, it can also be disabled for developers who don't want to provide it.

Performance

Just like Slide Over and Side View apps, Picture-in-Picture apps have to share resources like CPU and Memory with other onscreen apps. Because those are finite resources, developers are urged to manage them smartly.

For example, if someone leaves a video app but continues playing the video in PiP, developers should proactively discard everything else about the app they can to lighten the background load. That includes view controllers, views, images, and other data caches.

All other tasks performed by the app—anything that doesn't involve playback of the video, should likewise be paused.

If the video app is using HTTP Live Streaming, developers should provide a smaller, lower bandwidth variant, and annotate the variant in the master playlist, so iOS knows about it. That way, when PiP is engaged, the smaller version of the video can be used, and resources can be saved.

PiP is only for PiP

Apple is upfront when it comes to pointing out picture-in-picture video is only for video and the accompanying audio. It can't be used for any other purpose or to try and transit or sneak in any other function.

The same rules that apply to background audio and AirPlay apply.

Taking iPad video to the next level

Back in February I wondered out loud what could happen if the iPad was allowed to be its own thing: If, like the Apple Watch and Apple TV, it still ran the same systems under the hood, but got to present an interface that better took advantage of the unique capabilities of the device. Picture-in-Picture, like Slide over and Split View, is a great step in that direction.

I'll save the details and assessments for my iOS 9 review, coming this fall when Apple ships, so for now I'll leave it at this—Instead of being "just a big iPhone", the iPad has become "two or three big iPhones", and that makes it exponentially more useful.