Day 26 : People Occlusion with ARkit and ARFoundation in Unity

Introduction

In a previous article, we looked at implementing basic plane occlusion in our AR application using ARFoundation. We used an occlusion shader on detected horizontal planes in the scene to occlude and virtual objects we placed in the scene. You can find that article here. 

This sort of occlusion is fairly simple and not that performance expensive. At the same time, it doesn’t produce super realistic visualization as well. This is because it would only with horizontal or vertical planar surfaces. Now, that isn’t the most ideal real-world scenario.

In this article, we’ll explore and implement People Occlusion using ARFoundation and ARKit. 

Apple has just released iOS 13 along with the new iPadOS just a couple of months ago. The new OS brings with it People Occlusion which is a long-awaited missing feature of current AR experience. 

What is Apple’s People Occlusion & what value does it add to the AR experience?

With ARKit, now real-world objects like a person can stand between the device camera and a virtual object, the virtual object will now become hidden behind the person. virtual objects can now realistically interact with the real-world ones. 

How exactly does it work?

Basically, it uses Apple’s Neural Engine to perform segmentation analysis on the human shape and apply that data into a stencil and depth buffer just like we did in the occlusion shader.

We don’t have to worry about implementing all this. ARFoundation and ARKit have neatly.

Since it’s going to be ARKit, it means, unfortunately, no support for ARCore and Android devices just yet.

Additionally, not all ARKit compatible devices are supported. It specifically only supports ARKit 3 devices. These are devices with the A12, A12x or the A13 bionic chips with iOS 13 or later running on them. This limitation is largely due to the capability of the apple’s neural engine that is integrated with those SoC’s.

This is what our final result is going to look like. It can also occlude the whole human body as well.

Getting Started

We’ll use the plane detection article from here where we placed cubes on the plane for this article. We’ll apply people occlusion here so that the effect can be visualized against the placed cubes.

Prerequisites

1. Unity 2019.2.18f1 or above
2. An iOS device
3. Project setup from above.

Download Source Code

Implementation

Updating AR Foundation and ARKit to 3.1.0

First, we need to install or update our ARFoundation and ARKit package SDK’s to the latest preview to access these features.

If you’re using an older version of both like in the article mentioned above, we’ll need to upgrade them. Otherwise, the installation of them will be as usual.

ARFoundation 3.1.0

Open up package manager by going to Window -> Package Manager. Search for AR Foundation in the search bar. It should show up the currently installed version if you have it installed.

Click on the dropdown and select See all versions.

Now from the list select the latest preview. As of writing this article, the version for that is 3.1.0 preview 8. Note here that, for this feature to work you’ll need a minimum of 3.1.0 preview 1.

Click on install or update and Unity will download the package and import it automatically. It might take a couple of minutes.

ARKit 3.1.0

Now, if you’re upgrading from a previous version of ARKit or ARFoundation, you will see a lot of errors in the console window after the ARFoundation update. This is just Unity complaining of missing references. Nothing to worry about. We simply need to update ARKit as well. You will not see these errors if you’re doing a fresh installation.

Go to the Package Manager again and search for ARKit, in case of an upgrade, it will clearly show an error with the incompatible dependencies.

Select the drop-down and click on the preview.8 – 3.1.0 and install or update it.

Configuring the Scene for People Occlusion

AROcclusionManager

Just like other trackables within ARFoundation we saw before, people occlusion is handled by the AROcclusionManager.

We already have an AR Session Origin gameobject in the scene. As it’s child is the AR Camera to which we’ll add the AROcclusionManager component.

This will essentially provide the depth data to the ARCameraBackground that is already on the AR Camera gameobject.

Select the AR Camera and -> Add Component -> Search for AROcclusionManager.

You will now see a few properties for the AROcclusionManager.

The two properties are HumanSegmentationStencilMode and HumanSegmentationDepthMode. These properties allow us to adjust and balance the quality of information and then performance for rendering the occlusion pass.

HumanSegmentationStencilMode can take one of 4 possible values.

• Best – The human stencil image will have a resolution of 1920×1440.
• Medium – The human stencil resolution will be 960×720.
• Fastest – The human stencil image resolution will be 256×192.
• Disabled – Human Occlusion will be disabled altogether.

HumanSegmentationDepthMode can take these 3 possible values.

• Best – A filtering pass of resolution 256×192 is applied human depth image.
• Fastest – The depth image will have a resolution of 256×192.
• Disabled – No depth image and occlusion will be disabled.

You can experiment with different levels of modes and see the results for yourselves. For this article, in the demo, I have set the StencilMode to Medium and the depth mode to Best.

Note here that, disabling either one of them will disable human occlusion entirely.

Building the App

Finally, before building the application we need to set the Minimum OS Version for iOS in the player settings. Go to -> Project Settings -> Player.

Here, change the Target Minimum iOS Version to 13.

That’s it! Now build the app with Unity and XCode and test away!

Please leave any queries or comments below and I’ll get to them as soon as possible.

Download Source Code