Day 16 : Using Light Estimation in AR using ARKit and ARCore with Unity.

Introduction

Humans unconsciously perceive subtle cues regarding how objects or living things are lit in their environment. When a virtual object is missing a shadow or has a shiny material that doesn’t reflect the surrounding space, users can sense the object doesn’t quite fit into a particular scene even if they can’t explain why. This is why rendering AR objects to match the lighting in a scene is crucial for immersive and more realistic experiences.

In this article, we’ll look at how to use Light Estimation to light up our AR scenes with light information from the real world.

Light estimation enhances the AR experience by replicating the lighting conditions in the real world and blending it with the 3D AR World. When the AR Engine renders graphics, the rendering information with regard to lighting will be matched with the device camera’s real-world lighting conditions as it is captured.

Lighting Estimation does most of the work for you by providing detailed data that lets you mimic various lighting cues when rendering virtual objects. 

Getting Started

Within Unity,  we will use AR  Foundation and its  Light Estimation API.  This is a part of the AR  Camera Manager component. 

This article builds upon the previous articles on  AR and the series. You can find them here. For this,  you need to have set up AR Foundation from the first and second article here.  

This will hence work on both iOS and  Android but some parameters of the estimation will not work on android. We will get to that during the implementation.

Prerequisites

1. Unity 2019.2.18f1.
2. AR Foundation 3.1.0 setup
3. C# programming and understanding.

Implementation

In Unity, lighting is rendered on 3D objects using the Light component . To light up the entire scene, we use Directional Lights. They mimic ambient lights or even the sun in an open scene. You can read more about lighting and rendering in Unity here.

In AR, we’ll use light estimation to apply the properties onto a directional light in our scene. The ARKit and ARCore SDK’s use various hardware sensors on the devices to calculate this lighting data.

The following are the properties we can track.

1. Brightness – The estimated brightness of our physical environment.
2. Color Temperature.
3. Color Correction values.
4. Main light direction.
5. light color.
6. Main Light Intensity in Lumens.
7. Spherical Harmonics – Basically data representation for different lighting parameters and coefficients.

Note that if the platform supports this data and is available, then only the API will return these values.

LightEstimation Class

Let’s create a new Class called LightEstimation. This script will be attached to a Direction Light gameobject and we need the reference to it. Hence, we make it mandatory by using – RequireComponent. This makes sure that in case there is no Light component on it, it will add one automatically.

using UnityEngine;
using UnityEngine.Rendering;
using UnityEngine.XR.ARFoundation;

[RequireComponent(typeof(Light))]
public class LightEstimation : MonoBehaviour
{
}

First, let’s create a reference to our AR Camera Manager object and the Light component. Let’s also create the variables to hold the trackable properties for light estimation.

    [SerializeField]
    private ARCameraManager arCameraManager;

    Light mainLight;

    /// <summary>
    /// The estimated brightness of the physical environment, if available.
    /// </summary>
    public float? brightness { get; private set; }

    /// <summary>
    /// The estimated color temperature of the physical environment, if available.
    /// </summary>
    public float? colorTemperature { get; private set; }

    /// <summary>
    /// The estimated color correction value of the physical environment, if available.
    /// </summary>
    public Color? colorCorrection { get; private set; }

    /// <summary>
    /// The estimated direction of the main light of the physical environment, if available.
    /// </summary>
    public Vector3? mainLightDirection { get; private set; }

    /// <summary>
    /// The estimated color of the main light of the physical environment, if available.
    /// </summary>
    public Color? mainLightColor { get; private set; }

    /// <summary>
    /// The estimated intensity in lumens of main light of the physical environment, if available.
    /// </summary>
    public float? mainLightIntensityLumens { get; private set; }

    /// <summary>
    /// The estimated spherical harmonics coefficients of the physical environment, if available.
    /// </summary>
    public SphericalHarmonicsL2? sphericalHarmonics { get; private set; }

    void Awake ()
    {
        mainLight = GetComponent<Light>();
    }

This script will be assigned to a Directional light Gameobject. Hence, in the Awake() method we fetch its reference.

Next, the way we’re going to capture and get updated lighting information is by subscribing to the frameReceived callback from the ARCameraManager.

    void OnEnable()
    {
        if (arCameraManager != null)
            arCameraManager.frameReceived += FrameChanged;
    }

    void OnDisable()
    {
        if (arCameraManager != null)
            arCameraManager.frameReceived -= FrameChanged;
    }

Let’s create a new method FrameChanged which takes ARCameraFrameEventArgs type as a parameter.

void FrameChanged(ARCameraFrameEventArgs args)
{
}

Next, we’ll use the ARFrameEventArgs property -> lightEstimation to fetch the required parameters.

void FrameChanged(ARCameraFrameEventArgs args)
    {
        if (args.lightEstimation.averageBrightness.HasValue)
        {
            brightness = args.lightEstimation.averageBrightness.Value;
            mainLight.intensity = brightness.Value;
        }

        if (args.lightEstimation.averageColorTemperature.HasValue)
        {
            colorTemperature = args.lightEstimation.averageColorTemperature.Value;
            mainLight.colorTemperature = colorTemperature.Value;
        }
        
        if (args.lightEstimation.colorCorrection.HasValue)
        {
            colorCorrection = args.lightEstimation.colorCorrection.Value;
            mainLight.color = colorCorrection.Value;
        }

        if (args.lightEstimation.mainLightDirection.HasValue)
        {
            mainLightDirection = args.lightEstimation.mainLightDirection;
            mainLight.transform.rotation = Quaternion.LookRotation(mainLightDirection.Value);
        }

        if (args.lightEstimation.mainLightColor.HasValue)
        {
            mainLightColor = args.lightEstimation.mainLightColor;
            
#if PLATFORM_ANDROID
            // ARCore needs to apply energy conservation term (1 / PI) and be placed in gamma
            m_Light.color = mainLightColor.Value / Mathf.PI;
            m_Light.color = m_Light.color.gamma;
            
            // ARCore returns color in HDR format (can be represented as FP16 and have values above 1.0)
            var camera = m_CameraManager.GetComponentInParent<Camera>();
            if (camera == null || !camera.allowHDR)
            {
                Debug.LogWarning($"HDR Rendering is not allowed.  Color values returned could be above the maximum representable value.");
            }
#endif
        }

        if (args.lightEstimation.mainLightIntensityLumens.HasValue)
        {
            mainLightIntensityLumens = args.lightEstimation.mainLightIntensityLumens;
            mainLight.intensity = args.lightEstimation.averageMainLightBrightness.Value;
        }

        if (args.lightEstimation.ambientSphericalHarmonics.HasValue)
        {
            sphericalHarmonics = args.lightEstimation.ambientSphericalHarmonics;
            RenderSettings.ambientMode = AmbientMode.Skybox;
            RenderSettings.ambientProbe = sphericalHarmonics.Value;
        }
    }

Save this script and inside unity, add it to the directional light gameobject.

You can also show this info on a UI canvas and see how the values are changing to understand light estimation better.

That’s it! Now go ahead and drop in your 3D models and build out your app onto your device to see the magic happen.