Intro to VR with Unreal Engine

Luis Cataldi - Epic Games
Intro to VR
with
Unreal Engine

VFX house Framestore uses UE4 for “Field Trip to Mars”
2
VR with Unreal Engine is everywhere

3
NASA: Building the ISS in Unreal Engine

4
“The Quinn Building” from the South West College in the UK

VR for Education
5

6
ILMxLAB's Darth Vader VR Story Experience Teaser

How can we learn to
harness the power of
Unreal Engine?

VR Learning Resources for Unreal Engine:
Starting Out:
• Oculus Quick Starts
• SteamVR Quick Start
• Google VR Quick Start
• Gear VR Quick Starts
VR Platforms:
• Samsung Gear VR Development
• Google VR Development
• Oculus Rift Development
• SteamVR Development
VR Topics:
• VR Cheat Sheets
• VR Best Practices
• Motion Controller Component Setup
• VR Camera Refactor

Video:
• 2015 UE4 - VR and Unreal Engine
• Making Bullet Train and Going off the Rails in VR
• VR Bow and Arrow Tutorial w/ Ryan Brucks - Training Livestream
• Training Livestream - Sam and Wes' VR Stream: Cameras, Multiplayer, Tips and Tricks!
• Creating Interactions in VR with Motion Controllers 1-3
• Setting Up VR Motion Controllers
• VR Networking and 3D Menus
• Up and Running with Gear VR
• Developing for VR
• Integrating the Oculus Rift into UE4
Presentations:
• UE4 VR - Niklas Smedberg
• Lessons from Integrating the Oculus Rift into UE4
• Going Off The Rails: The Making of Bullet Train
Links:
• Sam Deiter - 10 VR tips for Unreal Engine
• Tom Looman’s - Getting Started with VR in Unreal Engine 4

VR Editor Starting Out:
• Activating VR Mode
VR Editor Guides:
• Setting Up VR Editor from GitHub
• Navigating the World in VR Mode
• Working with Actors in VR Mode
VR Editor Reference:
• VR Editor Controls
• Quick Select Menu
• Radial Menu
• Transforming Actors in VR
• Editor Windows in VR Mode

• Unreal Engine 4.13 Release Notes
The Unreal Engine Release Notes:

Mitchell McCaffrey’s - Mitch VR Labs
• Mitch's VR Lab - an Introduction
• Mitch's VR Lab - Look Based interaction
• Mitch's VR Lab - Simple Teleportation Mechanic
• Mitch's VR Lab - Introduction to SteamVR
• Mitch's VR Lab - Simple Head IK
• Mitch’s UE4 Forum Post
Education Community VR for UE4:

Free Unreal Engine Courses:
• Twin Stick Shooter
• 3rd Person Power-Up Game with C++
• 2D Sidescroller with Blueprints
• Endless Runner with Blueprints
• Unreal Match 3 Game
Free UE4 Community Youtube.com Learning Channels:
• World of Level Design UE4 Fundamentals
• Virtus Education series
• Unreal Engine 4 Beginner Tutorials
• Mathew Wadstein Tutorials
• Leo Gonzales Unreal Basics
• Tesla Dev Tutorials
• UE4 Style Guide

Free UE4 Community Blueprints:
• Communication Training - Zak Parrish
• Blueprints Compendium - VOLUME II
• BP_Compendium.pdf
• Network Compendium
Free UE4 Community VR Learning Channels:
• Unreal Engine VR Curriculum
Free UE4 Community ArchViz Learning Channels:
• Architectural Visualization Tutorials
Paid Elearning Courses:
• Unreal Engine 4: The Complete Beginner's Course
• Learn to Code in C++ by Developing Your First
Game
• Complete Introduction to Unreal 4
• An Introduction to Creating Realistic Materials in
UE4
• Master Blueprints in Unreal Engine 4 - Endless
Runner
• Create a Helicopter Game Control System in
Unreal Engine 4
• Unreal Essential Training - Lynda.com
• Unreal: Learn Lighting - Lynda.com
• 3dmotive - Unreal Engine courses
• Pluralsight - Unreal Engine courses

Improved metadata support
● Skill level
● Engine version
● Sitemap filters
● Checkpoints
Learning Resources for Unreal Engine:

Getting the most value from the UE4 Launcher Learn Tab

Getting the most value from Content Examples

Recent Developments
New: VR Project Template
We’ve added a new project template designed for Virtual Reality on desktop and console.

Recent Developments
The template can be selected from the Blueprint tab as a new project is created.

Recent Developments
The motion controller template provides examples for object interaction and manipulation as
well as point based teleportation.

Recent Developments
At Epic, we drive engine development by creating content.
We use these as real-world test cases to drive our framework and
optimizations to the engine.

Recent Developments
Instanced Stereo Rendering
Lets us use a single draw call to draw both the left and right eyes, saving
CPU (and some GPU) time.
• Currently works on PC, and coming soon on PS4 and mobile platforms
• Enable it in your project’s settings

Recent Developments
Instanced Stereo Rendering
Instanced stereo is available now for the deferred (desktop) renderer and the PS4.
We’re also implementing multi-view extension support for the forward (mobile) renderer, and
improving PS4 support to better utilize the hardware.

Recent Developments
Hidden And Visible Area Mesh
We draw meshes to cull out geometry
that you can’t see, and only apply post
processing where you can.
For the deferred renderer, the visible
area mesh is a bigger optimization!
This is specific per-platform.

Recent Developments
The Hidden Area Mask uses a mesh to early-out on pixels that aren’t visible in the final
image.

Recent Developments
The Visible Area Mask uses a mesh to constrain our post processing to the visible pixels
only.

Recent Developments
Camera Refactor
As of 4.11, we’ve completely rewritten
the camera system in order to make
developing much easier!
• Camera Components now move
exactly as the real HMD is moving
• You can attach components (meshes,
UI, etc) directly to the camera
component!

Platform Support
As of 4.12, we support the following platforms out of the box:
• Oculus Rift
• Steam VR (including the HTC Vive)
• PSVR
• OSVR (preview)
• Samsung Gear VR
• Google Daydream
• Leap Motion

Platform Support
Create once, and deploy anywhere.
Mobile
Desktop / Console
Oculus Rift
HTC Vive / Steam VR
PSVR
OSVR
Samsung Gear VR
Google Daydream

Platform Support
All of these platforms go through UE4’s
common VR interfaces, so you can make
your content once, and deploy it anywhere.
• Unified Camera System
• Motion Controller System
• Optimized rendering paths
• Low-latency optimizations
Oculus
Vive PSVR
OSVR

New native UE4 VR Forward Rendering
options with multi-sample anti-aliasing
(MSAA)
“Current state-of-the-art rendering often leverages screen-space effects, such as screen-space ambient
occlusion (SSAO) and screen-space reflections (SSR). Each of these are well known for their realistic and high-
quality visual impact, but they make tradeoffs that aren’t ideal in VR. Operating purely in screen-space can
introduce incorrect stereo disparities, which some find uncomfortable.
Forward rendering lets us use multi-sample anti-aliasing (MSAA), adds arithmetic to our texture-heavy shaders
(and removes GBuffer writes), removes expensive full-screen passes that can interfere with asynchronous
timewarp, and offers a moderate speedup over the deferred renderer. Switching to a forward renderer has also
allowed the easy addition of monoscopic background rendering, which can provide a substantial performance
boost for titles with large, complex distant geometry.
https://developer.oculus.com/blog/introducing-the-oculus-unreal-renderer/

DESKTOP FORWARD RENDERER
Coming this Fall*

There are advantages to both!
The UE4 Deferred Render is a full-featured workhorse, but takes a bit of skill to fully leverage.
Temporal Anti Aliasing limits how sharp your image can be.
A new UE4 Forward Renderer will be a specialized renderer, with less features, but faster
baseline. Multi Sampling Anti Aliasing (MSAA) is the sharpest solution for anti-aliasing.

One of the biggest issues for working in
VR is Motion/Simulation Sickness.

en.wikipedia.org/wiki/Virtual_reality_sickness
Sensory conflict theory believes that sickness will occur
when a user's perception of self-motion is based on
incongruent sensory inputs from the visual
system,vestibular system, and non-
vestibular proprioceptors, and particularly so when these
inputs are at odds with the user's expectation based on
prior experience.

Five typical causes of Motion/Simulation Sickness in VR
Read more about it
1. Non-forward movements
• No unnatural movements
2. Awareness of Vection
• When a large part of the visual field moves, a viewer feels like
he has moved and that the world is stationary
3. The feeling of accelerations
4. Too much camera YAW
5. Helped by adding a static reference frame

Things we CAN DO in Unreal Engine to
improve VR Games and Experiences

You MUST maintain framerate
For the VR experience to feel smooth, your game needs to run on 75 hz (Oculus DK2) or even 90
hz. (HTC Vive and Oculus CV1) depending on the device. To see the current framerate type in
“stat fps” or “stat unit” (for more detailed breakdown) in your console when running the game.

Here is where VR Instanced Stereo Can Help
“Basically, we’re utilizing hardware instancing to draw both eyes simultaneously with a single draw call and
pass through the render loop. This cuts down render thread CPU time significantly and also improves GPU
performance. Bullet Train was seeing ~15 – 20% CPU improvement on the render thread and ~7 – 10%
improvement on the GPU.” – Ryan Vance.
To enable this feature in 4.11
and above, go to your Project
Settings and look for
“Instanced Stereo” under the
Rendering category.

Things to keep in at the front of your mind:
Check your performance constantly to ensure that
you are hitting your VR performance targets.

○ Maintain a very simplistic approach to making your content.
○ Minimize complex shaders as best possible.
○ Add detail to the mesh within reason in lieu of relying of complex
shaders for surface details.

LOD's and aggressive culling are a must to ensure
that you are hitting your VR performance targets.

Known issues and possible workarounds:
Screen Space Reflections(SSR)
SSR will work in VR but may not give you the results that you
want. Instead consider working with reflection probes.

Normal Mapping Issues
When viewing Normal maps on objects in VR, you will notice that they do not
have the impact that they might have once had. This is because normal mapping
does not account for a binocular display or motion parallax. Because of this,
Normal maps have a tendency to look flat when viewed with a VR device.

Parallax Mapping
Parallax mapping takes Normal mapping to the next level by accounting for depth cues,
Normal mapping does not. A Parallax mapping shader can better display depth
information, making objects appear to have more detail than they do. This is because no
matter what angle you look at, a Parallax map will always correct itself to show you the
correct depth information from that viewpoint. The best use of a Parallax map would be
for stone pathways and fine detail on surfaces. (Parallax maps often look better up close
whereas many normal maps will look passable at average distances.)

Tessellation Shader Displacement
Tessellation Shader Displacement will displace 3D Geometry in real time by
adding details that are not modeled into the object. Tessellation shaders do a
great job of displaying information because tessellation shaders actually create
the missing detail by creating more vertices and displacing them in 3D Space.

Launching VR Preview:
Testing out your VR set is very straightforward, simply select “VR Preview” from the Play dropdown
button. By default the head tracking will work right away without any changes to your existing project or
template.
** Also new for 4.13 is Play from VR Editor, which can be enabled in Editor Preferences.

GPU Profiling:
To capture a single frame with GPU timings press Ctrl+Shift+, or type in “profilegpu” in the
console. This command collects accurate timings of the GPU, you will find that certain
processes are a heavy burden on the framerate (Ambient Occlusion is one common
example) when using VR.
See GPU Profiling & Performance and Profiling for documentation.

GPU Profiling:
Unreal Engine 4.13 now also has a new experimental Real Time GPU Profiler!
•To enable it, enter r.GPUStatsEnabled 1 in the console
•Then type stat gpu to launch it

Disable Heavy Post-Processors:
Due to the demanding requirements of VR many of the advanced Post Processing features that you normally use should
be disabled. To accomplish this, first you can consider disabling heavy post processors in your Project Settings first, or
use the below sets per attribute.
• Add a Post Process(PP) volume to your level if there is not already one there.
• Select the PP volume and enable the Unbound option so that these settings will be applied to the entire level.
• Expand each Settings of the Post Process Volume and disable any undesired active PP settings by enabling that property by
clicking on it and then set the value from the default, usually 1.0, to 0 to disable the feature.
• Consider first disabling the biggest offenders to VR performance like Lens Flares, Screen Space reflections, Screen
Space Ambient Occlusion, and anything else that might be impacting performance.
• While some of the features are disabled by settings in your .INI files, this ensures that performance will not be affected if the .INI is
removed by mistake.

UE4 – Lighting for VR
• Dimmer lights & colors can help reduce simulation sickness.
• Make sure your Stationary / Dynamic Lights do not overlap.
• If using Dynamic Shadows, only have one shadowing light.
• Use Stat LightRendering to see current lighting cost.
• Use Static Lighting over Stationary or Dynamic, baked lights are the best option for VR
environments.
• Profile, Profile, Profile to ensure you are maintaining performance goals.

Fake shadows Wherever You Can!!
Using cheats like fake blob shadow drop to simulate dynamic shadows are a good
consideration for keeping VR project running at frame.
Blob shadow example. Image by Eric Chadwick

UE4 – Effects for VR
• Mesh based VFX work the best for VR.
• Camera Facing particles do not hold up well in VR on their own due to the stereoscopic view.
• The Dither Temporal AA Material Function can make Opacity masked objects look like
Translucent ones.
• Local Space rotation does not look correct in VR.

UE4 – Environments for VR
• Use Reflection probes instead of screen space reflections.
• Again… Texture Blob shadows are a cheap alternative to dynamic shadows.
• The Merge Actor Tool can help cut down on Static Mesh draw call without having to do work
outside of UE4.

Some very important things we all need
to know about Unreal Engine.

The Unreal Engine Framework
GameInstance
GameMode
Pawn Class
HUD Class
PlayerController
Class
GameState
Class
PlayerState
Class

GameInstance
GameMode
Pawn Class
HUD Class
PlayerController
Class
GameState
Class
PlayerState
Class
The GameMode is the definition of
the game.
● It should include things like
the game rules and win
conditions.
● It also holds important
information about:
○ Pawn
○ PlayerContoller
○ GameState
○ PlayerState

GameInstance
GameMode
Pawn Class
HUD Class
PlayerController
Class
GameState
Class
PlayerState
Class
The Pawn class is the base class of
all Actors that can be controlled by
players or AI.
● The Pawn represents the
physical location, rotation,
etc. of a player or entity within
the game.
● A Character is a special type
of Pawn that has the ability to
walk around.

GameInstance
GameMode
Pawn Class
HUD Class
PlayerController
Class
GameState
Class
PlayerState
Class
A PlayerController is the interface
between the Pawn and the human
player controlling it.
● The PlayerController decides
what to do and then issues
commands to the Pawn (e.g.
"start crouching", "jump").
● Putting input handling or other
functionality into the
PlayerController is often
necessary.
● The PlayerController persists
throughout the game, while the
Pawn can be transient.

GameInstance
GameMode
Pawn Class
HUD Class
PlayerController
Class
GameState
Class
PlayerState
Class
The GameInstance is a class who’s
state persists switching of levels,
game modes, pawns etc. Where
classes like GameMode or
PlayerController are being reset
and data stored in those classes is
removed.

GameInstance
GameMode
Pawn Class
HUD Class
PlayerController
Class
GameState
Class
PlayerState
Class
A PlayerState is the state of a
participant in the game, such as a
human player or a bot that is
simulating a player. Non-player AI
that exists as part of the game
would not have a PlayerState.

GameInstance
GameMode
Pawn Class
HUD Class
PlayerController
Class
GameState
Class
PlayerState
Class
The GameState contains the state
of the game, which could include
things like the list of connected
players, the score, where the
pieces are in a chess game, or the
list of what missions you have
completed in an open world game.

GameInstance
GameMode
Pawn Class
HUD Class
PlayerController
Class
GameState
Class
PlayerState
Class
The HUD is the base object for
displaying elements overlaid on the
screen. Every human-controlled
player in the game has their own
instance of the AHUD class which
draws to their individual Viewport.

Object
Actor
Pawn
Character
Base building blocks in the Unreal Engine
Any object that can be placed into a level
Subclass of Actor and serve as an in-game avatar
Subclass of a Pawn that is intended
to be used as a player character

CharacterMovementComponent
Pawn
Character
Default Pawn
Wheeled
Vehicle
CapsuleComponent
SkeletalMeshComponent
Etc.
VehicleMovementComponent
SkeletalMeshComponent
PhysicsHandle
Etc.
SpectatorPawn
DefaultPawnMovementComponent
StaticMeshComponent
CollisionComponent, Etc.
Controller
1-to-1 Relationship

How about programming
interactivity for VR?

Programming VR Interaction with Blueprints
Blueprints in Unreal Engine is a complete visual scripting system based on the
concept of using a node-based interface to create interactions from within Unreal
Editor.

Programming VR Interaction with Blueprints
Learning Blueprints through Content Examples

Hey!! We need AUDIO for VR too!!

UE4 – Audio for VR
Ambient Sound Actors in VR
Ambient Sound Actor can be used for
many purposes such as ambient
looping sounds and non-looping
sounds. Generally, the Ambient
Sound Actor conforms to the real
world where the closer you are to a
sound, the louder it will appear.

Sound Properties
You can assign a sound asset from
the Details panel by selecting an
asset from the Sound settings drop-
down menu or by highlighting a sound
asset in the Content Browser and
clicking the button.

Attenuation Properties
Attenuation is the ability of a sound
to decrease in volume as the player
moves away from it.
It is advisable to use Sound
Attenuation objects whenever
possible, if for no other reason than
to give broad control over the
settings for many Actors.

New: Stereo Spatialization
3D spatialization is now possible for
stereo audio assets.
The 3D Stereo spread parameter
defines the distance in game units
between the left and right channels
and along a vector perpendicular to
the listener-emitter vector.

Audio Volume
Audio Volumes allow you to control
and apply various sounds in your
level as well as provide an avenue
to create compartmentalized audio
zones where you can control what is
heard inside and outside of the
volume.

Additional toolsets in Unreal Engine to enhance VR:
Complete state of the art suite of AI Tools.

Additional toolsets in Unreal Engine to enhance VR:
Complete set of tools for animation and animation retargeting

84
VR with Unreal Engine is a lot of fun.
Remember to go to check out the UE4
VR learning resouces for more great
info.
luis.cataldi@epicgames.com

Intro to VR with Unreal Engine

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Intro to VR with Unreal Engine (20)

Recently uploaded (20)

Intro to VR with Unreal Engine