Introduction to Virtual/Augmented Reality and Telepresence

MM-806

(Fall 2024)

Ivan Sutherland's HMD (1968)	Modern HMDs (Today)

General Information

Instructor: Pierre Boulanger
Tel: 780-492-3031
Email: pierreb@cs.ualberta.ca

URL: www.cs.ualberta.ca/~pierreb
Office: 411 Athabasca Hall
Office hours: By appointment only.

Lectures will start on September 7

All lectures will be delivered vitually

Every Tuesday and Thursday, from 13:00-14:20

o Please register in advance for the lectures using the zoom link

o After registering, you will receive a confirmation email about joining the meeting.

All lectures will be recorded.

TAs: Ningbo Zhu

Course Description

Virtual reality and augmented reality can provide an immersive environment where many scenarios can be simulated. For example, manufacturing and engineering tasks, medical planning and training, art and design, rehabilitation, Physics, Biology and Chemistry concept exploration, and many others can benefit from a virtual reality environment. This course focuses on the challenges of setting up a user-friendly virtual reality scene where users can interact intuitively and naturally. The use of interactive techniques and sensor-based devices, such as haptic and head-mount display, in creating a virtual environment for scientific analysis, visualization exploration and Tele-presence, as well as how mobile users can participate in these applications, will be discussed.

Prerequisites

Some knowledge of computer graphics and multimedia systems

Assignments

Assignments must be submitted electronically to VRARMM806@gmail.com. Most of the assignments will be using Unity 3D.

There will be five problem sets. Don't be misled by the few points assigned to homework grades in the final grade calculation. While the grade that you get on your homework is at most a minor component of your final grade, working on the problems is a crucial part of the learning process and will invariably have a significant impact on your understanding of the material

Course Project

A group project will culminate in a final ten-page report in IEEE format and a presentation at a day workshop. Progress and checkpoints before the last due date will count toward the final grade.

Course Grade

The final grade for the course is based on our best assessment of your understanding of the material and your commitment and participation. The problem sets and final projects are combined to give a final grade:

ACTIVITIES	Weight
Final Project	50%
Assignments (5 x 10%)	50%

Lecture Notes

Course calendar.
TOPICS	Slides	Extras
First Class Class Overview	Class Overview VR-Intro
Introduction o Definition of Virtual Reality, Augmented Reality, Virtualized Reality, and Telepresence o Generic Configuration of Virtual Reality Systems (image rendering systems, sound rendering systems, haptic rendering systems, communication systems, physical modelling systems, etc.) o A brief overview of the applications of Virtual Reality (in remote robotic control, medicine, e-commerce, communication, in industrial design)	VR-Intro
History of VR	VR-History
Short Review of Computer Graphics o Overview and Transformations	VR-Graphics	Installing Unity 3D CAMERAS AND EFFECTS in Unity 1. Cameras 2. Image Effects: Overview GEOMETRY IN UNITY 1. Meshes 2. Mesh Renderers and Mesh Filters Assignment 1: Due October 6
o Lighting and Shading	VR-Graphics	LIGHTING IN UNITY 1. Unity 5 Lighting Overview 2. Unity 5 - Lighting and Rendering 3. Lights 4. Using Skyboxes RENDERING AND SHADING in Unity . The Standard Shader 2. Materials 3. Textures 4. A Gentle Introduction to Shaders 5. Using detail textures for extra realism close-up 6. Frame Debugger
Introduction to Human Vision	VR-Human Vision	Group Description:Due October 6
Visual Rendering Systems o Essential characteristics of human visual perception o Stereoscopic display systems (LCD, CRT, HMD, flat, panoramic, hemispheric screens, etc.) o Detailed analysis of advanced visualization systems o New display devices such as real-time auto-stereogram and retinal writing o Cinematic display	VR-Displays	Assignment 2:Due October 20
Digitizing the Human Body o Three-dimensional position sensors (magnetic, ultrasound, photogrammetric, mechanical, and inertial sensors)	VR-Tracking	VR/AR Term Projects By October 20, I would like each team to send me the following information: o A project titles o The name of each member of the team o A detailed description of the project o A diagram of the VR/AR interface functionality o A preliminary system design o An estimated timeline of the execution of the project A list of the equipment/data required Once I get this, I will meet with each team individually and discuss the logistic of the project.
o Digitizing body motion and forces (DataGlove, CyberGlove, PowerGlove, DHM Dexterous Hand Master, etc.)	VR-Tracking	Assignment 3:Due October 31
o Review of student projects	List of Projects
Sound Rendering Systems o Essential characteristics of human auditory perception o Synthesis of 3D sound (convolvotrons, Beachtrons and Acoustetrons, simple 3D sound) o Unity 3D Sound Spatializer	VR-Sound	Assignment 4:Due November 13
Haptic Rendering Systems o Human requirements for good haptic perception o Tactile and force feedback (difference between tactile and force feedback, various tactile feedback systems, haptic rendering, force feedback systems) o Combination of tactile and force feedback.	VR-Haptic
Computing Architectures Graphic requirement of human perception vs. VR graphic engines The underlying architecture of a VR graphic engine (graphic pipeline, graphic cards for PC, etc.) Detailed analysis of modern graphics hardware Highly parallel VR graphic engines Distributed virtual reality systems	VR-Hardware
Modelling for VR o Geometric Modeling (geometric primitives, scene hierarchies, constructive geometry, etc.) o Cinematic model (object motion, collision detection, navigation models, motion hierarchies, etc.) o Physical Modeling (gravity, collision, deformable model, surface texture, etc.) o Behavioural Modeling (Artificial life, responsive model, etc.) o Model segmentation (segmentation in cells, LOD, etc.) o Modelling real-life from sensors	VR-Modeling
VR Programming o Java 3D o Vizard Toolkit o Unity 3D o MiddleVR	VR-Programming
Human Factors in VR and Augmented Reality	VR-Human-Factors	Assignment 5: Due December 5
Augmented Reality	Augmented Reality
Tele-presence and Cinematic VR	VR-Telepresence
Applications of VR and AR o Medicine and rehabilitation (surgery, anatomic simulator, remote surgery, hybrid systems) o VR games o Arts (virtual actors, virtual museum, virtual music, virtual theatre) o Virtual product design (CAD display, process simulation, virtual prototyping) o Robotic (robot and virtual reality, design of robots, robot programming, supervisory control, the Mars Rover) o Teaching systems (military training 'SIMNET', NASA training systems, flight simulators) o Virtual teleconferencing systems	VR-Future
Students' Presentations and Demos		Project demos start the week of December 13
Final Report		Due no later than December 19
Recommended Reference Books o Handbook of Virtual Environments: Design, Implementation, and Applications. 2nd Edition. K. S. Hale, K. M. Stanney, Eds. 2014. o 3D User Interfaces: Theory and Practice. D. A. Bowman, E. Kruijff, J. J. LaViola, and I. Poupyrev. 2014. o Understanding Virtual Reality: Interface, Application, and Design. W. R. Sherman and A. B. Craig. 2012. Developer References o Unity3D Interface Overview o Unity3D Manual o Oculus Best Practices Guide Research Articles o Stereoscopy and the Human Visual System o Optometric and Perceptual Issues with Head-mounted Displays Other Online Resources o Reddit/Oculus o A developer's perspective on immersive 3D computer graphics o Road to VR o Oculus Connect keynotes o Blur Busters Internet Links o https://www.edx.org/professional-certificate/virtual-reality-vr-app-development o http://stanford.edu/class/ee267/ o http://moodle.epfl.ch/course/view.php?id=6841 o https://www.evl.uic.edu/aej/528/ o http://allvirtualreality.com/learning/university-college-school-virtual-reality-vr-courses-training.html