VAMA

NAVIGATION MENU

• PRIMARY OBJECTIVE OF VAMA
• MOTION CAPTURE OF MUSICAL PERFORMANCE
• MOTION CAPTURE OF DANCE PERFORMANCE
• VIRTUAL PRODUCTION FOR MUSICAL PERFORMANCE
• VIRTUAL REALITY PERFORMANCE CAPTURE
• VAMA PHASE 1: RESEARCH AND DEVELOPMENT
• MOTION CAPTURE WITH WITNESS CAMERA SYSTEMS
• 360 VIDEO VIRTUAL REALITY
• 3D MODELLLING FOR THE ARTS
• BEHIND THE SCENES VIDEOS
• LEAD ARTIST BIO
• PROJECT PARTNERS
• PROJECT CONTACT

PRIMARY OBJECTIVE OF VAMA

TO FEATURE TOOLS AND TECHNIQUES THAT ARE LOW-COST AND CAN BE REPLICATED BY ARTISTS AND ARTIST ORGANIZATIONS. ADVANCED RESEARCH AND DEVELOPMENT WITH A FOCUS ON MOTION CAPTURE USING MOCAP SUITS, FACIAL RECOGNITION FOR EXPRESSIONS AND LIP SYNC, 180 AND 360 VIDEO CAPTURE FOR VR, AND VIRTUAL PRODUCTION WITH PROJECTIONS.

Motion capture using Rokoko body suit and face capture with iPhone and iOS app for violin performance by Victoria Yeh

MOTION CAPTURE OF MUSICAL PERFORMANCE

Software and Hardware Used for Musical Performance Motion Capture:

• Rokoko Smartsuit Pro2
• Smartgloves 2
• Face Capture Headrig with iPhone 11 or newer
• Coil Pro
• Rokoko Studio Pro Software

Part 1 Musician Duo Performance Capture: Motion capture of pianist Daniel Vnukowski and violinist Vera Zmiyiwsky using the Rokoko Smart Suit body suit for motion capture (MoCap) in a home, without a special dedicated studio space. This MoCap can occur anywhere musicians are, without a purpose built motion capture stage, including interiors or exteriors, such as a theatre stage or even a living room in a home.

Part 2 Musician Duo Performance Capture: 3D Avatars of Daniel and Vera from the data captured using the MoCap Rokoko suits. Additional data of the instruments can be animated or captured seperately but there is no low cost soluation for this at this time.

Part 1 Solo Violinist Capture: Victoria Yeh performs a solo violin performance showcasing the Smat Suit 2 with facial capture using the Rokoko system. Performed live on stage at Meaford Hall - simulating a pre-concert MoCap proof of concept.

Part 2 Solo Violinist Capture: Victoria Yeh as a Cyborg Meta Human using the MoCap from Part 1 of her live on stage performance.

Part 1 Vocal Performance Capture: Hanne Poulsen has her vocal performance captured with the Rokoko Smartsuit 2 and Facial Capture with a live vocal recording on stage at the Meaford Hall. This capture similates a live on stage performance or pre-performance recording of an artist without requiring an additional space or day to achieve this MoCap recording.

Part 2 Vocal Performance Capture: Hanne Poulsen is re-created as a Mime Avatar using the MoCap data captured in Part 1. This could be used to live stream to an avatar during a live performance or to create new avatar based performances for backgrounds or special video segments during a concert of theatre production.

MOTION CAPTURE OF DANCE PERFORMANCE

Part 1 Duo Dance Performance Capture: Motion capture of dance duo from Active Arts Dance performing with Rokoko Smart Suit 2, live on theatre stage at Meaford Hall.

Part 2 Duo Dance Performance Capture: MoCap data used to render live Meta Human characters of a zombie and warrior, showcasing what could be possible for a live dance performance virtualization.

VIRTUAL PRODUCTION FOR MUSICAL PERFORMANCE

Software and Hardware Used in Capturing Virtual Production for Music Performance:

• Blackmagic Pocket Cinema Camera 4K with 35mm Lens
• Digital Projector and screen (existing equipment in theatre space)
• Digital animated backgrounds prepared in DaVinci Resolve editing software
• Post-production using Avid Media Composer of the filmed footage

Victoria Yeh filmed live on stage peforming her original song at Meaford Hall. By utilizing the existing projection system, virtual backgrounds can be projected to create a Virtual Production set-up that offers a setting without the cost of building a set or using printed backdrops.

VIRTUAL REALITY PERFORMANCE CAPTURE

180 Degree and 360 Degree VR capture of live musical and live dance performances.

Software and Hardware Used for Virtual Reality (VR) Performance Capture:

• GoPro Fusion 360 Camera (current model if GoPro Max)
• GoPro Omin 8K 360 Camera (6 Camera Rig)
• Mistika VR stitching software for 360 equirectangular renders
• Blackmagic Pocket Cinema Camera 4K with a 220 degree ultra fisheye lens (for 180 degree VR footage)
• Mistika VR stitching software for 180 equirectangular renders
• Avid Media Composer for editing and rendering 1:1 ratio equirectangular footage
• Boris FX VR Continuum plug-in for viewing footage as VR and for titles in VR mode

Filming 180 video using a 220 degree ultra fisheye lens on a Blackmagic PCC 4K camera with a dancer performing live on stage at Meaford Hall.

360 video capture with GoPro Omni rig to offer full 360 degree of viewing angles. Filmed live on stage at Meaford Hall with a competition dance crew from Active Arts.

180 video capture with a 220 degree film lens and two difference camera angles for editing purposes.

180 Video using the 220 degree ultra fisheye captures the trio of Hanne Poulsen, Will Smythe, and Matthew Shepperdson.

Victoria Yeh captured with 360 Video using the GoPro Fusion camera 2 lens system. Victoria plays live on stage at Meaford Hall with a projected background.

VAMA PHASE 1: RESEARCH AND DEVELOPMENT

Research and development activities that include LiDAR laser scanning, photogrammetry 3D modelling, virtual reality capture with 360 Video, and augmented reality experiences

MOTION CAPTURE WITH WITNESS CAMERA SYSTEMS

Motion capture of live performance such as dancers and musicians for Augmented Reality applications

The 3D Render Test showcases the process of capturing a live dance performance with a single camera, extracting Motion Capture data using Deep Motion software, and applying the Motion Capture data to 3D avatar versions of the dancers, based on scans using Polycam, for a final 3D animated render of the dance performance. This 3D render data from the animation can be used for Augmented Reality applications or for mixed uses in Virtual Reality 3D viewing.

Featured dancers in this sample are: Erin McAndrews, Deanne Service, Alyssa Service.

Software Utilized in this production: Polycam for Mac iOS using an iPad Pro for Photogrammetry and LiDAR Scanning of Performers and Performance Space, Deep Motion for Mac OS for Motion Capture data capture, Blender for 3D Animation and rendering of 3D test.

360 VIDEO VIRTUAL REALITY

360 Video Virtual Reality capture for virtualization of live performances and live music concerts

Captured using a 360 Video Camera (GoPro Fusion) and a 180 Degree Lens on a Blackmagic Pocket Cinema Camera, the footage can be converted into a 360 Video experience.

Software used for the creation of 360 Video samples: Mistika VR for Mac OS for stitching, Avid Media Composer for Editing of 360 Video.

3D MODELLLING FOR THE ARTS

3D scanning and laser scanning of art to create 3D models for Augmented Reality applications such as exhibition and previewing

Utilizing a portable iPad Pro and Polycam software, media artist Tom Strnad captured both LiDAR laser scans and Photogrammetry photo captures in 3D of the dimensional art. It was discovered that for small to medium sized dimensional art, Photogrammetry is the best solution for offering a 3D model of the artwork and for Augmented Reality applications.

Test A showcases the ceramic and pottery based dimensional artwork by artist Anke Lex. The scans were taken at the artists' studio and were captured in available sunlight. The video showcases the Polycam Photogrammetry capture and the resulting Augmented Reality placement of the works inside a home for possible gallery viewing or testing of placement by a client.

Test B features metal dimensional sculptures by artist, Michael Pocock. The video showcases the Photogrammetry 3D capture using Polycam and the application or Augmented Reality to place the pieces in a garden setting for AR based testing of placement for these dimensional artworks or for virtual gallery display outdoors.

Software used for dimensional art virtualization: Polycam for Mac iOS using iPad Pro, Adobe Aero for Augmented Reality application, Blender for finalizing 3D models (OBJ files).

BEHIND THE SCENES VIDEOS

Capturing Performance: A Step-By-Step guide to 3D Scans and Motion Capture to capture a performance for Augmented Reality presentations.

Capturing 360 Video: A Step-By-Step guide to capturing a live performance using 360 Video Cameras or Fisheye Lens and creating a VR 360 Video experience.

LEAD ARTIST BIO

Tom Strnad is a honours graduate of York University's Film & Video Program with a specialization in Production with over 20 years of filmmaking experience. Tom's films have screened around the world including festival premieres at SXSW, TIFF, VIFF, FIPA and CINEQUEST. Tom is a recipient of funding and grants from Telefilm Canada, Ontario Creates, Canada Council for the Arts, Ontario Arts Council and the National Film Board of Canada. As a producer and director, Tom is a member of the Academy of Canadian Film and Television, CMPA and DOC Institute. As a picture editor and sound designer, Tom is a full member of DGC Ontario and an associate member of the Canadian Cinema Editors (CCE). Tom is a lead digital strategist for the South Georgian Bay Media Association Inc. and actively teaches digital media arts throughout his home region of Grey County, Ontario.

PROJECT PARTNERS

We acknowledge the support of the Canada Council for the Arts

PROJECT CONTACT

• tom@mountaingoatfilm.com