Transcript 3D Tele-Collaboration over Internet2 Herman Towles, UNC-CH representing members of the National Tele-Immersion Initiative (NTII) ITP 2002 Juan-les-Pins, France 06 December 2002

3D Tele-Collaboration
over Internet2
Herman Towles, UNC-CH
representing members of the
National Tele-Immersion Initiative (NTII)
ITP 2002
Juan-les-Pins, France
06 December 2002
1
NTII Collaborators & Co-authors
• University of North Carolina at Chapel Hill
– Wei-Chao Chen, Ruigang Yang, Sang-Uok Kum, and Henry Fuchs
• University of Pennsylvania
– Nikhil Kelshikar, Jane Mulligan, and Kostas Daniilidis
• Brown University
– Loring Holden, Bob Zeleznik, and Andy Van Dam
• Advanced Network & Services
– Amela Sadagic and Jaron Lanier
2
Clear Motivation to Provide
• Higher Resolution
• Larger, more immersive Field-of-View
• Participants at Accurate Geometric Scale
• Eye Contact
• Spatialized Audio (Group settings)
• More Natural Human-Computer Interfaces
3
Related Work
• Improved Resolution & FOV
– Access Grid – Childers et al., 2000
– Commerical, multi-channel extensions of ‘1-camera to 1-display’
• Gaze-Awareness
– MONJUnoCHIE System – Aoki et al., 1998
– Blue-C Project - Kunz and Spagno, 2001-2002
– VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002
• 3D Reconstruction/New Novel Views
– CMU’s Virtualized Reality Project – Narayanan, Kanade, 1998
– Visual Hull Methods – Matusik, McMillan et al, 2000
– VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002
• Human Computer Interfaces
– T-I Data Exploration (TIDE) – Leigh, DeFanti et al., 1999
– VisualGlove Project - Constanzo, Iannizzotto, 2002
4
XTP: ‘Xtreme Tele-Presence
UNC ‘Office of the Future’
Andrei State 1998
5
Research Snapshots
6
Presentation Outline
• Motivation and Related Work
• NTII Tele-Collaboration Testbed
– Acquisition and 3D Reconstruction
– Collaborative Graphics & User Interfaces
– Rendering & Display
– Network
• Results
• Future Challenges
7
Scene Acquisition & Reconstruction
• Foreground: Real-Time Stereo Algorithm
– Frame Rate: 2-3 fps (550MHz Quad-CPU) - REAL-TIME!
– Volume: 1 cubic meter
– Resolution: 320x240 (15K-25K foreground points)
• Background: Scanning Laser Rangefinder
– Frame Rate: 1 frame in 20-30 minutes - OFFLINE!
– Volume: Room-size
– Resolution: More data than you can handle!
Composite
Live Foreground & Static Background
8
Real-Time Foreground Acquisition
• Trinocular Stereo Reconstruction Algorithm
– After background segmentation, find corresponding pixels in
each image using MNCC method
– 3D ray intersection yields pixel depth
– Median filter the disparity map to reduce outliers
• Produce 320x240 Depth Maps (1/z, R,G,B)
=
+
Images courtesy of UPenn GRASP Lab
9
UNC Acquisition Array
Seven Sony Digital 1394 Cameras – Five Trinocular Views
Five Dell 6350
Quad-Processor
Servers
10
Stereo Processing Sequence
Camera Views
Disparity Maps
3 Views of
Combined
Point Clouds
Images courtesy of UPenn GRASP Lab
11
Collaborative Graphics & User I/F
12
Shared 3D Objects
• Scene Graph Sharing
– Distributed, Common Scene
Graph Dataset
– Local Changes, Shared
Automatically with Remote
Nodes
• Object Manipulation
with 2D & 3D Pointers
– 3D Virtual Laser Pointing
Device
– Embedded magnetic tracker
– Laser beam rendered as part
of Scene Graph
– One event/behavior button
13
Rendering System Overview
14
3D Stereo Display
• Passive Stereo & Circular Polarization
– Custom Filters on Projectors
– Lightweight Glasses
– Silvered Display Surface
• Front Projection
– Usable in any office/room
– Ceiling-mounted Configurations
• Two Projector Stereo
– 100% Duty Cycle
– Brighter & No flicker
– Permits multi-PC Rendering
15
View-Dependent Rendering
• HiBall 6DOF Tracker
– 3D Position & Orientation
– Accurate, Low latency &
noise
– Headband-mounted Sensor
– HiBall to Eyeball Calibration
• PC Network Server
16
Rendering Configurations
• One PC Configuration (Linux)
– Dual-channel NVIDIA graphics
• Three PC Configuration (Linux)
– Separate left & right-eye rendering PCs w/NVIDIA graphics
– One PC used as network interface, multicasts depth map stream to
rendering PCs
• Performance – 933MHz PCs & GeForce2
– Interactive Display Rates of 25-100fps
– Asynchronous updates of 3D Reconstruction (2-3Hz) & Scene Graph
(20Hz)
• Newest Rendering Configuration 10-20X
– 2.4GHz, GeForce4, Multi-Threaded, VAR Arrays
17
Network Considerations
• All Tests over Internet2
• Data Rates of ~20-75 Mbps from Armonk,
NY and Philadelphia into Chapel Hill
– 320 x 240 Resolution
– Up to 5 Reconstruction Views per site
– Frame Rates 2-3 fps
• TCP/IP
• Latency of 2-3 seconds typical
18
Presentation Outline
• Motivation and Related Work
• NTII Tele-Collaboration Testbed
– Acquisition and 3D Reconstruction
– Collaborative Graphics & User Interfaces
– Rendering & Display
– Network
• Results
• Future Challenges
19
Results
‘Roll the Tape’
20
Summary
• ‘One-on-One’ 3D Tele-Immersion Testbed
• Life-size, view-dependent, passive stereo
display
• Interact with shared 3D Objects using a
virtual laser pointer
• Half-Duplex Operation today
• Operation over Internet2 between Chapel
Hill, Philadelphia and Armonk
• Audio over H.323 or POTS
21
Future Challenges
• Improved 3D Reconstruction Quality
– Larger Working Volume, Faster Frame Rates – 60 cameras
– Fewer Reconstruction Errors (using structured light and
adaptive correlation kernels)
• Reduce System Latency and Susceptibility
to Network Congestion
– Pipelined architecture
– Shunt Protocol (between TCP/UDP and IP layers) that allows
multiple flows to do coordinated congestion control
• Full Duplex Operation
• Unobtrusive Operation
– No headmounts, No eyeglasses!
22
Thank You
Research funded by
Advanced Network and Services, Inc. and
National Science Foundation (USA)
23
UPenn Acquisition Array
Fifteen Sony Digital 1394 Cameras – Five Trinocular Views
24
System Overview
25
Past Experiments
With Collaboration
2D Video + Audio
Advanced
Scene Bus Data
UNC
Chapel Hill
Armonk, NY
3D Data + 2D Images
w/o Collaboration
UPenn
3D Data + 2D Images
Philadelphia
2D Video + Audio
Advanced
Armonk, NY
UNC
Chapel Hill
3D Data + 2D Images
26