Transcript Grids for Real-time and Streaming Applications I-Light Symposium IUPUI September 22 2005 Geoffrey Fox Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401 http://grids.ucs.indiana.edu/ptliupages/presentations/ [email protected] http://www.infomall.org.

Grids for Real-time and
Streaming Applications
I-Light Symposium IUPUI
September 22 2005
Geoffrey Fox
Computer Science, Informatics, Physics
Pervasive Technology Laboratories
Indiana University Bloomington IN 47401
http://grids.ucs.indiana.edu/ptliupages/presentations/
[email protected]
http://www.infomall.org
1
Four Data Streaming Application Areas

Data Assimilation applied to link the data deluge
(satellites, sensors, seismometers) in real time to large
scale parallel simulations
• Use in Earthquake Science

Department of Defense (and Homeland Security) have
built the Global Information Grid with a target
architecture NCOW (Network Centric Operations and
warfare)
• They submit no jobs; rather stream data to brokers from they
are filtered and distributed
• Includes their rather dated distributed simulation HLA


Audio-Video Conferencing implemented with services
and Grid messaging
Hand-held Grid linking PDA/cell-phones to Grids
2
Key Concepts





Grid of Grids (System of Systems) allows “library”
approach to composing Grids
Service Oriented architectures (Web or Grid services)
are attractive for many/most distributed systems
There are many applications that are NOT best
considered as jobs and files (classic Grid) but rather as
streams and filters (services)
Services exchanging messages becomes
Services exchanging streams (sets of messages)
Publish-Subscribe messaging gives better QoS and
management than point to point messaging with
negligible performance loss
3
Typical use of Grid Messaging in NASA
Sensor Grid
Grid Eventing
Datamining Grid
GIS Grid
4
GIS and Sensor Grids






OGC has defined a suite of data structures and services
to support Geographical Information Systems and
Sensors
GML Geography Markup language defines
specification of geo-referenced data
SensorML and O&M (Observation and Measurements)
define meta-data and data structure for sensors
Services like Web Map Service, Web Feature Service,
Sensor Collection Service define services interfaces to
access GIS and sensor information
Grid workflow links services that are designed to
support streaming input and output messages
We are building Grid (Web) service implementations of
these specifications for NASA’s SERVOGrid
5
Grid Eventing via NaradaBrokering

The Scripps Orbit and Permanent Array Center
(SOPAC) GPS station network data published in RYO
format is converted to ASCII and GML
6
Grid Messaging with Grid of Grids Architecture
Filter or
Datamining Grid
Sensor Grid
Post before
Processing
Database
Archives
DoD Principles
Post after
Processing
Narada
Brokering
Web Feature Service
Notify
Subscribe
HPSearch
Manages
WS-Context
Stores dynamic data
WFS
(GIS data)
GIS Grid
Geographical
Information System
7
Earthquake Faults from the WMS Client
8
WMS uses WFS that uses data sources
<gml:featureMember>
<fault>
<name> Northridge2 </name>
<segment> Northridge2
</segment>
<author> Wald D. J.</author>
<gml:lineStringProperty>
<gml:LineString
srsName="null">
<gml:coordinates>
-118.72,34.243 118.591,34.176
</gml:coordinates>
</gml:LineString>
</gml:lineStringProperty>
</fault>
</gml:featureMember>
`
WMS
le
ec
tio
n
Fe
a
ol
tur
eC
eC
oll
Ge
tF
ea
e
r
tu
r
tu
a
Fe
a
Fe
et
G
tur
e
Client
io
ct
n
s
ad
i l ro ]
a
R [a-b
Railroads
WFS Server
Hi
River [a-d]
Bridge [1-5]
ry
SQL Query
ue
LQ
SQ
SQ
L
gw
ay
[1
2-
Q
ue
18
ry
]
Interstate
Highways
Rivers
Bridges
90
9
Google Map Client
Archived
Real Time
Databases with
SERVOGrid Faults
Sensor
Grid
WFS1
WFS2
Google Central
HTTP
Google Map
Client
Helper
Services
UDDI
SOAP
DoD and Homeland Security can in a crisis combine custom
geo-referenced data with that available from hundreds of
thousands of computers from Microsoft, Yahoo and Google
Just build simple services using Interoperability standards!10
Real Time GPS
and Google Maps
Subscribe to live GPS
station. Position data
from SOPAC is
combined with Google
map clients.
Select and zoom to
GPS station location,
click icons for more
information.
11
Google maps
can be
integrated with
Web Feature
Service
Archives to
filter and
browse seismic
records.
Integrating
Archived Web
Feature Services
and Google Maps
12
Google Maps
as Service
accessed from
our WMS
Client
13
Google plus GIS Grid Integrated with Los Alamos
Critical Infrastructure Simulations for DHS
Natural Gas Layer
Energy Power Layer
14
Traditional NaradaBrokering Features
Multiple protocol
transport support
In publish-subscribe
Paradigm with different
Protocols on each link
Transport protocols supported include TCP, Parallel TCP
streams, UDP, Multicast, SSL, HTTP and HTTPS.
Communications through authenticating proxies/firewalls &
NATs. Network QoS based Routing
Allows Highest performance transport
Subscription Formats
Subscription can be Strings, Integers, XPath queries, Regular
Expressions, SQL and tag=value pairs.
Reliable delivery
Robust and exactly-once delivery in presence of failures
Ordered delivery
Producer Order and Total Order over a message type. Time
Ordered delivery using Grid-wide NTP based absolute time
Recovery and Replay
Recovery from failures and disconnects.
Replay of events/messages at any time. Buffering services.
Security
Message-level WS-Security compatible security
Message Payload options
Compression and Decompression of payloads
Fragmentation and Coalescing of payloads
Messaging Related
Compliance
Java Message Service (JMS) 1.0.2b compliant
Support for routing P2P JXTA interactions.
Grid Feature Support
NaradaBrokering enhanced Grid-FTP. Bridge to Globus GT3.
Web Services supported
Implementations of WS-ReliableMessaging, WS-Reliability
and WS-Eventing.
15
Transit Delay (Milliseconds)
Mean transit delay for message samples in
NaradaBrokering: Different communication hops
9
8
7
6
5
4
3
2
1
0
hop-2
hop-3
hop-5
hop-7
100
1000
Message Payload Size (Bytes)
Pentium-3, 1GHz,
256 MB RAM
100 Mbps LAN
16
JRE 1.3 Linux
Standard Deviation for message samples in NaradaBrokering
Different communication hops - Internal Machines
0.8
hop-2
hop-3
hop-5
hop-7
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1000
1500
2000
2500
3000
3500
4000
4500
5000
Message Payload Size
(Bytes)
17
Average Video Delays for one broker –
Performance scales proportional to number of brokers
Latency ms
Multiple
sessions
One session
30 frames/sec
# Receivers
18
Consequences of Rule of the Millisecond

Useful to remember critical time scales
•
•
•
•
1) 0.000001 ms
– CPU does a calculation
2a) 0.001 to 0.01 ms – Parallel Computing MPI latency
2b) 0.001 to 0.01 ms – Overhead of a Method Call
3) 1 ms
– wake-up a thread or process (do simple things
on a PC)
• 4) 10 to 1000 ms – Internet delay


2a), 4) implies geographically distributed
metacomputing can’t in general compete with parallel
systems
3) << 4) implies a software overlay network is possible
without significant overhead
• We need to explain why it adds value of course!

2b) versus 3) and 4) describes regions where method and
message based programming paradigms important
19
OGC or
OGSA-DAI
Grid Services
Databases
and/or Sensors
Grid
Grid Data
Assimilation
HPC
Simulation
Analysis
Control
Visualize
This Type of Grid
integrates with
Parallel computing
Multiple HPC
facilities but only
use one at a time
Many simultaneous
data sources and
sinks
Distributed Filters
massage data
For simulation
SERVOGrid (Complexity) Computing Model
20
DoD Core Services and WS-* plus OGSA I
NCOW Service or Feature
WS-* Service area
GGF
Others
A: General Principles
Use Service Oriented Architecture
Core Service Model
(#1)
Build Grids
Services
on
Grid of Grids Composition
Web
Industry Best Practice
(IBM, Microsoft …)
Strategy
for
legacy
subsystems and modular
architecture
B: NCOW Core Services (to be continued)
CES 1: Enterprise Services
Management
WS-* #8 Management
GGF #6: Management
CIM
CES 2: Information
Assurance(IA)/Security
WS-* #5
WS-Security
GGF #7,
Grid-Shib, Permis Liberty
Alliance etc.
CES 3: Messaging
WS-* #2, #3
CES 4: Discovery
WS-* #6
CES 5: Mediation
WS-* #4 workflow
CES 6: Collaboration
VO
CES 7: User assistance
WS- * #10
JMS, MQSeries,Streaming
/Sensor Technologies
Treatment
of
systems.
Transformations
GGF VO.
XGSP,
Shared
Service ports
Portlets,
NCOW
Interfaces
Legacy
Data
Web
JSR168
Capability
21
DoD Core Services: WS-* and OGSA II
NCOW Service or Feature
WS-* Service area
GGF
Others
B: NCOW Core Services Continued
CES 8: Storage (not real-time
streams)
GGF #4 Data
NCOW Data Strategy
CES 9: Application
GGF #2
Best
Practice
in
building
Grid/Web
services
GGF #5;
GiG itself; Ad-hoc
networks important
Environmental
Services ECS
Control WS-*, #9
Resource Infrastructure
C: Key NCOW Capabilities not directly in CES
Meta-data
WS-* #7
Semantic Grid
Globus MDS
Semantic
Annotation
Web;
Resource/Service
Matching/Scheduling
Distributed
GGF scheduling work Extend
computer
Scheduling
and extended to networks
scheduling to networks
SLA’s (GGF # 3)
and data flow
Sensors (real-time data)
OGC Sensor standards
GIS
OGC GIS standards
22
Collaboration Grid
WS-Context
HPSearch
UDDI
Narada
Broker
Audio Mixer
Video Mixer
Narada
Broker
WS-Security
Gateway
XGSP Media
Service
Narada
Broker
Gateway
SharedWS
Transcoder
Thumbnail
Replay
Record
Annotate
SharedDisplay
WhiteBoard
23
GIS
TV
Chat
Video Mixer
Webcam
GlobalMMCS SWT Client
24
e - Annotation
Player
Archieved
stream list
Archived stream
player
Real time
stream list
Annotation
player
/ WB
e - Annotation
Whiteboard
Real time stream
player
25
PDA Download video (using 4way video mixer service)
Desktop
PDA
26
NB Features Released 2005-2006











Production implementations of WS-Eventing, WS-RM and WSReliability.
• WS-Notification when specification agreed
SOAP message support and NaradaBrokers viewed as SOAP
Intermediaries
Active replay support: Pause and Replay live streams.
Stream Linkage: can link permanently multiple streams – using in
annotating real-time video streams
Replicated storage support for fault tolerance and resiliency to
storage failures.
Management: HPSearch Scripting Interface to streams and
brokers (uses WS-Management)
Broker Topics and Message Discovery: Locate appropriate
Integration with Axis2 Web Service Container (?)
Support of IBM MQSeries functionality and Legacy MQSeries
Systems as a Grid of Grids gateway
Better Security tracking endless changes of WS-Security
27
High Performance Transport supporting SOAP Infoset
What to Remember









Grids are Services exchanging Messages
Developing messaging paradigm for Grids using
Message Oriented Middleware or Software Overlay
Network or Grid Service Bus
Web Service container replaces computer
Service replaces process
A stream is an ordered set of messages
NaradaBrokering replaces MPI with different
applications and different requirements
Service Internet replaces Internet: messages replace
packets
(Sub)Grids replace Libraries in Grid of Grids
Architecture
Use W3C OASIS OGC GGF standards
28
Location of software for Grid Projects in
Community Grids Laboratory






htpp://www.naradabrokering.org provides Web service
(and JMS) compliant distributed publish-subscribe
messaging (software overlay network)
htpp://www.globlmmcs.org is a service oriented (Grid)
collaboration environment (audio-video conferencing)
http://www.crisisgrid.org is an OGC (open geospatial
consortium) Geographical Information System (GIS)
compliant GIS and Sensor Grid (with POLIS center)
http://www.opengrids.org has WS-Context, Extended
UDDI etc.
The work is still in progress but NaradaBrokering is
quite mature
All software is open source and freely available
29