Transcript Slide 1

JSOC SDP Agenda
•
•
Significant Level 4 requirements
SDP Architecture Decomposition
–
–
–
–
–
–
–
–
–
–
•
•
•
•
•
•
•
•
•
•
•
Datacapture System
Datacapture System Components
Pipeline Processing System
Pipeline Processing System Components
Software Configuration Items
Hardware Configuration Items
Data Archive
Data Distribution
Performance Analysis
Heritage
SDP Network Design & Security
Test approach
Implementation plan
Deliverables hardware/software/documentation
Development environment/CM Process
Sustaining support pre-launch and post-launch
Identify organization(s) providing support
Identify types of support: vendor licenses, h/w, etc.
Trade Studies and Prototyping Efforts since PDR
Risks & Mitigations
Procurement status
GS CDR May 2005
Page 1
DMR – JSOC SDP Requirements
• 8000.2.4 Science Data Processing, Archiving and
Distribution
•
Each SOC shall provide the necessary facility, software, hardware and
staff to receive, process, archive and distribute the science data
generated by its instruments.
My Documents\DMR_SOC_Req.ppt
GS CDR May 2005
Page 2
JSOC (Stanford Science Data Processing) Configuration
DDS
SDP
Pipeine Processing
Datacapture System
Switch
CISCOS YSTE MS
ethernet
Switch
File Server
CISC OS YSTEMS
Data Base Server
Analysis Cluster
heartbeat
Passive
Active
Disk arrays
Server
Server
Fibre Channel Switch
[etc.]
ID C
Disks
Tape Robot
Pipeline Processor
Tape Robot
LAN
jim\My Documents\hardware_config3.vsd
GS CDR May 2005
Page 3
JSOC Data Capture System Components
DDS
lev0 “module”
Operator
Utility Libraries
Primary
Storage Disks
GUI
FTP
Disks
Data Capture
Interface
Record Cache
DSDS
Processing
History Log
Data Storage &
Distribution System
Database Server
GS CDR May 2005
Tape Server
Tape
Drive
Page 4
JDAT network
GS CDR May 2005
Page 5
JSOC Pipeline Processing System Components
Pipeline
Operato
r
Pipeline
processing
plan
JSOC Science
Libraries
Processing
script, “mapfile”
PUI
Pipeline User
Interface
Pipeline Program, “module”
List of pipeline
modules with
needed datasets for
input, output
Utility Libraries
SUMS Disks
DRMS Library
Record
Manage
ment
Keyword
Access
Link
Manage
ment
Record Cache
Data
Access
SUMS
Storage Unit
Management System
DRMS
Processing
History Log
Data Record
Management System
SUMS
Tape
Farm
Database Server
GS CDR May 2005
Page 6
S/W Configuration Items
•
SUMS
•
DRMS
•
PUI
•
Event Mgr
•
Science Libs
•
Pipeline Util Libs
•
Export Support
•
Lev0
•
Lev1
•
Standard Products
GS CDR May 2005
Page 7
H/W Configuration
GS CDR May 2005
Page 8
H/W Configuration (cont)
GS CDR May 2005
Page 9
Telemetry Data Archive
•
Telemetry data is archived twice
•
The Data Capture System archives tlm files for offsite storage
•
Archive tapes are shipped to the offsite location and verified for
reading
•
A feedback mechanism will be established to ack the SDP that a tape is
verified or that another copy needs to be sent
•
The Data Capture System copies tlm files to the Pipeline Processing
System
•
The Pipeline Processing System archives tlm data for local storage and
acks the SDP when it is successful
•
Only when the SDP has received positive acks on both archive copies
does it inform the Data Capture System to include the tlm file in the .arc
file to the DDS, who is now free to remove the file from its tracking
logic
•
The lev0 data on the Pipeline Processing System is also archived
GS CDR May 2005
Page 10
JSOC Data Export System
Grid
VSO
Researcher B
CoSEC
Grid Adaptor
VSO Adaptor
CoSEC Adaptor
Space Weather
API
FITS
VOTable
plain
(FITSz)
(CDF)
(JPEG)
General Public
Keywords
Range
Overview
New/Avail
Statistics
Format
Selecte
d
Data
Records
Custom
Keywords
SOHO
GONG
WCS-x
Utilities
DRMS
GS CDR May 2005
Package
Drilldown
Browse
Researcher A
Search
Script Access
Preprocess
Filter
Mask
Track
Auto
Update
Redistribute
Notify
Filename
Schema
20090401_1958.fits
AR12040_171.vot
magCR2801:lon064lat20S.txt
Compress
gzip
jpeg
low-resolution
Page 11
Performance Analysis
•
AIA/HMI combined data volume
2 PB/yr = 60 MB/s
– read + write
x2
– quick look + final
x2
– one reprocessing
x2
– 25% duty cycle
x4
2 GB/s (disk)
0.5 GB/s (tape)
•
NFS over gigabit ethernet (50-100MB/s/channel)
1 – 4 GB/s
– 4 – 8 channels per server, 5 servers (today)
•
SAIT-1 native transfer rate (25-30 MB/s/drive)
0.5 – 0.6 GB/s
– 10 SAIT-1 drives per library, 2 libraries (today)
GS CDR May 2005
Page 12
Heritage
•
MDI ground system design and implementation
•
MDI production processing for 9+ years
•
MDI sustaining engineering for h/w and s/w upgrades
•
MDI “lessons learned” folded into JSOC SDP design
GS CDR May 2005
Page 13
Stanford/Lockheed Connections
Stanford
DDS
World
JSOC Disk array
Front End
Firewall
Sci W/S
Pipeline
Router
Router
Firewall
Firewall
NASA
AMES
LMSAL
1 Gb
Private line
MOC
CMD
Router
Display
Pipeline
Router
Router
JSOC Disk array
Firewall
GS CDR May 2005
LMSAL W/S
“White”
Net
Firewall
Page 14
Test Approach
•
Telemetry data is simulated with MDI data packaged in SDO VCDUs
•
Various telemetry pathologies can be created to validate telemetry processing
•
The DDS is simulated by packaging the VCDUs into files and injecting them into
the SOC front end using the DDS/SOC protocol
•
The lev0 processing is validated by comparing input and output images
•
Timing analysis and throughput studies will be performed to evaluate h/w and
s/w tradeoffs
•
Instrument sunlight tests will be used to validate the lev1 processing
•
Each s/w library and subsystem has a regression test suite that is run to verify
each new CM release
•
Full end to end testing is accomplished via the mission I&T plan
GS CDR May 2005
Page 15
Integrated Schedule
JSOC Science Data Processing (SDP) / DDS I&T Start Dates
Delivery of Flight EGSE SDP
Prototype SDP System Ready
JSOC Network Ready
DDS-JSOC Testing
GSRT#2- Science Data Processing Test (Ka-band)
HMI Connectivity, Dataflow, Retransmissions Test
AIA Connectivity, Dataflow, Retransmissions Test
GSRT#3-Mission Operations& RF Communications Test
GSRT#4-Fully Integrate Ground System
Ground System Freeze
GSRT#4-Launch Readiness Test
GS CDR May 2005
June 2005
Dec 2005
Dec 2006
Dec 2006
Jan 2007
Feb 2007
Feb 2007
Mar 2007
Mar 2007
Jan 2008
Feb 2008
Page 16
Implementation Plan
•
TBD – Get from Carl
GS CDR May 2005
Page 17
Deliverables
•
EGSE hardware
•
EGSE software
•
TBD – Get from Carl (from contract)
GS CDR May 2005
Page 18
Configuration Management & Control
•
•
•
•
•
Capture System
–
Managed by JSOC CCB
–
Controlled in CVS
SUMS, DRMS, PUI, etc. Infrastructure
–
Managed by JSOC CCB after launch
–
Controlled in CVS
PUI Processing Tables
–
Managed by HMI and/or AIA Instrument Scientist
–
Controlled in CVS
Level 0,1 Pipeline Modules
–
Managed by HMI and/or AIA Instrument Scientist
–
Controlled in CVS
Science Analysis Pipeline Modules
–
Managed by program author
–
Controlled in CVS
GS CDR May 2005
Page 19
CM with CVS
GS CDR May 2005
Page 20
Prototype/Trade Study Sequence
•
“MDI Heritage” software for EGSE Science W/S (currently running)
•
Study to separate keywords from image data in a keyword Database
•
MDI database optimization study
•
MDI internal data set representation study
•
Storage management trade study (commercial HSM vs. MDI heritage)
•
Archive media study to replace MDI Ampex robotic units
•
Storage Area Network (SAN) / Network Attached Storage (NAS) trade study
•
Location of offsite media storage
GS CDR May 2005
Page 21
Risks and Mitigations
•
Physical location of JSOC
- Tracking through Stanford Management
- Rent space off-campus
•
Cost risk in assumed rate of computer technology development
- Advances already made
- Wide margin of costing allowed
•
Data Base Performance
- Prototyping underway
- Design alternatives
- Higher performance equipment
GS CDR May 2005
Page 22
Development & Test Plans and Procurement Schedule
•
HMI and AIA Data EGSE installed
–
–
•
Mar, 2005 onward
June 2005
JSOC Capture System
–
–
–
•
Prototype for I/F testing with GS
Version 2 to support flight inst.
Purchase computers
Support DDS testing
Final system installed
Summer 2006
Fall 2006
Spring 2007
JSOC SDP Infrastructure, SUMS, DRMS, PUI
–
–
Prototype testing of core system
Fully functional
June 2005
Dec, 2005
•
Purchase computers for JSOC
Jan, 2007
•
Infrastructure Operational
April, 2007
•
Data Product Modules
Jan, 2008
•
Test in I&T and with DDS,MOC as called for in SDO Ground System schedule
•
During Phase-E
GS CDR May 2005
•
Add media and disk farm capacity in staged plan, half-year or yearly increments
•
First two years of mission continue Co-I pipeline testing support
Page 23
GS CDR May 2005
Page 24
Test Approach
•
Unit test
•
Sub System Integration
•
Regression Tests
•
Performance Test
•
System Integration
•
I &T with DDS/MOC
•
On-line test results and history
GS CDR May 2005
Page 25
Procurement Status
–
–
–
GS CDR May 2005
2004 - 2005: First 2 Years
•
Procure development system with most likely components (e.g. tape type, cluster vs SMP, SAN vs NAS,
etc)
•
Modify pipeline and catalog infrastructure and implement on prototype system.
•
Modify analysis module API for greater simplicity and compliance with pipeline.
•
Develop calibration software modules.
•
Complete level 0 processing code to support HMI/AIA instrument testing.
2006 - 2007: Two years prior to launch
•
Complete Level-1 analysis development, verify with HMI/AIA test data.
•
Populate prototype system with MDI/TRACE data to verify performance.
•
Procure, install, verify computer hardware.
•
Implement higher-level pipeline processing modules with Co-I support
During Phase-E
•
Add media and disk farm capacity in staged plan, half-year or yearly increments
•
First two years of mission continue Co-I pipeline testing support
Page 26