Document 7713000
Download
Report
Transcript Document 7713000
INFN-GRID-WP2.4:
Computing Fabric
& Mass Storage
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
Bologna
Catania
CNAF
G.P. Siroli
P. Mazzanti
C. Rocca
E. Cangiano
Tecnologo
A. Chierici
L. Dell’Agnello
F. Giacomini
P. Matteuzzi
C. Vistoli
S. Zani
Genova
Lecce
LNL
G. Chiola
G. Ciaccio
G. Aloisio
M. Cafaro
Z. Zzzz
L. Depaolis
S. Campeggio
E. Fasanelli
L. Berti
M. Biasotto
M. Gulmini
G. Maron
N. Toniolo
Padova
Roma 1
Torino
S. Balsamo
M. Bellato
F. Costa
R. Ferrari
M. Michelotto
I. Saccarola
S. Ventura
D. Anzellotti
C. Battista
M. De Rossi
F. Marzano
S. Falciano
A. Spanu
E. Valente
A. Forte
1
Terminology
PC
PC network
+ midleware
LAN
LA
N
PC Farm network
+ midleware
LA
N
= PC Farm/Fabric
PC Cluster
= GRID
LA
N
LA
N
LA
N
LA
N
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
2
Why this WP?
Commodity components like PCs and LANs are mature to
form inexpensive and powerful Computing Fabric
Computing Fabrics located in different sites are integrating
to form a Computational/Data GRID
But
– How to design a fabric of 1000s nodes balancing
computing power and efficient storage access?
– How to control and monitor the basic system
components?
– How to “publish” the monitored values to the GRID
monitor system?
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
3
Computing Fabric WP
This WP wants to address the mentioned problems adding a
technological tracking task to follow and to test, with real use
case, the evolution of the basic constituents of a fabric.
WP Break Down
– Fabric Design (INFNGrid Wp 2.4)
Overall architecture and fabric setup
LAN and SAN (System Area Network) technologies
Communication protocols for high speed network fabrics
Storage Systems
Microprocessor Technology
– Fabric Management (DataGrid WP4)
Configuration management and automatic software installation
System monitoring
Dynamic System Partition
Problem management
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
4
Fabric Design - INFNGrid WP 2.4 - Institutions
Dipartimento di Scienze dell’Informazione - Universita’ di
Venezia
Dipartimento di Scienze dell’informazione - Universita’ di
Genova
Ingegneria Informatica – Universita’ di Lecce
8 sezioni INFN, 1 Laboratorio Nazionale
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
5
Fabric Management - DataGrid WP4 - Institutions
CERN
Konrad Zuse Zentrum (Berlin)
Kirchhoff Institute (Heidelberg)
IN2P3 (Lyon)
INFN
Nikhef
RAL
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
6
Fabric Design Detailed Program (2000-2001)
Fabric Architecture
– Network topologies
– Data Server connections and network file systems
– System break down
Interconnection Networks
– 100/1000 Ethernet
– Myrinet
– Infiniband
Communication protocols for high speed network fabrics
Storage Systems
– Ultra SCSI (160/320/…)
– Ultra and Serial ATA
– Storage Area Network (SAN)
Fibre Channel
Microprocessor Technology
– Dual slim processors
– IA64
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
7
Fabric Design Deliverables
Deliverable Description
D2.4.1
Requirements gathering from
fabric users
D2.4.2
Survey of existing fabric/cluster
architectures, architectural issues
to design a fabric of commodity
pcs
D2.4.3
INFN toolkit to design and realize
local fabrics
D2.4.4
Integration of the DataGrid fabric
management softw are tools
D2.4.5
Integration of the Fabric
Management into the GRID
environment
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
Delivery Date
3 Mo n t h s
6 + 6 Months
12 Months
24 Months
36 Months
8
Fabric Design Milestone
Milestone
M 2 .4 .1
Description
NFS connection topology: disk
SITE
LNL
Start
Date
9/2000
End Date
3/2001
server based
M 2 .4 .2
NFS connection topology:
CNAF/BO 9/2000
3/2001
Microprocessor Technology: dual CNAF/BO 9/2000
3/2001
distributed disks
M 2 .4 .3
slim processors
M 2 .4 .4
Storage Systems: Fibre Channel -
PADOVA
9/2000
3/2001
PADOVA
PADOVA
LECCE
5/2001
7/2001
11/2000 3/2001
9/2000
1 2/ 2 0 01
SCSI Ultra 160/320
M 2 .4 .5
M 2 .4 .6
M 2 .4 .7
Storage Systems: Serial ATA
Microprocessor Technology: IA64
Interconnection Networks: Myrinet
+ IA64
M 2 .4 .8
Interconnection Networks: Gigabit GENOVA/ 9/2000
Ethernet and efficient communication ROMA1
5/2001
protocols
M 2 .4 .9
M 2 . 4 . 10
Setup of a cluster of more than
100 nodes
Interconnection Networks:
LNL
1/2001
1 2/ 2 0 01
LNL
5/2001
1 2/ 2 0 01
All
All
2002
2003
2 00 3
2 00 3
Infiniband
M 2 . 4 . 11
M 2 . 4 . 12
Installation of the INFN fabrics
TIER 0 Integration
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
9
DATA GRID Fabric Management WP4
Automatic Software Installation and Management (OS and
Applications)
Configuration Management
System Monitoring
Problem Management
Local Authorization Services and Grid Integration
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
10
WP4 Fabric Managemet Deliverables
Requirements document and survey of existing tools and
technologies (month 6)
A configuration and installation management demonstrated to
work on a cluster of more than 100 nodes (month 12)
A fully deployed service level monitoring system for a computer
centre. Hooks to provide remote requests for meta information
like policies and quality measures, to allow schedule decisions
(month 24)
A fully integrated system to accept remote resource requests in
the form of tape mounts r jobs to run and provide monitoring
information about progress og requests, and final accounting
report back to the sender (month 36)
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
11
NFS Disk Servers Based Farm (Legnaro)
LNL is testing this farm module (with PIII at 450 MHz)
Computational Nodes:
- Dual PIII 800 MHz (40+40 SI95)
-Mem 512 MB
Ethernet Switch:
- 10 FE
- 1 Uplink GE
Disk Server:
-Dual PIII 800 MHz
- Mem 512 MB
- SCSI Adapters
N1
FE
ANIS
Boot
Server
N2
N10
S
W
I
T
C
H
GE
Farm Module
DISK
SERVER
This Farm has been funded
By Comm Calcolo for LNL
Off-line computation (gr. 2/3)
Requests for 2000:
- 2 Raid SCSI Controllers
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
10 Ml
12
LNL PC Farm (phase I – May 200)
N1
Farm Module I
FE
ExtremeNetworks
Summit 48
48 Fe
2 Ge
N2
S
W
I
T
C
H
GE
2 ULTRA 160 SCSI
ADAPTEC 39160
DISK
SERVER
N10
FrontEnd
(ANIS)
ASUS P3BF
440 BX chipset
1 PIII 450 MHz
PCI 33/32
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
Problemi con
Memoria SDRAM
SuperMicro PIIIDME
840 chipset
1-2 PIII 600 MHz
PCI 33/32-66/64
13
LNL PC Farm (phase II – Nov 2000)
N1
Farm Module I
FE
ExtremeNetworks
Summit 48
48 Fe
2 Ge
N2
S
W
I
T
C
H
GE
Raid Controller U3
Compaq
SmartArray 5300
DISK
DISK
SERVER
SERVER
N15
SuperMicro 370DLE
ServerWorks III LE chipset
2 PIII 800 MHz
512 Mbyte
PCI 33/32-66/64
Case minitower
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
Enclosure Compaq 4354R
14 dischi 36GB SCSI U3
500 Gbyte Storage Array
Case Test: SuperMicro SC810
14
Low Price 1U case example: SM SC810
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
15
CMS Fabric at LNL (2001)
N10
N2
N10
N2
N1
N10
N2
SWITCH
Farm Module
Farm Module
N10
N2
SWITCH
Farm Module
N1
FE
SWITCH
N1
FE
FE
FE
HP2524
N1
SWITCH
Farm Module
HP8000
FastIron 4000
GigaEthernet Switch
DISK
SERVER
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
DISK
SERVER
DISK
SERVER
DISK
SERVER
16
LNL Farm
32 PCs, PIII 600 MHz, 3 Farm modules,
120 Gbps GE/Cu switch, 500 Gbyte disk
Used for:
- Sadirc/CMS Event Builder prototypes
- Data Analysis Production for LNL exps
- NFS and topological tests
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
17
NFS Distributed Disks based Farm (CNAF)
N1
FE
ANIS
Boot
Server
N2
N10
S
W
I
T
C
H
Farm Module
- Topology tests
- dual slim processors (rack mounted) tests
- remote file systems (nfs, ams, etc. ) tests
Requests for 2000:
- 10 dual slim processors
- Fast ethernet switch
- 10 SCSI disks 36/72 GB
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
90 Ml
10 Ml
20 Ml
-----120 Ml
18
Storage Systems: FC and SCSI road map
SCSI
Fibre Channel
2004
Ultra640
640 MB/sec
2003
8/10Gb
1000 MB/sec
2002
Ultra320
320 MB/sec
2001
2000
Ultra3/Ultra160
160 MB/sec
1999
2Gb
200 MB/sec
400MB/sec (full duplex)
1998
Ultra2
80 MB/sec
1997
1Gb
100 MB/sec
1996
Wide Ultra
40 MB/sec
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
1995
19
Storage Systems: Fiber Channel/SCSI evaluation (PD)
Arbitrated loop (AL) FC
N1
Switched FC
Requests for 2000
- 4 Disk Servers
- 1 8 ports FC Switch
- 4 FC adapters
- 2 Raid Arrays
- 4 Ultra/160
SCSI disks
N1
FC
N2
N3
30 Ml
20 Ml
15 Ml
30 Ml
8 Ml
-----103 Ml
F
C
S
W
T
SCSI
N1
N4
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
20
Storage Systems: Serial ATA (Padova)
Serial
ATA
N1
Requests for 2001
- 4 ATA Cntr. 5 Ml
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
21
Interconnection Networks: Gigabit Ethernet (Genova)
Communication Protocols for high speed networks:
- Communication System GAMMA
- Use of Programmable GigaEthernet NICs
- 1000 Base T GigaEthernet (on copper) reliability evaluation
- Implementation of efficient parallel/distributed RAID systems
Requests for 2000
- 8 PC
- 1 12 ports Geth Switch
- 8 GEth NICs
TOTAL
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
24 Ml
30 Ml
16 Ml
-----70 Ml
22
Interconnection Networks: Myrinet
Full-duplex 2.5+2.5 Gigabit/second links, switch ports, and
interface ports
Flow control, error control, and “heartbeat” continuity
monitoring on every link
Low latency, cut-through crossbar switches
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
23
Interconnection Networks: Myrinet (Lecce)
N1
M
Y
R
I
N
E
T
N2
DISK
SERVER
Myrinet based farm
- Disk Server based
- Distributed disks based
N10
N1
N2
N10
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
M
Y
R
I
N
E
T
Requests for 2000
- 10 Biproc Comp. Nodes
- 1 16 ports Myrinet Switch
- 10 Myrinet NICs
- 10 disks 72 GB
- 1 disk server
- Rack + cables
TOTAL
50 Ml
8 Ml
35 Ml
35 Ml
10 Ml
5 Ml
-----143 Ml
24
Interconnection Networks: InfiniBand (I)
TODAY
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
25
InfiniBand (II)
The Infiniband Model
Legacy host architecture
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
26
InfiniBand (III)
What?
– Initial single link signaling rate of 2.5Gbaud
Means unidirectional transfer rate of 250MB/sec with a
theoretical full duplex rate of 500MB/sec
– Initial support for single, 4, and 12 wide link widths
– Point to point switched fabric
– Message based with multicasting support
Fibre Channel
Memory
CPU CPU
CPU CPU
Memory
Controller
PCI-X
Host
Bridges
Multi
Stage
Switch
SCSI
Link
HCA
Gig. Ethernet
Links
TCA
I/O Controller
HCA- Host Channel Adapter
TCA - Target Channel Adapter
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
27
Interconnection Networks: InfiniBand (LNL)
Simple test system (4 servers + Storafe Area Network +
Network) for 2001 is possible
Early access to the products
Test Beds
Requests 2001 (valuations) Comm V (Sadirc2000)
• 4 servers
50 Ml
• 1 IBA Switch
20 Ml
• IBA Adapters 10 Ml
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
28
Microprocessor Technology: Intel Itanium (IA64)
Requests for 2001
– Padova: Investigation IA64
4 PC IA64
40 Ml
– Lecce: IA64 in Myrinet
10 Dual IA64
100 Ml
1 Disk Server +
disks
16 Ml
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
29
Resources
Sites
Persons
FTE
Materials
(Ml)
Ital. Trav.
(Ml)
Inter.Trav
(Ml)
2000
2001
2000
2000
Legnaro
5
2.6
10
80
Padova
7
2.1
103
45
CNAF
6
1.7
120
-
Bologna
2
0.6
-
-
Lecce
6
1.4
143
116
Torino
1
0.2
-
-
Genova
2
1.0
70
-
Catania
3
0.7
-
-
Roma 1
7
1.5
*
*
TOTAL
39
11.8
446
161
241
G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000
2001
2001
30