Transcript Institutional Management Review August 30/31, 2004 Managing CEBAF Accelerator Operations Andrew Hutton Thomas Jefferson National Accelerator Facility Operated by the Southeastern Universities Research Association for.

Institutional Management Review
August 30/31, 2004
Managing CEBAF Accelerator Operations
Andrew Hutton
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004 /SC-PAC2001-6.19.01
Outline
CEBAF Accelerator Characteristics
Response to Hurricane Isabel
Accelerator Achievements in FY04
G0 Experiment completed
Hypernuclear Experiment completed
HAPPEx-He and HAPPEx-II initial runs completed
Operations Metrics
Preparing for Upcoming Challenges
Path forward – new Operations Vision
Summary
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Brief Description of CEBAF
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004 /SC-PAC2001-6.19.01
Continuous Electron Beam Accelerator Facility
0.6 GeV linac
(20 cryomodules)
1497 MHz
67 MeV injector
(2 1/4 cryomodules)
1497 MHz
Gain switched
lasers @
499 MHz,
Df = 120
A
B
RF separators
499 MHz
C
B
A
Pockels cell
C
A
Gun
B
C
Double sided
septum
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
CEBAF Capabilities
 CEBAF delivers independent beams to all three Halls
 Energy – must be multiple of linac energy
 1, 2, 3, 4, or 5-pass to any Hall
 All Halls can simultaneously have 5-pass beam
 Current – fully independent
 Halls A & C take up to 140 μA
 Hall B takes up to 50 nA (and down to 100 pA!)
 Polarization – orientation of longitudinal polarization depends on Hall
energy due to precession
 At least 50% of experiments want longitudinal polarization
 An increasing number of experiments want “parity quality” beams
 Small helicity-correlated change in current, position, angle, polarization
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Dynamic Operational Requirements
 Unlike a storage ring, the operating conditions of CEBAF are changed
frequently based on User needs
 In FY02, FY03, FY04 there were:
6
9
3 linac energy changes
21
15
5 pass changes in Hall A
8
6
5 pass changes in Hall B
4
10
4 pass changes in Hall C
25
30
14 accelerator state changes
 On average, the accelerator state changes about once per operating week
 This does not include special set-ups for Moeller runs, energy
measurements, etc.
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Response to Hurricane Isabel
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004 /SC-PAC2001-6.19.01
Hurricane Isabel
 Isabel arrived ashore as a Category 1 hurricane on September 18, 2003
 Removed electrical power from site for four days – specifically from
CHL so cryomodules warmed up
 Recovery took six weeks
 Aggressive preventive maintenance carried out on almost every
component - improved reliability during the year
 Engineering, SRF Institute, Operations
 Accurate beam set-up provided a solid, reproducible base for
operations
 CASA, Operations
 Launched us into extremely successful year operating period
Details on Poster
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Improving Hurricane Preparedness
 Evaluated back-up power options
 Full back-up power is expensive, requires active management
 Renting seems better (RFP is out)
 Major investment in switchgear and long term contractual obligation
 Decided to implement emergency power loop
 Provides power to critical systems
 Pumps to maintain insulation on cryomodules, valve actuators
 Special funds from DOE awarded June 2004
 Expect completion before next hurricane season (May 2005)
 Interim, temporary solution developed (extension cords, UPS, small
generators, etc.)
 Ready to implement if needed
 Initiated aggressive tree-cutting near to offsite power line
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Tree Clearing near Power Line
Insert Photo Here
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Accelerator Achievements in FY03/4
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004 /SC-PAC2001-6.19.01
Experiment Successes FY03/FY04
 G0 required 40 μA at 31.2 MHz – every 16th bucket filled
 Bunch charge 6.5 times more than original specification
 “Parity quality” beam imposed optics constraints
 Hall A hypernuclear experiment required:
 Energy spread < 3x10-5
 Scheduled in parallel with G0
 HAPPEx-II and HAPPEx-He required:
 Tightest “helicity correlated asymmetries” ever
 Position asymmetries < 2 nm
 Energy asymmetry < 0.6 ppm
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
G0 Parity Quality Beam
Total of 744 hours (103 Coulombs) of parity quality beam
Beam
Parameter
Achieved
“Specs”
(IN-OUT)/2
Charge
asymmetry
-0.14 ± 0.32
ppm
1 ppm
x position
differences
3 ± 4 nm
20 nm
y position
differences
4 ± 4 nm
20 nm
x angle
differences
1 ± 1 nrad
2 nrad
y angle
differences
1.5 ± 1 nrad
2 nrad
Energy
differences
29 ± 4 eV
75 eV
All parityThomas
quality
specs
have
been Facility
achieved!!
Jefferson
National
Accelerator
Andrew Hutton IM Review 2004
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Hypernuclear Experiment Energy Spread
Energy Spread x 10-5
Data from April 21-29
Spec
3x10-5
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
HAPPEX-II
Photon Detector
Signal/Background > 10
Demonstrated feasibility of maintaining
Compton polarimeter background count
rate: <100 Hz / mA at 5mm (10-10)
Electron only
Photon only
Preliminary
New superlattice photocathode
Polarization >85%
CASA and EGG have worked closely with
HAPPEX to meet stringent requirements on
helicity-correlated position differences.
After correcting early problems at source,
the ability to meet helicity-correlated
specifications was demonstrated.
Dx (nanometers)
Figure of Merit improves
by ~30%
(over strained-layer cathode)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
“slug” number
DOE Metrics for FY03
Specific Metric per M&O Contract with
the US Department of Energy
Delivered Physics Research Operations
Goal
Actual
Points Points
possible actual
6019.50 hrs
6646.29 hrs
Accelerator Down (new metric)
< 15%
15.0%
40 39.973
Experimental Equipment Availability
78.4%
89.6%
20
Effectiveness of the Scheduling Process
100%
95.997%
30 wks
30.397 wks
Overall Operational Effectiveness
Total Points
100
100.0
20.0
20 19.199
20
20.0
200 199.172
Metrics for FY03 were excellent
Availability for multi-Hall Physics operation not as good as our Users would
like, but performance better than DOE goal
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
DOE Metrics for October – July FY04
Specific Metric per M&O Contract with
the US Department of Energy
Delivered Physics Research Operations
Goal
Actual
Points Points
possible actual
4853.95 hrs
6566.99 hrs
100
100.0
Accelerator Down
< 15%
11.79%
40
40.0
Experimental Equipment Availability
78.5%
86.8%
20
20.0
Effectiveness of the Scheduling Process
100%
86.19%
20
17.24
26.13 wks
26.24 wks
20
20.00
Overall Operational Effectiveness
Total Points
Post-hurricane
maintenance
extremely effective
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
200 197.24
Hall A septum
Andrew Hutton IM Review 2004
Preparing for Upcoming Challenges
Energy
Parity
Polarization
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004 /SC-PAC2001-6.19.01
Energy Outlook for FY04/05
 Scheduled to deliver 5.75 GeV, 100 kW beams in September 04
 Hurricane reduced accelerating voltage by ~40 MV/turn, 200MeV from
top beam energy
 Predicted RF trip rate will be “high” ~15/hour
 Will make operation of accelerator difficult
 Required to reach goals of experimental program
 Compromise accepted by Users
 Expect RF trip rate to improve when new 12 GeV prototype cryomodule
replaces NL11 (operational by July 05)
 RF trip rate at 5.75 GeV will be acceptable ~10/hour
 Refurbishment of existing cryomodules would provide 6 GeV operation by
July 06 with acceptable trip rate (~10/hr)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Parity Violation Experiments at CEBAF
Helicity-correlated asymmetry specifications
1999
2007
Experiment
Physics
Asymmetry
Max run-average
helicity correlated
Position
Asymmetry
Max run-average
helicity correlated
Current Asymmetry
HAPPEX-I
13 ppm
10 nm

1.0 ppm

G0
2 to 50 ppm
20 nm

1.0 ppm

HAPPEX-He
8 ppm
3 nm

0.6 ppm

HAPPEX-II
1.3 ppm
2 nm

0.6 ppm

Lead
0.5 ppm
1 nm
0.1 ppm
Qweak
0.3 ppm
20 nm
0.1 ppm
4 ± 4 nm
Achieved for G0
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
-0.14 ± 0.32 ppm
Andrew Hutton IM Review 2004
Superlattice Cathode

Polarization 87% (recent User measurement)
 Typical polarization from traditional strained layer material ~75%

Quantum Efficiency ~ 1%
 Typical QE of traditional strained layer material 0.2%

Analyzing power 4%
 Factor 3 better than strained-layer material in the lab
 Smaller intensity and position asymmetries on beam
 Improvement not yet seen in experimental data

Installed on Accelerator 5/17/04
 Successfully operated for experimental program (HAPPEx)
 Lifetime was not good – attributed to bad vacuum
 NEG pumps replaced in present accelerator shutdown

Will be standard for all experiments
Matt Poelker and Maud Baylac (Injector)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
New Laser Clean Room for Injector
Insert Photo
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Path Forward
New Operations Vision
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004 /SC-PAC2001-6.19.01
Drivers for Change
 Our accelerator operations are second to none
 Biennial Workshop on Accelerator Operations initiated by JLab
 Our Control System is one of the world’s best managed
 Karen White is regularly invited to lecture on managing software
But, we believe in continuous improvement (really)
 Four main drivers for change:
 Main Control Room (MCC) needed renovating
 Aging flooring, improve air conditioning, bad ergonomics, needed
better integration of ODH alarms, fire alarms and access controls
 ORACLE database available, needed EPICS integration
 Full accelerator model will be available soon and we should plan for it
 Must prepare to commission and operate 12 GeV
Goal – use these drivers to revamp operations processes
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
MCC Upgrade
 Layout modified to provide:
 Crew Chief oversight of operators
 Station for Program Deputy accessible to support staff
 Responsible for program oversight for two-week period
 Stations for Principle Investigators
 Direct special machine set-ups and beam studies
 Improved teaching environment for operators
 Discussion area with “mirrored” computer screen
 Existing tall racks replaced with desk height work stations
 Multiple small monitors replaced with few large screens
 Better visibility of access controls (personnel safety system)
 Integrated beam diagnostics displays
Managed by Mike Spata and Tom Oren (Operations)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Old MCC
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
New MCC (three weeks later)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Operations Vision
 Primary focus – are beams meeting User requirements?
 Secondary focus – is each region performing correctly?
 Provides common structure for thinking about accelerator operations,
database, accelerator model, HLA, new installations
 Hierarchy based on the accelerator layout
 Usual focus on kinds of element (magnets, steering, RF) - WBS
 Change to “functional segmentation system” derived from beambased set-up
 Highest level derived from User requirements
 Halls, energies, currents, polarizations, beam specifications
 Increases focus on diagnostics to ensure that beam meets specifications
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Highest Hierarchical Level
 Defined standard set of beam specifications for Users
 User may negotiate tighter specs when proposing experiment (TAC)
 Experiment schedule defines which experiments are running
 User requirements are known – import requirements from database
 Use these requirements to configure the accelerator
 Derive set-points for the machine set-up
 Energy, current, polarization . . . . .
 Integrate beam specs with instrumentation to monitor compliance
 Energy spread, spot size, helicity-correlated effects . . . . .
 Highest level display shows if beam specifications are being met, and if
not, which parameters are out of tolerance
Managed by Hari Areti (Experiment Coordinator)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Beam Specifications
DC Beam Properties
Parameter
Energy
Energy Spread
Nominal Value
Max.
Deviation
from
nominal
Measurement
Accuracy
Stability
(~ 1 shift)
Measurement
Device(s)
0.6 GeV < E < 5.75
GeV
< 10-4
< 10-4
< 10-4
Harps
< 10-4
2.0 * 10
< 5.0 * 10-5
SLI(Hall A)
SLI (Hall A)
E/E<5.0 * 10
-5
-5
Monitoring
Device(s)
Current
10 nA< I < 120 µA
< 10 %
< 1%
<5%
Unser,BCMs
Faraday Cup
(Hall B)
BCMs
Beam Position at target
(x,y)
Any within 3 mm of
optics axis
<0.2 mm
< 50 µm
< 100 µm
BPMs
BPMs
Beam Size at target
(x,y)
50 µm < σx and y <
200µm
< 20 %
< 10 µm
25 %
Harps
<50 µr
<100 µr
BPMs
BPMs
<50 µr
50%
BPMs
BPMs
Moeller
Compton
(Hall A)
Beam direction at target
Beam divergence at
target
< 1 mr wrt optics axis
σx and y < 200µr
Polarization
>75%
Beam Halo
1 KHz/µA @ 3mm
< 100µr
There are also AC Beam Properties and Helicity-correlated Beam Properties
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Example
Parameter
Current:
Measurement
Nominal
Tool
Value
ibc1h01
< .11 uA
.21 uA
.31 ppm
X: 1 um
X: 92 um
X: 172 ppb
Y: 22 um
Y: 102 um
Y: 182 ppb
 Experiment beam request
2x01
Stability
Helicity Correlated
Value
X: 32 um
X: 112 um
X: 192 ppb
2C20
Y: 42 um
Y: 122 um
Y: 202 ppb
3C02
X: 52 um
X: 132 um
X: 212 ppb
Y: 62 um
Y: 142 um
Y: 222 ppb
3C04
X: 98 um
X: 96 um
X: 94 ppb
Y: 97 um
Y: 95 um
Y: 93 ppb
273 ppm
Position:
 Experimental requirements
Energy:
harps 1-4
253 Gev
263 MeV
Energy Spread:
sli1c12
284 MeV
294 MeV
Bleedthrough:
smrposa
< 305 %
Polarization:
No Data Entered
---
Spot Size:
ipm1h04x
ipm1h04y
326 mm
336 mm
Background:
background tool
25 %
346 mm
356 mm
366 ppm
376 ppm
418 mr
428 mr
438 ppm
448 ppm
Angular
ipmFacility
1h04z
Thomas Jefferson NationalDivergence
Accelerator
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
at Target:
398 mr
408 mr
Andrew Hutton IM Review 2004
Diagnostics
 Each beam specification is mapped to at least one diagnostic
 Diagnostics are of three main types
 Run-time monitors that function at all times
 BPMs, Synchrotron light monitors, OTR, beam loss monitors,
experiment detectors, Compton back-scattering
 Invasive monitors that cannot take full power
 Screens, Harps
 Infrequent monitors that require special set-up
 Moeller and Mott measurements, current and energy calibrations
 Long term goal is to monitor all beam specifications to required accuracy
non-invasively over complete range of operating conditions
 Diagnostics must be integrated with software packages and tightly coupled
to User-specific beam specifications
Managed by Arne Freyberger (CASA)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Database
 Master copy of all information will be held in a database
 “Authoritative source”
 All other instances will reference database to obtain current value
 Vital for maintaining control over machine changes
 Information will be assigned to one of two databases, depending on the
frequency of change
 We already have a dynamic, “run-time” database – EPICS
 Adding master database for static and slowly changing data - ORACLE
 Databases will eventually manage all accelerator data
 Database will be the information source for everyone
 Engineering support groups, operations, controls
Managed by Theo Larrieu (Controls)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Impact on Control System
 Robustness requires nested checks at all levels of software
 Example of making tools robust:
 BPM passes self-check
 Feedback system uses model to determine best corrector, BPM
configuration based on Optics
 System measures BPM response to corrector kicks
 Compare corrector-BPM response to model
 Downstream elements monitored to ensure feedback system is
performing desired function
 Providing all necessary hooks requires global re-examination of Control
System at every level
 Device drivers, low-level applications (Matt Bickley)
 High level applications, communication protocols (Brian Bevins)
Managed by Karen White (Controls)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Optics Model-Database Relationship
 Model obtains input from
 ORACLE
 Component layout derived from Survey group
 Component specifications from Engineering Support Groups
 Impacts all Support Groups
 Vehicle for configuration control
 Global settings
 Configured from User Requirements
 Off-line optics calculation by CASA
 Result goes into Oracle database
 Set points calculated for dipoles, quadrupoles, RF
 Model server output is available to all high level applications
 Eventually, all high level applications will be model driven
Managed by Yves Roblin (Controls)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Optics Model Improvements
 Model requires accurate knowledge of magnets over wide energy range
 We have ~2000 magnets, not all properly characterized
 Uncertainty due to dipole gradients from remanant fields
 Additional uncertainty from orbit-related focusing errors due to badly
characterized “gold orbit”
 Diagnostics added in spreaders and recombiners
 Beam-based measurements being used to measure errors
 Requires special optics (weak focusing)
 Data taken over last year, dedicated period at end of last run
 Evaluated during the summer accelerator down
 Will be used for setting up the machine in September
Managed by Mike Tiefenback (CASA) and Tommy Hiatt (Engineering)
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Implementation Status
 MCC refurbishment complete (MCC visit during Tour)
 Planning, implementation and result are fantastic success
 Requirements Document for Control System being written
 “Executive Summary” complete
 Ensures coherency of Vision across Division
 Some aspects already implemented
 Model under active development
 Guiding principles of the Vision will be integrated into new and upgraded
software for years to come
 Expect positive impact on operations within six months
 Changes the way we do business for years to come
 Prepares operations for commissioning and operating 12 GeV
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004
Summary
 FY03 operations were excellent, FY04 were outstanding
 G0, an incredibly difficult experiment, got more data than requested,
beam exceeded all specifications
 Hypernuclear experiment received beam with outstanding energy
spread – run average ~2.2 X 10-5
 Even more impressive as experiments ran in parallel
 HAPPEx tight parity quality specs achieved
 Availability for Physics much improved since hurricane due to additional
maintenance that was performed
 New Vision will improve Operations in coming months
 Motivates and energizes multiple Groups
 Prepares for commissioning and operating 12 GeV Upgrade
Thomas Jefferson National Accelerator Facility
Operated by the Southeastern Universities Research Association for the U.S. Depart. Of Energy
Andrew Hutton IM Review 2004