IceCube Project Monthly Report - November 2008
Accomplishments
· Completed commissioning of the IceCube Seasonal Equipment Site and started deep ice drilling on December 3rd, two days ahead of schedule. The quick start-up of the drill resulted in less fuel consumption than originally planned.
· Successfully deployed the first three IceCube strings of the 2008/09 installation season by December 11, five days earlier than last season. The drill is performing well.
· Nearly all of the cargo needed for this season arrived by December 4th.
· Eight cycles of DOM acceptance tests were completed at the South Pole without any major problems. All minor problems were resolved and all DOMs passed.
· With the start of the Pole season and increased activities on the South Pole systems, IC40-string operations during Nov 2008 achieved an overall 98.3% uptime.
· Completed the installation of equipment in the IceCube Laboratory (ICL), for the additional strings being deployed this year.
· Fifteen refurbished DOMHub computers arrived at the South Pole and were installed into the ICL racks. The total count of hubs at Pole is now 79.
· The UW data warehouse capacity increases are nearing completion. The work was slightly delayed due to ongoing reliability issues with the old data disks.
· Offline filtering of the IC40 data and simulation data production are delayed pending establishing stable operations of the UW data warehouse and the capacity enhancements.
Action at Hole 36 on December 6, l-r: fork-lift towing IceTop water transport tank to fill tanks at station 36; red crates with test of DRMs being conducted by Jan Auffenberg from ICL; TOS-1 and drill tower being towed into position for drilling 3rd hold of the season. Photo – Tom Gaisser
Cost and Schedule Performance - The project is 88.2% complete versus the plan of 88.1% complete, as measured using earned value techniques. Remaining contingency is $8.7M and the contingency as a percentage of remaining work is 27.5%. The contingency continues to be carefully managed to ensure the successful completion of the MREFC project with 80 strings plus six Deep Core strings. There was no change to the contingency budget during this month.
The Schedule Variance at the end of October 2008 is a favorable $207K, reduced from $270K at the end of the previous month. The variance is mostly related to the RPSC’s FY2008 schedule performance variance.
The Cost Variance at the end of October 2008 is a favorable $1,459K, an increase from $1,290K at the end of last month. RPSC has a positive cost variance of $554K due to lower than planned labor and On-Ice support charges in FY2008. The $656K favorable variance in Implementation was caused by lower than planned training costs, lagging invoices and labor ramp down ahead of schedule.
Contingency Status and Plans
The contingency % of remaining work has increased from 26% to 27.5% during October 2008. The cost of fuel for the FY2010 season is now anticipated to be less than planned since it will be purchased this month. The cost of fuel in FY2011, the final installation season, will be purchased in one year.
10/31/08 / IceCube Total Project Budget Baseline - Change Log ($K)No. / Description / Total Baseline / Allocated Budget / Allocated Budget Change / Contingency Budget / Budgeted Cost of Work Remaining / Contingency % of Remaining Work
NA / Status as of September 2008 / 275,293 / 266,638 / 0 / 8,656 / 33,315 / 26.0%
NA / Status as of October 2008 / 275,293 / 266,638 / 0 / 8,656 / 31,461 / 27.5%
Risk Assessment & Potential Contingency Adjustments
Item / Estimate ($K)
1. / Potential cost impact of mitigating technical, cost and schedule risks associated with the defined scope of work. Risk assessments are made at WBS-Level 4 to determine the value of the risk exposure as a percent of the cost of work remaining. / $2,523K
2. / Pre-Operations for additional enhancements to the data storage and network systems. / $250K
3. / RPSC FY08 Closeout: Potential return of RPSC authorized budget to IceCube contingency / -$400K
4. / Annual Baseline Review of Project Years 8-10 of all WBSs excluding WBS 1.6 RPSC. / $1,000K
5. / Instrumentation critical path materials for up to 86 strings (purchase and additional five surface to DOM cables and build an additional ~100 DOMs). / $1,350K
6. / Current RPSC estimate of base cost to support the installation of six strings during the 2010/11 drilling season. / $1,340K
7. / The estimate cost to retro IceCube equipment/materials from the South Pole at the end of the project in FY2012. / $825K
8. / Potential cost exposure to additional fuel price increases beyond normal inflation and for transferring the installation of 4 strings from FY2009 to FY2011. / $600K
9. / Potential cost exposure should a full FY2011 drilling season be required resulting in the need to send UW personnel to the South Pole in FY2012 to dismantle the drill equipment/materials and prepare the equipment for retro out of South Pole. / $450K
Total / $7,938K
Available Contingency as of / Oct 31, 2008 / $8,656K
Drill Operation and Installation – The first hole, hole #18, of the 2008 to 2009 season commenced at approximately 10:30 a.m. on Wednesday, December 3rd, two days ahead of schedule. Drilling proceeded smoothly with no major issues and completed at approximately 11:30 p.m. on Thursday, December 4th. String deployment was completed by 3:00 p.m. on Friday, December 5th. Hole #27, the second of the season, began on Friday, December 5th and finished on Saturday, December 6th and the string was deployed on Sunday, December 7th. The third hole, hole #36, started on Tuesday, December 9th and was completed on Wednesday, December 10th. Deployment of this string finished on Thursday, December 11th. Overall both the hot water drill and firn drill are performing well.
Nearly all of the cargo needed for this season, including the Paro pressure sensors required for deployment, had arrived by December 4th. Also, two retrograde shipments have already left the South Pole. The data magnetic media tapes were shipped to the IceCube office in Wisconsin and an ICDS Compressor was shipped to McMurdo.
Day-shift deployment team with the last DOM on String 18. Left-right, Ben Benischek, Jens Berdermann, Freija Descamps, Matthew (“Red”) Mathiason, Erik Verhagen, Andy Laundrie. Photo-Tom Ham
Digital Optical Module Testing – South Pole Acceptance Test number 8 (SPAT 8) was completed without any problems. SPAT 2 included four just arrived Japanese Golden HQE DOMs. Minor problems have included plastic connectors breaking and a few light leaks. All minor problems were resolved and all DOMs have passed testing.
Digital Optical Module and Cable Production Status and Plans – DOM Integration ceased as planned at all production sites pending the delivery of materials in November and December. It will resume in January 2009 at PSL and in Sweden. DESY in Germany successfully completed all of their DOM Integration and Test activities on schedule. DOM production is not a critical path activity.
Detector Commissioning and Verification – Verification continues to run since the start of IC40 data taking. New disk and new computers have been deployed at the South Pole and are being tested by C&V personnel to insure that they will work as expected. An interface to the Bad Run list has been created, modeled on a similar interface to the Bad DOM list, and is undergoing testing. A framework for verification of simulation, modeled closely on the framework used for verifying data quality, has been completed and being used in test mode by Dawn Williams (U. Alabama), working with personnel at UW and elsewhere.
As a first step to improve the way we treat the saturation response of the IceCube DOMs, the measured response of an “average” DOM will be used for all deployed DOMs. Until now, a single DOM’s response was also used, but it was not deemed “average.” Future improvements entail calibrating each DOM separately since the variation from DOM to DOM is not small. This will require significant effort, both in data taking and in data analysis. Planning for this process is underway, under the leadership of Chris Wendt (UW) with close interaction with Dawn Williams for data taking questions, and Kurt Woschnagg (UC-Berkeley) as L3 lead for calibration.
Data Acquisition Hardware and Software– Fifteen refurbished DOMHub computers arrived at the South Pole and were installed into the IceCube Laboratory (ICL) racks. The total count of hubs at Pole is now 79. Next year a final shipment of 25 hubs will complete the hub delivery for the entire IceCube array. Planning for the migration to three new Proliant DL380 server platforms was completed, however the actual change over occurred in the month of December.
The DAQ software remained at “Bay Wolf” release - the next release is expected mid-December. Numerous improvements to flasher calibration operation, soft Local Coincidence, and “IceCube Live!” integration were checked into the code tree leading to this next release. Operationally the DAQ remained stable during the month of November - the daily uptime chart is plotted below - availability of data for science analysis is above 98%. The Winter-Over experiment operator crew rotated and the new personnel have begun operation of the DAQ and online systems.
IceCube Maintenance and Operations
Detector Up Time / 98.3%IceCube Clean Run Up-Time not including AMANDA Array / 97.1%
Unscheduled Downtime / 0.85%
Events from DAQ / 3.51 billion
With the start of the Pole season we have seen increased activities on the South Pole systems. Nevertheless data taking during Nov 2008 achieved an overall 98.3% uptime. The in-ice clean run uptime was 97.1%, lower than typical but high considering the large amount of activities and special runs/configurations.
The main activity on the detector during this month (besides normal data-taking) was calibration and characterization of the AMANDA DAQ electronics. This is an important and timely activity because the system is due to be decommissioned in April 2009. While the AMANDA array was calibrated and characterized, the rest of IceCube kept taking data as usual. About 20% of this month, the detector ran without collecting data from AMANDA as a consequence.
Data Handling – South Pole systems in the IceCube Laboratory (ICL) began IC40 physics operation in April 2008. Operations continued through October with systems operating smoothly. The figure below shows the daily satellite data transfer rates.
The work in November included completing the installation of equipment in the ICL for the additional strings being deployed this year, and continuing work on upgrading the UW data warehouse. ICL computing installation is complete for operation of the IC56 detector. In addition to added hardware for the 16 new strings, there were major upgrades to the DAQ and PnF servers and disk systems. These systems now have the performance required to accommodate the full IceCube/IceTop detector when completed. The data warehouse capacity increase is nearing completion, with current estimate of the switch to the new disks for data of January 1, 2009. The work was slightly delayed due to ongoing reliability issues with the old data disks taking away manpower for the enhancements. The instability of the over taxed old data disks were the main reason for the needed enhancements.
Filtering, Software & Database – Filtering of IC40 data began in April 2008 representing the full transition from IC22 to IC40 running. The Joint-Event Builder and Processing and Filter server (JEB/PnF) merging of IceCube-40 and Amanda TWR data streams continues at South Pole sending filtered data sets of about 35 GB/day or more over the satellite.
Significant effort in November went into improved and upgraded integration of the PnF system into the IceCube Live monitoring system, which is planned to come into full operation with the start of the IC56 run in April 2009. We see no significant delays or technical issues in meeting this milestone.
The PnF/JEB filter system was moved to the new upgraded computers in the ICL. The transition went smoothly with no major technical issues.
Software and architectural work on the PnF/JEB online filter system continues in preparation for the larger data rates next year with the IC56 detector. This final major architectural upgrade to the online filter system is in final testing for scheduled delivery for the IC56 run. This will be the last major JEB/PnF release under the MREFC construction next year, and this system will be the final design with full performance for the 80-string detector.
Work continued in earnest for setting the online physics filters for next year’s IC56 run. This process continues to be ahead of last years schedule. The preliminary estimate for all online filter proposals from the physics working groups is 52 GB/day, which is within next year’s IceCube satellite bandwidth allocation.
Offline filtering of the IC40 data was a stand still last month. Although the L2 reconstruction software has been tested and is ready to go, there has been a delay waiting for the upgrade to the data warehouse, which is required for stable offline L2 reconstruction to begin. We expect that full L2 production processing will start when the data warehouse upgrade is complete by January 1, 2009. Although about four to six weeks behind where we wanted to be, but still well ahead of last years schedule.
Simulation – Simulation production has essentially been halted to wait for the data warehouse upgrade. We do have sufficient simulation datasets available for preliminary IC40 studies, IC56 filter studies and most IC22 data analysis. We expect that full production will resume in January 2009.
Education and Outreach – Jim Madsen, UW River Falls, reached agreement with the Knowles Science Teaching Foundation and PolarTrec to support teachers to work with IceCube for the next year and tentatively beyond that. The basic plan is that four KSTF Fellows (high school teachers) will train in Alaska with PolarTrec and at UW River Falls with IceCube in the summer of 2009. Two additional teachers will train at UW River Falls with a possibility of training in Alaska. One of the teachers will travel to the South Pole in the 2009 to 2010 season while the others will be in contact and bring polar science to their classrooms. PolarTrec will make a formal announcement in January. Dr. Madsen will solicit collaboration members to join the training at River Falls or work with either teachers or students in the Upward Bound program.