MICE-NOTE-DAQ-139
September 24, 2007
MICE Tracker Data Acquisition Using D0 Analog Front End IIt Boards
Terry Hart
Illinois Institute of Technology, United States
I. INTRODUCTION
The Muon Ionization Cooling Experiment (MICE) is an accelerator technology experiment designed to demonstrate the feasibility of muon cooling, necessary for the next generation of high energy physics facilities including neutrino factories and muon colliders [1]. MICE will be built at Rutherford Appleton Laboratory on a new beam line that comes off the ISIS synchrotron [2].
Two fiber spectrometers will measure the momenta of tracks before and after being “cooled” (alternately slowed down by passage through liquid hydrogen and then accelerated along the MICE beam line by 201.25 MHz radiofrequency (RF) cavities). Power considerations limit the duty factor of the RF cavities to 10-3. Constraints from the target system result in a maximum time for the target beam crossing of about 1 ms. These two constraints lead to a nominal MICE operation cycle of 1 ms time periods with the RF powered and the beam on target repeated once per second. During each ms that the RF is on, the fiber spectrometers will need to acquire, digitize, and buffer data from as many as 600 through-going muons.
AFE (Analog Front End) electronics boards [3] will perform the data acquisition (DAQ) and are currently used for DAQ for the D0 experiment at Fermilab to digitize input analog pulses from the D0 trackers. Upgraded AFE II boards are being tested and have already been used for the DAQ of the MICE KEK test beam run in September, 2005 and for cosmic ray test runs. The AFE II boards include Trigger and Pipeline (TriP) chips which store analog charge data in 48-level analog pipelines and also provide digital discriminator information. The latest upgrade to the AFE II board is the addition of timing information to the TriP chips; these modified chips are TriP-t chips [4]. The AFE II boards with TriP-t chips are AFE-IIt boards. Figure 1 shows the layout of the tracker DAQ components.
II. AFE-IIt BOARDS FOR MICE TRACKER DAQ
The AFE-IIt boards with the current implementationof data acquisition software, are capable of collecting charge, time and discriminator data from the trackers in about 5700 ns. This digitization time is about a factor of 3 or 4 longer than the time necessary to process an expected incoming muon rate of about 600 KHz (corresponding to an average time between muons of 1667 ns). If only discriminator data are needed for nominal MICE tracker DAQ, then the data collection goal of 600 muons/ms may be straightforward. Even if discriminator-only data taking is sufficient for MICE running, occasional calibration runs in which charge and timing data are collected will probably still be necessary. The rest of the note will discuss the acquisition of charge, time, and channel bitmap data.
Figure 1: A schematic of the layout for the cryostat, AFE-IIt board cassettes, and AFE-IIt boards for MICE. Each picture corresponds to one tracker. Data from the trackers go to VLPC cassettes read out by AFE-IIt boards, and the VLPCs are cooled by the cryostats. This arrangement can accommodate a total of 8192 channels from 2 trackers. MICE will have 2 trackers, 4 cryostats, 8 cassettes, and 16 AFE-IIt boards.
In MICE, through-going muons induce light pulses in the tracker scintillating fibers. These light pulses are converted to charge pulses and amplified by Visible Light Photon Counters (VLPCs). As shown in Fig. 3, these analog pulses are inputs to TriP-t chips on the AFE-IIt boards which generate three outputs for each channel: (1) a digital discriminator signal, (2) an analog pulse proportional to the amplitude of the input pulse charge (the A-pulse), and (3) an analog pulse proportional to the amplitude of the input pulse time relative to the time between the firing of the discriminator and the closing of the time gate.
The discriminator output is routed to one of the Field Programmable Gate Arrays (FPGAs) called the DFPGA (“digital” or “discriminator” FPGA). The A-pulse and the t-pulse are stored in 48 sample analog pipelines in the TriP-t chips until being read out upon receipt of an external L1ACCEPT trigger. In D0, a trigger causesthese discriminator bits to be sent to the Digital Front End system. In MICE, the discriminator bits, which indicate which channels are above a predetermined threshold, will be directed to another FPGA called the AFPGA (“analog” FPGA), and will be used for the digitization of the data in the channels above threshold. MICE will use a modified scheme which digitizes only those channels above threshold and rapidly cycle through the channels below threshold. This “zero-suppression” scheme will digitize the channels more quickly than the nominal digitization scheme. Also, a 4-level analog data buffering scheme will be used so that incoming data can be held for later digitization while data from a previous even is being digitized.
In D0, a TriP-t chip receiving an L1 ACCEPT signal causes the AFE-IIt board to direct the TriP-t chip to execute Acquire, Digitize, and Readout cycles for the A-pulse and t-pulse from a fixed channel in the pipeline. Then the A-pulse and t-pulse are read
out over a ribbon cable to the SVX Sequencer (originally designed for initialization, control, and readout of SVX IIe silicon vertex detector chips). For MICE, different event sizes makes a modification to the Acquire-Digitize-Readout mode structure necessary. This will be described in more detail later in the note. The A-pulse and t-pulse are read out over a Low Voltage Differential Signal (LVDS) path to a VME LVDS SERDES (Serializer/Deserializer) Buffer (VLSB; see Fig. 2) upon receipt of a trigger signal (L1 ACCEPT). In MICE, the L1 ACCEPT will be formed from signals from Time-of-Flight counters and signals from the ISIS accelerator; a combination of these signals will indicate the passage of a through-going muon.
- MICE REQUIREMENTS FOR AFE-IIt BOARDS
- Charge and Time Digitization
The AFE-IIt boards were designed for the D0 tracker to collect data produced from the TeVatron at Fermilab. The TeVatron operates at a fixed frequency of 53.104 MHz and the AFE-IIt boards are driven by TeVatron clock signals of this frequency. The trigger rate for D0 is much lower than the expected 600 KHz MICE trigger rate so that in the D0 implementation of the TriP-t chip, the TriP-t pipeline does not run during data digitization and the 4-level analog buffers of the TriP-t chips are not used. For each AFE-IIt board, the data from each of its 16 TriP-t chips (each chip consisting of 32 channels) are processed in parallel. Within each chip, the 32 channels are processed in series, and for each channel, 6 clock cycles of 18.831 ns (corresponding to 53.104 MHz) are used. Then, 6 32 18.831 ns = 3616 ns contribute to the total time to digitize the data from 32 channels in a TriP-t. In addition to this time are 36 preparation cycles, and 6 more cycles for readout of a dummy channel (791 ns). For MICE, the pipeline would need to be reset after a digitization if the current D0 implementation is used. This would take ~ 1300 ns so that if the D0 implementation of data readout would be used for MICE, then for every trigger, digitization of all the data would take about 5700 ns.
Our method for digitizing 600 muons/ms is to sparsify the data, to ignore channels that are below a preset threshold which are not of interest. We implement the sparsification for the AFE-IIt board with the following sequence:
1)Using existing data pathways on the AFE-IIt board, deliver the discriminator bitmap to the time and charge digitizers so that the digitizers have a record of which channels are above and below threshold.
2)Using the information from the discriminator bitmap, digitize only those of the 32 channels of a TriP-t chip that are above threshold. Only a very small fraction of channels (~ 0.5%) will be above threshold. The digitization of the series of 32 channels is stopped after the last hit channel; this shortens the total digitization time.
3)Use the 4-level analog buffer of the TriP-t chips to store incoming events (bitmap, and analog data) while an event is being digitized. Code revisions for the AFPGA and DFPGA are necessary for event buffering to work.
We made a study of the effects of reducing the digitization time of an even and implementing a four-level buffer. This study provides an estimate on the number of muons that the tracker data readout can record from an average input rate of 600 kHz. This numerical study makes a simplifying assumption that the digitization time is the same for all events. The digitization time will vary slightly from event to event depending on the number of hits and which channels are hit in a TriP-t chip. Figure 2 shows a graph of the recordable rate; using 4 levels of the buffer and with a digitization time of about 1500 ns, we can expect to be able to record about 550 kHz if the incoming rate is 600 kHz.
Figure 2: Recordable muon rate as functions of buffer level and time to digitize an incoming event. An incoming 600 kHz muon rate is assumed. With a reduction of the digitization time to about 1500 ns and use of all four levels of the analog data buffer, we can expect to be about to record about 550 kHz of a maximum possible 600 kHz muon rate.
B. MICE Trigger from ISIS
MICE collects data from the ISIS synchrotron which has a variable frequency during 1 ms wide proton spills. The frequency varies from 2.9 MHz to 3.086 MHz from the start to end of each spill. The clock driving the AFE-IIt boards will be derived from ISIS signals so that the period of each burst within a spill will vary from 345 ns to 324 ns from the start to the end of each spill. The ISIS frequency will be multiplied by 18 so that the clock period will start at 19.2 ns and end at 18 ns by the end of the spill. Synchronization with the ISIS signals will ensure that the times that the TriP-t accepts data are aligned with the ISIS beam. The synchronization will be accomplished by implementing a phase lock loop (PLL) that can lock on to the input ISIS clock signal. The 53.104 MHz clock oscillator ships on the AFE-IIt boards need to be replaced by oscillator chips that are capable of implementing the PLL that can lock on to the maximum ISIS frequency of 55 MHz. Fermilab engineer Kwame Bowie and RAL engineer Craig MacWaters are designing and building the trigger system. Firmware modifications are also necessary and will be described in more detail later in this note.
Figure 3: Schematic diagram of the MICE signal path from the TriP-t chips to the VLSB memory banks. The TriP-t chip produces three outputs: 1) analog pulses proportional to the amplitudes of an input pulse’s charge, 2) pulses proportional to the time between the firing of the discriminator and the closing of a gate, and 3) a digitized discriminator bitmap. The charge and time pulses are put into a analog pipelines. If there’s an L1 ACCEPT signal, the charge and time data are formatted and read out to a VLSB memory bank. Also, on the AFE-IIt board, one DFPGA services two AFPGA which is different than what this picture suggests.
B. Data Readout
Tracker data are transferred from the AFE-IIt boards to the VLSB modules via Low Voltage Differential Signaling (LVDS) links. The data bus between the DFPGA in the AFE-IIt board and the VLSB module is 21 bits wide. Each DFPGA is associated with two AFPGAs. Each AFE-IIt board reads out data from 4 DFPGAs.
Assuming about 0.5% occupancy yields roughly 30 total hits for the two trackers. As an estimate of the total amount of data that needs to be transferred to the VLSB banks, we note that for this format, all 64 DFPGAs will read out a trigger word, a header word, and 8 words for bitmap data for 10 total words per DFPGA. Then 30 single hits (a sufficient estimate) with 5 bits per hit (channel, TriP0 time, TriP1 time, TriP0 charge, and TriP1 charge) yield 150 more words for a total of (64 × 10) + (30 × 5) = 790 total words for a MICE event. As a conservative estimate, we’ll assume 1000 words/event. Then for 600 events/spill and one spill every second, the AFE-IIt boards will have to read out roughly 600,000 words every second to the VLSB banks via LVDS links via VME. If the data are read out to 16 VLSB modules housed in two crates with each VME word transfer taking 1 µs, the 600,000 word readout would take roughly 0.3 sec. An option for reducing the total data transfer time would be using 32-bit DMA transfers instead of single word transfers (another factor of two).
IV. FIRMWARE OVERVIEW AND PROGRESS
The firmware needs to be modified to accommodate the expected MICE trigger rate of roughly 600 KHz and the ISIS beam structure. By the end of each 1 ms wide spill, the muon beam arrives at MICE every 324 ns in approximately 100 ns wide bursts. The MICE DAQ imposes a trigger constraint that each L1ACCEPT triggers produces a trigger holdoff of 550 ns so that triggers cannot occur on consecutive bursts (separated by 324 ns). However, since a burst may contain multiple muons and since data from an entire burst in stored in the TriP-t pipeline, an L1ACCEPT can be associated with multiple muons; the muon rate is larger than the trigger rate. The timing and spatial separation of multiple muons in a burst will determine if they can be properly reconstructed.
A. AFE-IIt Firmware
i. Data Digitization(AFPGA)
The firmware controlling the data digitization needs to accommodate the high muon rate from MICE. The modifications include turning on the TriP-t pipeline during digitization, using the TriP-t 4-level analog buffer, and implementing the zero-suppression scheme in which each channel with charge data below threshold is processed in one (or two) 18 ns clock cycles. Six 18 ns clock cycles are used to digitize the data from channels with charge amplitudes above threshold. Firmware incorporating these modifications has been written and the syntax and logic of the code has been tested with simulations of the signals.
Hardware tests of the TriP-t chip and the digitization process controlled by the AFPGA have been performed with successful results. Signal sequences corresponding to those generated by the modified firmware have been sent to a TriP-t chip mounted on a test board. These tests have demonstrated that the pipeline, buffer, and one-cycle zero suppression function properly on a TriP-t chip. The operation of the modified AFPGA firmware incorporating all these modifications except for data buffering has been tested by reading out data from just before the output of the DFPGAs. The output signals which include formatted data, strobe signals, and byte counts, match the simulation output indicating that this preliminary code works as expected. Also, signal sequences corresponding to those generated by the modified firmware have been sent to a TriP-t chip mounted on a test board. Tests have demonstrated that the pipeline, buffer, and one-cycle zero suppression function properly on a TriP-t chip.
ii. Data Processing (DFPGA)
Processed data from the AFPGAs are sent to the DFPGAs and eventually to the VLSB modules. The data transfer sequence between the DFPGA and its two associated AFPGAs has been modified for event buffering. Synthesis between the modified AFPGA and DFPGA firmware, both having been tested extensively in simulations,is underway.
The modification to accommodate event buffering is implementing a protocol for the transfers of data on the bidirectional data bus between the AFPGAs and DFPGAs. On this bidirectional bus, discriminator bitmap data are transferred from the DFPGA to the AFPGA to be used for the zero-suppression scheme. Then digitized data are sent from the AFPGA to the DFPGA along the same bidirectional data bus. With a high trigger rate of about 600 kHz, these two transfers can’t always be done in sequence for each event. The AFPGA includes a 4-level buffer to store incoming bitmaps for later digitization while the digitization of a previous event is proceeding. When the 4-level buffer is full, the DFPGA sends a BUSY signal to the COLLECTORto hold off further triggers thus avoiding the 4-level buffer overflow.