\documentclass[aps]{revtex4} \usepackage{epsfig} %\documentstyle[epsfig,preprint,aps]{revtex} \font\eightit=cmti8 % Used in address list \def\r#1{\ignorespaces $^{#1}$} % Used in author and address list %----------------------------------------------------------------------- \begin{document} \draft % \onecolumngrid \title{ \begin{center} A Detector Control System for MINOS\\ Third Review: January 9, 2001 \end{center} } \author{ \hfilneg \begin{sloppypar} \noindent T.~Alexopoulos\r {1}, A.~Erwin\r {6}, A.~Habig\r {5}, P. Huican\r {6}, N.~Longley\r {2}, J.~McDonald\r {4}, M.~L.~Marshak\r {3} and C.~Velissaris\r {6} \end{sloppypar} } \address{ \begin{center} \r 1 {\eightit Department of Physics, Athens Technical University, Athens, Greece}\\ \r 2 {\eightit Department of Physics, Macalester College, St. Paul MN} \\ \r 3 {\eightit School of Physics and Astronomy, University of Minnesota, Minneapolis MN 55455} \\ \r 4 {\eightit Department of Physics, University of Pittsburgh, Pittsburgh PA}\\ \r 5 {\eightit Department of Physics, University of Minnesota, Duluth MN} \\ \r 6 {\eightit Department of Physics, University of Wisconsin, Madison WI} \\ \end{center} } % \begin{abstract} The MINOS Detector Control System (DCS) implements an integrated set of hardware and software to display, control, and log the various sub-systems of the MINOS Far Detector and Near Detector. This document provides an overview of the DCS as part of the documentation for the Third Review of the DCS scheduled for January 9, 2001. \end{abstract} \maketitle \twocolumngrid % %---Text-------------------------------------------------------------------- --- % \section{Introduction} \label{sec:info} The purpose of the Detector Control System (DCS) is to display, control and log the operating parameters of the MINOS Far and Near Detectors. The detector systems that will interface with the DCS and the role of the DCS with respect to these systems are listed below. The {\bf control} function means that the DCS will set operating parameters for the system. The {\bf monitor} function means that the DCS will display current information from this system in its online displays. The {\bf log} function means that the DCS will generate ROOT files containing information from this system for transmission via the Dispatcher to the Oracle database. The detector systems that interface with the DCS are: \begin{enumerate} \item LeCroy 1440 High Voltage -- control, monitor and log \item BiRa rack monitors -- control, monitor and log \item Environmental parameters -- monitor and log \item Beam monitors -- monitor and log \item Calibration (flasher) data -- log settings and carry communications \item Detector Magnetization data -- monitor and log \end{enumerate} In addition to these detector systems, the DCS will interface with the Data Acquisition System (DAQ), especially the Dispatcher process, and the Oracle Database system. Systems that will coordinate with the DCS include the Data Acquisition System (DAQ), the Magnet system, and the Calibration system. The DCS is designed as a robust, modular system comprised almost entirely of off-the-shelf, industrial control hardware and software. The DCS system uses established hardware and software protocols, such as OPC (Object Linking and Embedding for Process Control), Samba (Linux server software for PC networks) and ROOT TSockets (a protocol for interprocess communications). Using standard interfaces permits plug-in replacement of DCS hardware and software either for upgrades during the life of the MINOS detectors or for rapid debugging. The DCS as currently designed uses multiple dedicated processors in order to achieve modularity and reliability. Because the DCS is a real-time critical system, dedicated systems also avoid difficult-to-diagnose problems due to variable loads on processor resources by other tasks. The DCS group has purchased and tested a number of special-purpose processors for these applications, but we have concluded after these tests that using general-purpose PC's as dedicated processors is more flexible and cost effective. For example, the cost of an ethernet-to-serial converter, which uses a microprocessor, is essentially the same as a low-end PC, because the converter has a small market over which to amortize fixed costs and the PC has a large one. We are unsure how the proposed MINOS policy of no dedicated processors applies to this design. We hope that this review will help clarify such questions. \section{Overall Design} \label{sec:design} At the heart of the DCS design is the Intellution SCADA (Supervisory Control and Data Acquisition) system called iFix. iFix runs under Windows 2000 and provides the following functionalities: \begin{enumerate} \item Data Acquisition and Control Interfaces: iFix is able to acquire data using up to eight I/O drivers. The I/O drivers for the DCS application include a general driver to handle OPC (Object Linking and Embedding for Process Control) servers and a specific driver for the Intelligent Instrumentation EDAS 1002 unit that provides the ethernet interface for the BiRa rack monitors. Since these drivers are bidirectional, they are able to both acquire data and supply control information. \item Alarms: iFix automatically checks the acquired data against preset ranges in order to detect alarm conditions. Alarm conditions will trigger display changes to notify detector operators and generate emails to appropriate MINOS personnel to inform them of operating problems. \item GUI Interface: iFix uses graphical programming of display objects to implement flexible visual interfaces. The DCS plans to use one dedicated computer screen that would always be visible to the detector operator. Normally, the screen would show a summary of detector operating conditions with sub-areas showing green backgrounds for each of the major detector systems. If an alarm is generated in a detector system, the system area in the main display would turn yellow or red, depending on the alarm severity. The system blocks would also include clickable ``buttons'' that would enable an operator to ``drill-down'' into specific information about a system under either normal or alarm conditions. \item Database: iFix maintains its own historical database of recent detector operating parameters. This database will enable displays of historical detector parameters for problem diagnosis or other reasons. \item Programmability: iFix is programmable using Visual Basic for Applications, a customized version of Microsoft Visual Basic. Visual Basic is an object-oriented language which has many of the features of C++, while retaining the syntax of Basic. Despite the fact that programmers who dealt with the original BASIC cringe at the thought of it, Visual Basic is currently the most popular programming language in the world. Thus, there is a large knowledge pool available for support in the commercial world, and MINOS students will gain a very marketable skill. \end{enumerate} \section{High Voltage} \label{sec:hv} \section{Rack Monitoring} \label{sec:rackmon} The DCS group is charged with monitoring the state of the experimental electronics in the racks at the detectors. Three general tasks will be accomplished: \begin{enumerate} \item For safety reasons, immediately shut down the AC power to equipment where a possible danger such as fire or leaking water is present. \item Monitor conditions which might affect the electronics performance, such as crate voltages and temperatures, and periodically log the resulting data. Warning or Alarm conditions on any of the parameters of concern will be immediately logged. \item Use the existing rack monitoring hardware to assist with secondary communications to the electronics being monitored. \end{enumerate} The heart of the rack monitoring system is the RPS-8884 unit from BiRa Systems. This is a 1U rack mounted module which performs most of the functions the DCS system needs, and which communicates with the DCS control computer over a standard ethernet link. Software drivers to control this device exist in the iFIX environment (see Sec.~\ref{sec:design}). An overview of the device can be found at: \centerline{ \texttt{http://www.bira.com/cat/rps.htm}} \noindent This web page is only an overview, and the details of the specific system which will be installed in MINOS are being worked out in cooperation with BiRa systems. Which bits of this unit will get used where are detailed in the following sections. \subsection{Power Shutdown for Safety} \label{sec:safety} In order to comply with safety regulations, any electronics rack which contains AC power and which will be left unattended must have the ability to shut itself off in event of a fire. For our purposes, this self-shutdown can be extended to other alarm conditions that could lead to damage to the electronics, such as large power fluctuations, water leaks (in water cooled racks), and high temperatures caused by cooling failures or electronics overheating. Any of the alarm conditions discussed below will cause the AC power to the rack to be cut off, and alarm data will be sent to the DCS logging system. The power cutoff is accomplished by running the AC power to the rack through a relay box (also from BiRa) controlled by the RPS unit. \begin{itemize} \item The RPS-8884 monitors a ``Fenwal CPD7051'' ionization smoke detector, which will be installed near the top of each rack. In event of smoke, an alarm condition is set, and the AC power to the rack is cut. \item The quality of the AC power is monitored by the AC relay box itself, and an alarm raised and power cut if there is a power surge, protecting the electronics from harm. \item The DC voltages provided by the data acquisition VME crate to its backplane is monitored. Should any of these voltages cross a preset threshold, a warning is logged to the DCS system. Should any of these voltages cross a second preset threshold, an alarm condition is set and the AC power to the rack is cut. These voltage thresholds are set with resistors when the RPS unit is installed, allowing the acquisition electronics people to choose what thresholds they want. The 9U VME crate to be monitored has post connectors to the appropriate backplane voltages, which will be attached to the RPS unit via wires with crimped ring terminal ends. \item The temperature of the VME crate's cooling air flow is checked both before and after it passes through the crate. A warning is set at $40^\circ$C and an alarm at $50^\circ$C for the input air, and if there is a difference of $20^\circ$C between input and output air temperatures a warning is set, with an alarm at a $\Delta$T of $30^\circ$C. Additionally, if the output air temperature is at 100\% relative humidity, a warning is set (this condition is most relevant for the water cooled near detector electronics). Again, both alarms and warnings are logged, and alarms trip the AC power. \item Water leak detectors will be installed underneath the near-detector's water cooled front-end electronics crate. This detector is a 1U 19" rack mounted parallel plane screen mesh conductivity sensor available from BiRa, which will raise an alarm condition should water drip from a leak and hit the mesh. Such an event will cut AC power to the rack and log an alarm. \item The water flow itself in the front-end racks will be monitored with an inline ``GEMS RFO-2500'' series RotorFlow sensor. The RPS main chassis detects a warning of low flow at 3.5gpm and an alarm at 2.5gpm. A water temperature sensor is also mounted inline, a LM35 type device mounted inside a copper block that is secured to the water inlet pipe. The RPS main chassis sets a warning at $20^\circ$C and an alarm at $30^\circ$C. Both warnings and alarms are logged, and alarms will cut the AC power to protect the electronics. \end{itemize} The RPS main chassis unit requires AC power from the line at a point before the AC relay box, so that it can continue to function even after the rack power has been tripped. This allows the rack AC power to be re-enabled remotely, as well as the readout of the exact error and alarm status. Should power completely fail, the RPS unit will remember its alarm and warning status for up to a day without power, so a history of problems can be retrieved after truly catastrophic problems. \subsection{Electronics Monitoring} \label{sec:monitoring} Many of the quantities to be monitored were discussed above in Sec.~\ref{sec:safety}. Alarm and warning events will be logged to the MINOS data stream and audiovisual warnings displayed on the DCS main status machine in the control room. Analog quantities such as DC voltages and temperatures will be periodically read out and logged to the data stream as well. The time between such readouts can be set at will by the DCS control software, and a time period appropriate to the time scale over which changes actually occur will be chosen during the testing of the system. Additional monitoring of the 9U VME crate's status can be accomplished through a serial link to the Weiner power supply's RS-232 port. The RPS-8884 offers ethernet-to-serial communications with two serial ports, one of whom can be used to gather whatever extra information the Weiner power supply will give. This interface is normally used as part of their CaenNET interface, which we are not using, so we must rely on Weiner telling us the power supply's communication protocol to gain the use of this extra link, but it is not vital and could be free redundancy. In addition to the power, cooling, and the VME crates, there is a Power Distribution Box (``PDB'') to be monitored. This box supplies two voltages, +5V and -5V, each which must be monitored in a differential fashion with respect to its own ``return'' line rather than compared to ground. Additionally, the temperature inside the PDB must be monitored, as the box uses its own air cooling system. The allowable parameter ranges of voltage and temperature are currently under discussion, but warnings will be set if the yet-to-be-determined thresholds are crossed, and the analog values of these parameters will be periodically logged. One final item is monitored which can cause a warning but not an alarm, and that is the case of a fan failure in the fan try being built at FNAL for air cooling the VME crates. While catastrophic air flow failure will be detected by the air temperature sensors mentioned above, if individual fans die in the fan tray the overall air flow will not be slowed enough to notice a temperature problem. Thus, the trays are being built with a two-terminal MOSFET switch output to indicate the failure of one or more fan in the tray. The switch is normally open, and when closed indicates that all is well. Open indicates power off or a dead fan. Such a condition will issue a warning to be logged and displayed, so that the fan tray can be fixed before further failures cause a problematic reduction in air flow. The last inhabitant of the racks being monitored is the LED flasher box. This box contains nothing which needs to be monitored by the DCS, although it will communicate to the world through the RPS-8884's RS-485 serial port. \subsection{Secondary Communications} \label{sec:communications} In addition to the safety and monitoring jobs mentioned previously, the rack monitoring equipment provides a convenient way to communicate with some of the electronics in the racks being monitored. Two such uses are planned. First, the ability to do a warm reboot of the VME crate from a means independent of the normal VME communications chain is desired. This will cover the case of the VME crate locking up in a bad way, so that it can be recovered remotely. The RPS unit already taps into the VME backplane to monitor voltages. One extra wire will connect to the System Reset line, and the RPS unit will assert this line low (causing the VME processors to reboot) if the DAQ gives this command to the DCS. Second, the RPS-8884's second serial port (an RS-485 connection) in combination with the built-in EDAS ethernet to serial converter will provide the communications channel needed for the calibration PC to control the LED flashers in the flasher box. \subsection{Status} \label{sec:rackstatus} The status of the rack monitoring sub-system has progressed rapidly from a general concept to an actual plan. The requirements of the systems needing be monitored have been established and matched with hardware which will do the needed tasks. Some details of the hardware plan and requirements remain to be mapped out, for instance the exact DC thresholds the VME crates and PDB can tolerate, and the acceptable temperature inside the PDB. The first real test of this scheme will come later in January, when a prototype RPS-8884 will be delivered to UMD for testing. In addition to verifying the functionality of all the individual components, this will allow the large task of software integration to begin. Ensuring that the data can get out of the RPS unit and into the DCS software and that the mechanisms exist for the DAQ and Flasher computers to talk to the VME reset line and the flasher box will be the major immediate areas of work. To help with these tasks, additional money from the University of Minnesota has been procured to hire an undergraduate Physics major, Eric~Hall. Eric begins working this semester at UMD with Alec Habig, and helping with the hardware testing and software development will be his primary job. It looks promising to have a complete prototype rack monitoring system ready to be tested in the calibration detector by late summer, and that is the goal we are working towards. \section{Environmental Monitoring} \label{sec:environ} \section{Beam Monitoring} \label{sec:beam} \section{Calibration} \label{sec:flasher} The DCS system will provide the Calibration system with the means to control the LED flasher boxes and to log data such as which LEDs flashed at what times. The control will be done through the BiRa RPS-8884 rack protection system unit's RS-485 port, taking advantage of the RPS's secondary communications ability (Sec.~\ref{sec:communications}). The calibration group's control PC will be able to access this serial port through the RPS-8884's built-in EDAS ethernet-to-serial converter, and control the flasher box as if they had a direct serial connection to their control PC. For each LED flash, the calibration PC will present the DCS logging daemon (Sec.~\ref{sec:daq}) with a root object containing information about which LEDs flashed and at what time. This object will be logged by the DCS with other slow control information the system is generating, and picked up by the dispatcher to be integrated into the main MINOS data stream. Monitoring the the flasher boxes themselves does not need to be done - they are self contained with respect to both power and cooling, and if they stop working it is both fairly obvious and non-critical. \section{Detector Magnetization Data} \label{sec:magnet} \section{DCS-DAQ Communications} \label{sec:daq} Due to the separation of the data acquisition system and the detector control system, a communication path had to be developed. This communication is further complicated by the fact that the DAQ and DCS exist on separate operating systems. Thus a communication package that uses a network based communication layer has been developed. There are two principle network protocols which are supported on most modern operating systems. The protocols are TCPIP and UDP. The TCPIP protocol guarantees delivery but requires a ``server'' and a ``client'' via a ``socket''. UDP guarantees nothing but allows for broadcast communication which is not available with the TCPIP protocol. As reliability is a concern as well as security, we have chosen the TCPIP protocol as the communication protocol. It is possible to manage many ``client-server'' connections with a simple socket manager. However, one expects that most connections will be determined in advance and an adaptive socket manager is not required. Currently, the DCS-DAQ communication is visualized as communicating two type of notification objects. The types are ``command'' objects and ``messages'' objects. ``Command'' objects are effectively notification of the change of state of the system, such as a new run has started. ``Message'' objects are notification from monitoring processes regarding the quality of the data. ``Message'' objects shall be displayed and appropriate action will be determined by the operator. ``Command'' objects may require some automated action to be taken by the DCS. Such action may include recording the start time and generating the appropriate database links for the new run. It is expected that there will be several appropriate ``command'' objects that can be expected and any appropriate can be taken by DCS. The role of DCS is subservient to the DAQ. DCS has no ability to control or influence the acquisition of data. The DAQ system has the ultimate control of this acquisition. A prototype system has been developed to use sockets (TSockets) as implemented in the ROOT\cite{rootsys} C++ program library. ROOT was chosen because of its flexibility, platform independence and in order to limit the number of software dependencies of the system. Communication via the ROOT interface has been prototype using a root object which indicates the type of message and a short string which contains the actual message. In addition, GUI interfaces have been developed using ROOT GUI classes for displaying message types and for further developing the interface to DAQ-DCS. An important detail is flexibility for receiving messages from non-ROOT programs (a simple string message) which is elegantly handled with ROOT because one can identify the messages as a ROOT object or as a simple string type. Communication acknowledgement, another important detail, provides a fast acknowledgement that a message has been received. Currently, a response acknowledgement message is received by the sender of the message notifying the sender that the message has been received. \section{DCS-Oracle Database Interface} \label{sec:oracle} % %--------------------------------------------------------------------------- -- \begin{references} \bibitem{rootsys} Rene Brun and Fons Rademakers, ``ROOT - An Object Oriented Data Analysis Framework,'' Proceedings AIHENP'96 Workshop, Lausanne, Sep. 1996, Nucl. Inst. \& Meth. in Phys. Res. A 389 (1997) 81-86. See also http://root.cern.ch/. \end{references} \end{document}