Data recording used to be a fairly straightforward process–digitize the data and stream it to a storage subsystem fast enough to keep up in real time. Data analysis was done mostly offline, perhaps with some real-time display of channels during recording. But as sensors have gotten faster and more diverse, and requirements for data analysis have become real time, data recording has had to adapt. Recorders need to deal with data flowing in at different rates, in different formats and over multiple channels. Adding to that complexity is the synchronization required to correlate data on multiple channels to yield the needed results.
Coming to the rescue, as in many defense applications today, is advanced digital signal processing (DSP). With data now routinely digitized, more intelligence appearing in data recorders, and opportunities for better integration of data recording with the tactical system, data can be captured and processed into useful information quickly and accurately.
Signal Processing Elements
Sophisticated military electronic equipment, such as radars, electronic warfare (EW) systems, and Intelligence, Surveillance and Reconnaissance (ISR) systems, contains embedded computers that perform very sophisticated real-time DSP operations. The key elements in the data paths of these tactical systems include a maturing array of higher-speed sensor technologies, high-speed serial interconnects (HSSIs) and fabric technologies, very large FPGA devices and new multicore general-purpose processors.
A hypothetical embedded real-time DSP system is illustrated in Figure 1, showing components and interconnects in advanced DSP systems today. This hypothetical system shows four channels of data each coming across an HSSI from four sensors (via A/D converters). Significant layers of front-end data processing are done by FPGAs before intermediate results are sent to an array of Distributed Multi-Processing (DMP) nodes for further processing and final output to other systems such displays or weapon targeting systems.
From the sensors and between the FPGAs, the data flow is often unidirectional and routing is pre-determined and static. Here the system can utilize simple, low-overhead, point-to-point protocols such Serial FPDP from sensor to FPGA, and Aurora or RocketIO between FPGAs. Compared to switched fabric protocols, these lightweight protocols consume far less FPGA resources preserving more gates for processing, and also have lower packet overhead preserving more link bandwidth for data.
Bringing in the Data Recorder
These same DSP architectural ideas used in tactical systems are poised to collect and crunch mind-boggling amounts of data at speeds barely imagined a few short years ago. Once the data is digitized, FPGAs can be used to grab and pre-process it, and general-purpose processors can finish the task. In fact, data recording systems designed with these ideas can be integrated with the tactical system, delivering significant benefits.
With many tactical systems now using HSSI links on a VXS or VPX backplane, a data recording system based on a VXS or VPX board can integrate into the same backplane and allow connection into one or more of the desired HSSI. Not only can data be recorded efficiently this way, the data also may be played back at the same point in the system. Data captured during an expensive real-world test can be played back into the system endlessly to improve and refine the system performance. The number of recording channels can be easily scaled to match the need for speed by scaling the number of data recorder boards.
Figure 2 illustrates a VXS real-time data recorder, the VMETRO Vortex hybrid recording engine. Such a data recorder must support a variety of protocols that might be used within the system, dictating that the architecture must be a hybrid of FPGA technology and general-purpose processing. The FPGA technology allows integration with the HSSI and its protocol within the DSP system, and a general-purpose processor runs the data recorder software.
In a system, the desired HSSI can be routed to eight HSSI links on the VXS P0 connector of the recording engine. Behind the VXS ports is a Xilinx Virtex-4 FPGA, where IP cores implement specific HSSI protocols and other functions such as headers, timing and processing algorithms. Attached to the FPGA is a large block of high-speed SDRAM. This SDRAM can be used to stage and format data streams for recording, but its most important function is to supply ample elasticity buffering of very high burst rate data from the HSSI. Once captured, the high-rate data can then be drained to the recorder at much lower sustained data rates, reducing the sustained throughputs required from the rest of the recording system.
Keeping Pace with Data
Systems like these now include multiple HSSIs at up to 3.125 GHz each on the backplane, providing unprecedented opportunity to record and analyze enormous amounts of data. With A/D converters also now operating at 3 Gsamples/s and above, non-volatile capture of data streams has become a major challenge. The answer is to divide and conquer.
RAID0 striping of data across disk drives increases the throughput of disk storage to near the burst rate of the storage interface. On top of that, the storage interface links can also be striped in order to aggregate the speed of those connections. At the highest layer, the data recording engines themselves can be striped with data to linearly and infinitely scale the total throughput of a large recording system.
How does this work in practice? With Fibre Channel drives capable of storing data at 100 Mbytes/s sequentially across a drive, four of these drives can be combined on one 4 Gbit/s Fibre Channel loop to attain 400 Mbytes/s. Outfit the recording engine with two of these Fibre Channel ports and an aggregate rate of 800 Mbytes/s is theoretically possible. Then striping the sensor data across four recorders would allow 3.2 Gbytes/s or 100 percent capture of a 3 Gsample/s sensor stream.
While the example above is theoretically possible, in practical applications there are many elements that must be considered including inner track storage speeds, protocol overhead, protocol and bus conversion overheads, memory bandwidth, and the bursty nature of digital data as it flows through these many elements. Bandwidth is also required to allow various elasticity buffers to drain throughout the system while maintaining the specified sustained rate. The recorder shown previously is specified to record single or multiple streams at a sustained rate of 720 Mbytes/s. Next-generation recording engines are being developed that will more than double that rate in the not too distant future.
Storage of the End Results
To meet a broad spectrum of applications that spans from the relatively benign lab to the harshest operational and deployed environments, the ideal data recorder storage subsystem would be small, low power, fast, large capacity, removable, rugged, high-altitude capable and the lowest possible cost.
Unfortunately, costs can increase dramatically as each of these characteristics increase. This is true for the least expensive 1 Terabyte SATA hard drive at less than $1 per Gbyte, to the sealed and shock isolated FC drive at $30 per Gbyte, to the most expensive extended temp solid-state drives at about $90 per Gbyte.
Recording engine hardware and software must embrace storage technology that includes an extremely broad spectrum of storage capabilities. Fibre Channel Storage Area Network (FC SAN) technology offers the right breadth, allowing most any FC equipped devices to be used for data storage. The spectrum includes anything from one or more SATA drives with FC interposer modules, to FC/SATA RAIDs, to almost unlimited rotating or solid-state FC devices on either an FC loop or switched topology.
Going Rugged When Required
Most programs begin with a proof-of-concept system in the lab connected to hardware-in-the-loop simulation systems. In this early development phase it is often a waste of program funds to utilize fully rugged conduction-cooled versions of the processing and recording products. When it is time to deploy, rugged versions with software and performance identical to the lab systems are critical.
The processing and data recording elements of the advanced DSP system as well as the data recorder engine must be available in commercial and rugged air-cooled as well as the most rugged conduction-cooled versions. Figure 3 shows the Vortex Recorder in both rugged air-cooled and conduction-cooled variations.
New advanced signal processing architectures have not only transformed tactical systems design, but created an opportunity for data recording to be seamlessly added to systems. Data recorder engines now employ hybrid FPGA and general-purpose processing techniques that allow new interconnect strategies, higher recording speeds, large variations in storage technologies, and a full spectrum of robustness and ruggedization levels. The near-term goals for real-time data recording using techniques described here reach into the range of 1500 Mbytes/s for a single recorder, and as technology continues to improve that target is expected to grow further.
VMETRO, Inc.
Houston, TX.
(281) 584-0728.
[www.vmetro.com].
