RET: Archive Tutorial

High Volume Signal Recording

Rising Edge Technologies data recorders utilize removable media packs of various sizes to record and transport unprocessed signal data.  The ability to quickly exchange the recorded media elements is beneficial in applications where large amounts of data must be collected.   The removable nature of the media, however, can easily create a situation where media packs are created faster then they can be analyzed.  The result is a backlog of idle media packs that can’t be used for future recordings until the data has been processed. 

To support these high volume recording environments, a system is required that will allow a media pack to be copied to a temporary storage location for future processing without losing any of the salient characteristics of the recording.  The system should offer the ability to quickly unload a media pack, copy data back to a media pack, support network connectivity, and offer the ability to retain the data for future analysis.  Ideally, the system would be able to provide this “copy and store” process to any recording device or legacy system a user might need.   

Building systems to capture, store and distribute recorded information requires a careful look at the unique aspects of the signal being stored, the expectations of the user, and the application environment.  This paper will discuss the factors associated with creating a storage system to support recorder copy and transfer applications. 

This copy and transfer storage process will be collectively referred to as archiving.

Collecting and Converting the Data

The signal to be archived is usually in a format that is specific or proprietary to the recording device.  In most cases, the format is not compatible with the file structure of a traditional operating system.  Therefore, the archive system must transfer the recorded signal through a conversion process that can transform the signal file and store it in a format that can be recognized by the archive’s operating system while maintaining the original structure and content of the file to support play out and analysis.

The movement of data through the archive system can be summarized in four steps:

·         Transfer of the signal files from the media pack to the archive

·         Configure and manage the signal files throughout the entire system

·         Transfer of the signal files from the archive via a medium supporting analysis (media pack, network, tape, DVD, etc.)

·         Transfer of data to a removable media (tape or optical) for backup or long term data archive (if required)

A data management server is tasked with the responsibility for supporting the execution of these steps using appropriate software programs and utilities.  The selection of the storage medium used at each stage will reflect the operational goals of the overall archive system.  A sample system architecture is shown in Figure 1. 

Figure 1.  A Signal Transfer and Archive System

The Storage Medium

A number of storage mediums can be used in support of the archive including: the source media (from the recorder), a fast, temporary cache, and, optionally, the transfer and/or archive media.  For pure cache speed, scalable RAID systems can be used for long term storage, optical is a natural choice.  The transfer medium can be selected to match the user applications or environment.  There are, however, tradeoffs among the various storage mediums that must be considered.

Disk storage is growing in popularity as a data archive medium.  Disks, like tapes, are magnetic medium and therefore subject to degradation over time.  The average disk will provide reliable access to data for 3 to 7 years.  The disk will then have to be replaced.  A Redundant Array of Independent Drive (RAID) system can help reduce the risk of inaccessible data.

Tape is a low cost medium with increasing performance and capacity points.  The medium is ideally suited for backup and distribution applications but carries some inherent risk when used as an archive medium.  The laws of physics continuously work against tape over time.  Aging and environmental conditions all combine to slowly degrade the recoverability of information from this fragile medium.  For all of its positive qualities, tape is not an archival medium.  Many tapes made thirty years ago are barely playable today.  Although there are unknowns in qualifying modern tape technology, many experts believe well cared for tapes may last 15 years.  Tape, however, is not the only problem.  The equipment used to play the tape and the use of multi-generation copies easily adds to the degradation of the content. 

Optical media is available in rewritable or write once versions and is capable of offering long term access to the data files.  Optical disks are a highly stable storage media that are not affected by magnetic fields, head and media wear, or temperature variations and offer a minimum of 30 years of shelf life.  As a distribution medium, CD and DVD offer advantages over all magnetic mediums and are compatible with a wide range of commercial products for playback.

A good archive design not only represents a strategy for long term storage of material but for the long-term access of this material as well.  Table 1 offers a quick review of the advantages and disadvantages of various storage media. 

Table 1.  Archive Media Options

Supporting the Transfer Operations

The archive is designed to read the media pack from a recorder and copy or retrieve data into the selected system storage elements in conjunction with the 4 basic system functions described above.  Additional considerations for each of these operations are discussed below. 

Transfer to Server

The initial data transfer must achieve two basic functions – introduce the recording medium to the storage elements of the archive system and transfer the recorded data in a format that can be managed by the system’s data management server. 

The recorded data must be stored in a file-like data structure that is readily recognized by the operating system managing the archive.  This copied file may be in a hybrid format containing the recorded data as well as any user generated parameters that are associated with the file.  Once this hybrid file is stored on the archive system, a method must also be employed to support easy indexing for retrieval. 

Transfer to Storage Subsystem

The storage subsystem is designed to meet the access needs, usage requirements, and life expectancy of the data to be stored.  A pure disk solution, such as a scaleable RAID system, offers fast transfer rates to and from the media pack and is ideal for applications where the data has a useful shelf life of 3 to 5 years.  The volume of data in this application must also be considered.  If the data is not purged or copied to another medium, the amount of RAID storage required to hold the data over its useful life can rise dramatically. 

Transfer for Analysis

Once the data is removed from the recorder, a method must be employed to allow the data to be moved to a location and in a file format that supports analysis.  This can be accomplished by returning the data to a media pack for play-out in a recorder, by configuring the archive to support direct play-out, or by providing network access to the stored data.  The system should also allow users to group recorded data into meaningful sets to aid in analysis. 

Transfer to Long Term Archive

If the recorder data has long term value, then the system must include an archival quality medium such as optical disk.  Optical media has noticeably slower transfer rates than hard disk medium.  However, when the data has long term value, i.e. 15 years or more, optical storage is the best medium to use.  The introduction of an optical storage element, typically a robotic library system, requires the use of a specialized file system to manage the many individual platters that can be supported by the library.  This additional file system and management layer also impacts the performance of the copy and transfer process.  This tradeoff should be considered when planning the storage subsystem.

A True Archive Solution

This paper presents a high level overview of some of the issues to consider when designing a signal storage system and selecting appropriate storage technologies.  Rising Edge Technologies components and systems, such as our SA-1000 Signal Archive, can be used as a key piece in meeting the goals of an archive application.  Each of the topics presented must be considered in relation to the type of applications the archive will support, the level of user interaction with the media, workflow considerations, and performance demands. 

Rising Edge Technologies works with its customers to provide all the elements required to create a comprehensive signal capture, storage and archive system.  Contact us to discuss how the SA-1000 and our recorder products can be adapted to meet your mission needs.