Custom file systems as optimal solutions for huge media data archives and storages

During the last decade, the volume of industrial video and audio recordings has been growing exponentially. IDC, a leading market research firm, estimates that the total amount of stored digital information is around 2.8 Exabytes (2.8 million Terabytes) with tenfold expected growth every five years! This huge and growing volume sets difficult tasks for the industry of media archive management (MAM). An amount of data, both current and archived, stored by a single TV production company, can soon reach thousands of Terabytes (a Petabyte). This information needs to be securely stored, managed, and most important, quickly and efficiently searched through and accessed.

In this paper we will not discuss pros and cons of different media storages. Instead, we will explore the problems and solutions of archived storage management. The main issues with storage and archive managements may be roughly classified into four categories.

  1. The unusually big and growing volume of storages;
    The storage size requirements for video and audio increase constantly mainly because of two factors: increase of broadcast volume and increase in definition. There is no doubt that the advent of High Definition TV will not end the 'quality race', and picture improvement requirements and expectation will grow to match, at least, those provided by 3D OMNIMAXes.
  2. The heterogeneous and distributed nature of archives;
    While storage media itself becomes less and less expensive, the cost of the total archive upgrade to a new type of media is prohibitive for any media company. It is the ubiquitous situation that archivists and engineers face problems of managing and integrating the data stored in legacy formats, such as tapes with up-to-date archives on hard disks, solid media, etc.
  3. Need for fast search and retrieval based on metadata;
    Fast search and retrieval of information from distributed heterogeneous archives is a headache for archivists and company IT personnel. Data accumulated during decades needs to be quickly and efficiently found in archives by brief descriptions, keywords, date of creation, etc. TV production engineers also benefit from short low-resolution extracts from the footage called proxies, which the majority of outdated storages fail to provide.
  4. Reliability and impenetrability of storages.
    The last but not least problem, is insuring that all the data is stored safely and securely. By safety, we understand the protection against data loss due to system malfunction(s). Any archival system should be able to mitigate hardware failure and restore partially or completely data lost with minimal efforts. Data security functions protect data against an intruder striving to obtain and use data illegitimately. Since many investigation and footages can be confidential and damaging to the company or a third person, the issue of safeguarding them is of utmost importance.

There are several solutions on the market, addressing all or a majority of the problems discussed above. They range from fragmentary 'patching' approaches, trying to reconcile legacy with current and future requirements, claiming to be universal 'fit all in one box' approaches.

One of the possible solution to the storage and archive problems is the use of proprietary file systems. Huge data storage requirements may be easily and comfortably accommodated by a file system, providing either a convenient way of placement and retrieval of complex data into a single virtual storage, or by providing a user mode interface - presenting data stored elsewhere as a file and folder similar to "ordinary" file systems. Neither user, nor legacy third party application have to worry how these data are actually stored and retrieved. This task is left to developers implementing all required callback functions.

The same approach - the use of proprietary file system - helps with heterogeneous legacy storages. There is no need to invest large sums of money to transfer all archives onto storage media once or twice in a decade. A file system based on callbacks, allows easy extendability without sacrificing of the past development. Upon appearance of new storage formats or attractive media opportunities, only several new functions need to be implemented, while the system as a whole continues to work with minimal, if any, service interruptions.

Needless to say, the handling of metadata by widespread popular file systems is less than efficient. There is no easy way of searching for necessary piece of information by user defined non-standard multi-modal annotations. Efficient implementation of custom task-specific file system will permit the creation and storage of any metadata associated with an original footage: starting from additional manual annotation to complex proxies, extracts, etc. Any sort and retrieve operations based on these metadatas, will not require development and implementation of additional technologies.

Data integrity and reliable storing may be assured by using journaling function provided by custom file systems. Constant monitoring and recording of all data-related operations, make failure recovery a feasible task. This journaling can be performed on-a-fly without human involvement and in resource-economical way, thus making possible continuous data protection.

Data security is assured by the use of modern encryption / decryption algorithms. Many symmetric key based tools, included in as a standard option in custom file systems, may quickly and easily encode and decode files of any type during the upload or download process. This will efficiently protect the data from many attack vectors, and save human factor.

Temporary data integration solution designed to patch current deficiency, are not expendable and may be cost-effective only in a very short run. Modern developments of file-system softwares, significantly simplify development, implementation, and the use of large archival system with legacy issues, such as media data storages. These technologies help to bridge the legacy chasm in the most efficient way.

Ready to get started?

Learn more about Callback Technologies or download a free trial.

Download Now