Is Archiving Important?

We process more data now than ever before…

We process more data now than ever before and as video resolutions increase, the requirement for more space will continue to grow exponentially. Technology is helping us to adjust the way we store, operate and manage our information in order to keep up with the rapid growth of data. Some solutions include clever software like CAT VD or Dalet to help us index, categorise and search the content, these are often referred to as Media Asset Management (MAM) systems. Storage appliances are also becoming more advanced as our requirements change.

Adopting a more holistic approach to data storage is also important to manage sustainable (often collaborative) workflows and keep long term storage costs to a minimum. When it comes to storage I have a few thoughts to consider.
[unordered_list style=”tick”]

• More space isn’t always the answer, our data normally magically grows to fill up any unallocated terabytes.
• Professional storage is not the same as consumer storage. When it comes to performance, reliability and data integrity, professional products are worth the upfront investment. If you don’t believe me just ask anyone that has had to pay for data recovery.
• Define the purpose of the storage before you buy – there are thousands of alternatives out there, choose what you need according to your priorities. I’ll discuss this in a little bit more detail below.
• Think seriously about a tiered storage approach and archiving, it’s a very effective way to “recycle” your existing storage.
• What after sales support exists in South Africa for the product? Is it supplied and installed by a reputable reseller and backed by an established distributor? How long would it take to bring the system back online if there was a problem?

[/unordered_list]
So let’s look at how one might introduce a tiered structure with different storage systems, each with a unique purpose. The idea is that content is moved to less expensive storage over time in order to free up space for current project requirements.

Before we go any further it’s important to clarify that archiving is different to backing up. Backing up entails copying information and protecting against data loss. Archiving is simply moving data to a more suitable long term storage medium.

Please refer to the diagram below which may apply to post production facilities. Online storage is typically a really fast RAID (Redundant Array of Independent Disks) where content for current projects is stored, compiled and edited. Single users will normally connect with Thunderbolt or SAS, whereas a team of users needing to access the content concurrently will probably access the storage via Fibre Channel or 10GBe.

The Tiger Box is a great example of an online storage appliance that offers up to 16 users concurrent access to the storage, support for all major editing workflows (including AVID) and enough performance to edit 2K footage. A single box ships with up to 64TB of space and is scalable to 320TB by adding expansion units. Tiger Box also includes a basic MAM called ProjectStore that allows users to index and search the content, create low res proxy videos and archive projects to alternative storage on the network.

This is where near line storage becomes useful, it’s essentially a slower type of disk based storage that is also accessible to users on the network and is significantly more cost effective per gigabyte than online storage. Rather than continually expanding the online storage volume whenever it runs out of space, non-current content is moved over to the near line volume (where it is still available to users) and the online storage can effectively be redeployed for new projects. The easiest way for all users to have access to near line storage is via a network which is why many companies opt for Network Attached Storage (NAS). Thecus manufacture an excellent range of NAS products, some of which have 10GBe capabilities and can be deployed as very basic online storage units.

When it comes to long term storage of media the best solution is still LTO tape. Although it’s cost effective to store old projects on portable hard drives, there are several down sides to this approach. Hard drives have a short shelf life and many will not detect after standing on a shelf for 5 years, not to mention that it’s also very difficult to keep track of what has been saved on each drive.

LTO tapes are a little different to hard drives because they require a tape drive for reading and writing data and store content in .TAR or .LTFS file formats. LTO6 tapes have a capacity of 3TB and have a shelf life of at least 25 years, they are ideal for keeping data safe for long periods of time. This is ideal for raw footage and completed projects that are not needed on a regular basis Good archiving policies will free up volumes of space in online and near line storage arrays. The process of archiving to tape is made easier with software like Bru PE. Xendata is an advanced software solution that gives LTO archives the functionality of disk based storage which means that all content can be managed by MAM software, archived to tape, easily searched or retrieved almost instantly.