We’ve been talking about data retention. AZ Jim reminds us in his comments, that the best way to manage data, primarily the deleting of data, is to establish a policy and implementation prior to turning the system on. He also suggests that, for those situations where data may not be deleted, that we have different levels of storage, where data may not be available immediately, but has not been deleted forever.
One of the coolest things I’ve read about SANS having storage with different levels of performance, and cost. Large disk volumes of inexpensive disks can store data much like an archive, yet with faster retrieval than having to go off site for a tape. The same SAN may also have high speed disks, and even SSD storage, and may even include a large RAM cache for faster read/write tasks. Some of these SANS allow you to express the type of data you are storing, and will, with some algorithm, allocate storage on the device with the appropriate performance to the type of data you are saving.
What this means is that the SAN is the determining factor as to where your data should be stored. You configure the SAN with the storage, and the logic for saving your data, and the SAN does the heavy lifting on your behalf. Having written a large number of archival programs over the years, it wouldn’t take much to convince me to have a SAN take over that load.
In the end, regardless of how it is maintained, you are probably going to end up with a very large database storing data for all time, which is really not that bad if the database is small. For larger databases, you’ll probably want to divide your data up into different categories of importance. Many simply make a distinction by maintaining large amounts of detail in a reporting database, while purging older data out of the source OLTP database. The reporting database may even be broken out further with older data being partitioned and stored on slower, large volume disk, while the more current data is maintained on a partition hosted on fast performing disks.
Have you built a system from scratch where you were able to take advantage of different storage mediums? Share that, or other thoughts that may be on your mind, by leaving a comment.
Cheers,
Ben