In a blog introducing the Data Lake Framework, keen readers will be aware that in the diagram there’s a box titled “ARCHIVE” but it has not been brought up since. The reason why the Archive layer in the data lake has not been discussed is because we’ve been waiting for the Archive Tier in Blob Storage.
To remind readers of the framework and where the archive layer sits in it, here it is again with the archive layer highlighted.
The Archive Blob
The Archive access tier in blob storage was made generally available today (13th December 2017) and with it comes the final piece in the puzzle to archiving data from the data lake.
Where Hot and Cool access tiers can be applied at a storage account level, the Archive access tier can only be applied to a blob storage container. To understand why the Archive access tier can only be applied to a container, you need to understand the features of the Archive access tier. It is intended for data that has no or low SLAs for availability within an organisation and the data is stored offline (Hot and Cool access tiers are online). Therefore, it can take up to 15 hours for data to be made online and available. Brining Archive data online is a process called rehydration (fitting for the data lake). If you have lots of blob containers in a storage account, you can archive them and rehydrate them as required, rather than having to rehydrate the entire storage account.
Archive Pattern
An intended use for the Archive access tier is to store raw data that must be preserved, even after it has been fully processed, and does not need to be accessed within 180 days.
Data gets loaded into the RAW area of the data lake, is fully processed through to CURATED, and a copy of the raw data is archived off to a blob container with a Cool access tier applied to it. When the archive cycle comes about, a new Cool access tiered blob container is created and the now old container has its access tier changed to Archive.
For example, our Archive cycle is monthly and we have a Cool access tiered blob container in our storage account called “December 2017”. When data has finished being processed in the Azure Data Lake, the Raw data is archived to this blob container. January comes around, we create a new blob container called “January 2018” with Cool access tier settings and change the access tier of “December 2017” from Cool to Archive.
This data has now been formally achieved and is only available for disaster recovery, auditing or compliance purposes.
Meet the Team – Zach Stagers, Senior Consultant
Up next in our series of Meet the Team blogs is Zach Stagers, Senior Consultant.
Nov
A Cost Comparison between PaaS and IaaS Cloud Architectures
Introduction This aim of this blog is to do a cost comparison between two cloud
Nov
Providing Careers at Adatis
At Adatis, we believe in developing and training our employees which aligns with 2 of
Nov
Meet the Team – Donna Bavin, People Operations Lead
We couldn’t achieve all the great work we do without our amazing team, so in
Nov
Deep Dive into the Common Data Model Video Presentation
At SQLBits last month some of our team gave fantastic sessions on a variety of
Oct
The Adatis Values – No is OK and Listen & Challenge
Our nine company values are super important to us and are at the heart of
Oct
Optical Character Recognition (OCR) using Tesseract and Databricks
The topic of Optical Character Recognition (OCR) is not an unexplored field to the Adatis
Oct
An experiment with Azure Custom Vision
An experiment with Azure Custom Vision Azure Cognitive Services is a comprehensive family of APIs,
Oct