Quản trị dữ liệu và trực quan hóa

INTRODUCTION
Cloud object stores such as Amazon S3 [4] and Azure Blob
Storage [17] have become some of the largest and most widely used storage systems on the planet, holding exabytes of data for millions of customers [46]. Apart from the traditional advantages of clouds services, such as pay-as-you-go billing, economies of scale, and expert management [15], cloud object stores are especially attractive because they allow users to scale computing and storage resources
separately: for example, a user can store a petabyte of data but only run a cluster to execute a query over it for a few hours.

OLTP (ON-LINE TRANSACTION PROCESSING)
- is characterized by a large number of short on-line transactions (INSERT, UPDATE, DELETE).
- The main emphasis for OLTP systems is put on very fast query processing, maintaining data integrity in multi-access environments and an effectiveness measured by number of transactions per second.

Exercise (contd.)
1. Define a star schema to represent the above
multidimensional structure;
2. Define a snowflake schema that reduces (at least on one
dimension) the redundancy of the star schema defined at
the previous point;

Data Pipeline V1
• Took 1 engineer ~1 week to implement
• Was pretty robust for the early days of Databricks

Record Label.

A record label wants to keep track of all contracts they have with bands, records they are producing and the sales of these records. Currently they are only keeping data of their ongoing contracts; no historical information is kept. Therefore it is decided to construct a data warehouse for collecting and storing historical information. With the data warehouse the company wants to analyze its sales. Based on conversations with the managers of the company, you were able to compile the following description of the available data.

Agenda
- Big Data Architectures
- Why data lakes?
- Top-down vs Bottom-up
- Data lake defined
- Creating ADLS Gen2
- Data Lake Use Cases

Introduction
This paper argues that the data warehouse architecture as we know it today will wane in the coming years and be replaced by a new architectural pattern, which we refer to as the Lakehouse, char-acterized by (i) open direct-access data formats, such as Apache Parquet and ORC, (ii) first-class support for machine learning and data science workloads, and (iii) state-of-the-art performance.

What is Big Data?
• A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications
• Due to its technical nature, the same challenges arise in Analytics at much lower volumes than what is traditionally considered Big Data.

Introduction
Remember the Challenger O-ring debacle? On 28 January, 1986 NASA managers had to decide if the circular rubber rings used to join sections of the orbital spaceship could perform. Unusually cold for the Florida launch pad, temperatures had dipped to about 28 degrees Fahrenheit (-2 degrees Celsius). Pretty cold for rubber O-rings that must flex to work.
In a teleconference, engineers at Morton Thiokol recommended against launch. They said they had no
performance data at temperatures below 53 degrees Fahrenheit (11.7 degrees Celsius).

What is a Data Warehouse?
■ Defined in many different ways, but not rigorously.
■ A decision support database that is maintained separately from the organization’s operational database
■ Support information processing by providing a solid platform of consolidated, historical data for analysis.