Azure Data Fundamentals

Describe stream processing

Process that allows you to almost instantaneously from one source (i.e. device) to another (i.e. log file).

Describe batch data

Running process to completion one after another, at once. Batch processes are not interactive.

key-value store

Similar to a relational table, except that each row can have any number of columns

Analytical workloads

Typically read-only systems that store vast volumes of historical data or business metrics, such as sales performance and inventory levels

SaaS

Typically specific software packages that are installed and run on virtual hardware in the cloud

lifecycle management policy

automatically move a blob from Hot to Cool, and then to the Archive tier, as it ages and is used less frequently

Bar and column charts

enable you to see how a set of variables changes across different categories

ELT

Extract, Load, and Transform. The process differs from ETL in that the data is stored before being transformed

Azure Database for MySQL

is a PaaS implementation of MySQL in the Azure cloud, based on the MySQL Community Edition

entity

is described as a thing about which information needs to be known or held.

Block blobs

is handled as a set of blocks. Each block can vary in size, up to 100 MB

Page blobs

is organized as a collection of fixed size 512-byte pages

Archive tier

provides the lowest storage cost, but with increased latency.

Describe a bar chart

representation of categorical data using rectangles

Describe a pie chart

representation of data using a circle divided into proportional slices

Avro

row-based format. It was created by Apache. Each record contains a header that describes the structure of the data in the record.

Power BI visualization

a visual representation of data, like a chart, a color-coded map, or other interesting things you can create to represent your data visually

scaling

act of increasing (or decreasing) the resources used by a service

data warehouse

also stores large quantities of data, but the data in a warehouse has been processed to convert it into a format for efficient analysis.

Atomicity

guarantees that each transaction is treated as a single unit, which either succeeds completely, or fails completely

Synapse Spark pool

is a cluster of servers running Apache Spark to process data. You write your data processing logic using one of the four supported languages: Python, Scala, SQL, and C# (via .NET for Apache Spark). Spark pools support Azure Machine Learning through integration with the SparkML and AzureML packages.

Synapse SQL pool

is a collection of servers running Transact-SQL. Transact-SQL is the dialect of SQL used by Azure SQL Database, and Microsoft SQL Server. You write your data processing logic using Transact-SQL.

SQL Database server

is a logical construct that acts as a central administrative point for multiple single or pooled databases, logins, firewall rules, auditing rules, threat detection policies, and failover groups

Synapse Pipelines

is a logical grouping of activities that together perform a task.

NoSQL

is a rather loose term that simply means non-relational.

Azure Table Storage

is a scalable key-value store held in the cloud. You create a table using an Azure storage account.

Range index

is based on an ordered tree-like structure

transactional system (OLTP)

is often what most people consider the primary function of business computing, it records transactions

dot plot chart

is similar to a bubble chart and scatter chart, but can plot categorical data along the X-Axis

Provisioning

is the act of running series of tasks that a service provider, such as Azure SQL Database, performs to create and configure a service

Hot tier

is the default. You use this tier for blobs that are accessed frequently

Azure Cosmos DB

schema-agnostic database that allows you to iterate on your application without having to deal with schema or index management. Automatically indexes every property for all items in your container without having to define any schema or configure secondary indexes

scatter chart

shows the relationship between two numerical values

Analysis

You typically use batch processing for performing complex analytics. Stream processing is used for simple response functions, aggregates, or calculations such as rolling averages.

Azure Database for PostgreSQL

a PaaS implementation of PostgreSQL in the Azure Cloud. This service provides the same availability, performance, scaling, security, and administrative benefits as the MySQL service.

Azure HDInsight

a big data processing service, that provides the platform for technologies such as Spark in an Azure environment.

Change Feed

a blob provides an ordered, read-only, record of the updates made to a blob.

Azure Data Factory

a cloud-based data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale

Power BI datasets

a collection of data that Power BI uses to create its visualizations.

Power BI report

a collection of visualizations that appear together on one or more pages

column family database

a column family database can appear very similar to a relational database, real power lies in its denormalized approach to structuring sparse data

Cassandra API

a column family database management system

distributed database

a database in which data is stored across different physical locations

Azure Cosmos DB

a multi-model NoSQL database management system

Snapshots

a read-only version of a blob at a particular point in time.

data lake

a repository for large quantities of raw data.

Azure Virtual Network

a representation of your own network in the cloud.

Azure Blob storage

a service that enables you to store massive amounts of unstructured data, or blobs, in the cloud

Power BI tile

a single visualization on a report or a dashboard

document key

a unique identifier for the document

view

a virtual table based on the result set of a query

Parquet

another columnar data format. It was created by Cloudera and Twitter. Contains row groups. Data for each column is stored together in the same row group. Each row group contains one or more chunks of data.

Describe cognitive analytics

applying human like intelligence to tasks

Treemaps

are charts of colored rectangles, with size representing the relative value of each item

Cognitive analytics

attempts to draw inferences from existing data and patterns, derive conclusions based on existing knowledge bases, and then add these findings back into the knowledge base for future inferences--a self-learning feedback loop

unstructured data

audio and video files, and binary data files might not have a specific structure

stream processing

each new piece of data is processed when it arrives

SQL Database managed instance

effectively runs a fully controllable instance of SQL Server in the cloud. You can install multiple databases on the same instance. You have complete control over this instance, much as you would for an on-premises server

Line charts

emphasize the overall shape of an entire series of values, usually over time

Spatial indices

enable efficient queries on geospatial objects such as - points, lines, polygons, and multipolygon. used on correctly formatted GeoJSON objects. Points, LineStrings, Polygons, and MultiPolygons are currently supported

Azure File Storage

enables you to create files shares in the cloud, and access these file shares from anywhere with an internet connection

Azure Database Migration Service (DMS)

enables you to restore a backup of your on-premises databases directly to databases running in Azure Data Services

in-place updates

enabling an application to modify the values of specific fields in a document without rewriting the entire document

Consistency

ensures that a transaction can only take the data in the database from one valid state to another

Isolation

ensures that concurrent execution of transactions leaves the database in the same state that would have been obtained if the transactions were executed sequentially

Data processing solutions

fall into one of two broad categories: analytical systems, and transaction processing systems

Describe diagnostic analytics

form of analytics that answers the question, "Why did it happen?"

Filled map

geographical data, you can use a filled map to display how a value differs in proportion across a geography or region

Owner privilege

gives full access to the data including managing the security like adding new users and removing access to existing users.

Read/write access

gives users the ability to view and modify existing data.

Durability

guarantees that once a transaction has been committed, it will remain committed even if there's a system failure such as a power outage or crash

Cool tier.

has lower performance and incurs reduced storage charges compared to the Hot tier

Prescriptive analytics

helps answer questions about what actions should be taken to achieve a goal or target

Descriptive analytics

helps answer questions about what has happened, based on historical data

Predictive analytics

helps answer questions about what will happen in the future

Diagnostic analytics

helps answer questions about why things happened

index

helps you search for data in a table

Gremlin API

implements a graph database interface to Cosmos DB.

Composite indexes

increase the efficiency when you are performing operations on multiple fields

Describe descriptive analytics

interpretation of historical data to better understand changes that have occurred

Append blobs

is a block blob optimized to support append operations

Synapse Studio

is a web user interface that enables data engineers to access all the Synapse Analytics tools.

Azure Databricks

is an Apache Spark environment running on Azure to provide big data processing, streaming, and machine learning

Azure Synapse Analytics

is an analytics engine. It's designed to process large amounts of data very quickly

Azure Database for MariaDB

is an implementation of the MariaDB database management system adapted to run in Azure. It's based on the MariaDB Community Edition.

MongoDB API

is another well-known document database, with its own programmatic interface

Data Querying

looking in data for trends, or attempting to determine the cause of problems in your systems

Describe the concepts of data processing

manipulation of data by a computer

Read-only access

means the users can read data but can't modify any existing data or create new data.

modern data warehouse

might contain a mixture of relational and non-relational data, including files, social media streams, and Internet of Things (IoT) sensor data

Power BI dashboard

must fit on a single page, often called a canvas (the canvas is the blank backdrop in Power BI Desktop or the service, where you put visualizations).

Optimized Row Columnar format (ORC)

organizes data into columns rather than rows

clustered index

physically reorganizes a table by the index key. This arrangement can improve the performance of queries still further, because the RDBMS system doesn't have to follow references from the index to find the corresponding data in the underlying table

Business Intelligence

refers to technologies, applications, and practices for the collection, integration, analysis, and presentation of business information. The purpose of business intelligence is to support better decision making

document database

represents the opposite end of the NoSQL spectrum from a key-value store. Each document has a unique ID, but the fields in the documents are transparent to the database management system.

Single Database

resource type creates a database in Azure SQL Database with its own set of resources and is managed via a server

Azure database administrator

responsible for the design, implementation, maintenance, and operational aspects of on-premises and cloud-based database solutions built on Azure data services and SQL Server

Elastic Pool

similar to Single Database, except that by default multiple databases can share the same resources, such as memory, data storage space, and processing power.

Azure Database for PostgreSQL single-server

single-server deployment option for PostgreSQL provides similar benefits as Azure Database for MySQL. You choose from three pricing tiers: Basic, General Purpose, and Memory Optimized. Each tier supports different numbers of CPUs, memory, and storage sizes—you select one based on the load you expect to support

ETL

stands for Extract, Transform, and Load. The raw data is retrieved and transformed before being saved

Data Visualization

the graphical representation of information and data. By using visual elements like charts, graphs, and maps, data visualization tools provide an accessible way to spot and understand trends, outliers, and patterns in data

Soft delete

This feature enables you to recover a blob that has been removed or overwritten, by accident or otherwise.

Table API

This interface enables you to use the Azure Table Storage API to store and retrieve documents.

SQL API

This interface provides a SQL-like query language over documents, enable to identify and retrieve documents using SELECT statements

Azure Database for PostgreSQL Hyperscale

Hyperscale (Citus) is a deployment option that scales queries across multiple server nodes to support large database loads. Your database is split across nodes. Data is split into chunks based on the value of a partition key or sharding key. Consider using this deployment option for the largest database PostgreSQL deployments in the Azure Cloud.

Disadvantages of batch

The time delay between ingesting the data and getting the results. All of a batch job's input data must be ready before a batch can be processed. Even minor data errors, such as typographical errors in dates, can prevent a batch job from running.

Azure Data Services

These services are a series of DBMSs managed by Microsoft in the cloud. Each data service takes care of the configuration, day-to-day management, software updates, and security of the databases that it hosts. All you do is create your databases under the control of the data service.

batch processing

Arriving data elements are collected into a group. The whole group is then processed at a future time

Azure Analysis Services

Enables you to build tabular models to support online analytical processing (OLAP) queries

Versioning

can maintain and restore earlier versions of a blob.

matrix

visual is a tabular structure that summarizes data

Data Size

Batch processing is suitable for handling large datasets efficiently. Stream processing is intended for individual records or micro batches consisting of few records.

PaaS

Allows you to specify the resources that you require (based on how large you think your databases will be, the number of users, and the performance you require), and Azure automatically creates the necessary virtual machines, networks, and other devices for you

Data Scope

Batch processing can process all the data in the dataset. Stream processing typically only has access to the most recent data received, or within a rolling time window

Describe differences between batch and stream data

Batch data requires a dataset to be collected over time, stream data depends on an analytic tool that consumes data.

data analyst

Enables businesses to maximize the value of their data assets, responsible for designing and building scalable models, cleaning and transforming data, and enabling advanced analytics capabilities through reports and visualizations.

IaaS

Enables you to create a virtual infrastructure in the cloud that mirrors the way an on-premises data center might work. You can create a set of virtual machines, connect them together using a virtual network, and add a range of virtual devices

Describe data visualization

Field that deals with the graphical representation of data.

Data

Is a collection of facts such as numbers, descriptions, and observations used in decision making.

Semi-structured data

Is information that doesn't reside in a relational database but still has some structure to it. Examples include documents held in JavaScript Object Notation (JSON) format.

Advantages of batch

Large volumes of data can be processed at a convenient time. It can be scheduled to run at a time when computers or systems might otherwise be idle, such as overnight, or during off-peak hours.

normalized

Splitting tables out into separate groups of columns

Performance

The latency for batch processing is typically a few hours. Stream processing typically occurs immediately, with latency in the order of seconds or milliseconds.

data engineer

collaborates with stakeholders to design and implement data-related assets that include data ingestion pipelines, cleansing and transformation activities, and data stores for analytical workloads

Azure Data Lake Storage

combines the hierarchical directory structure and file system semantics of a traditional file system with security and scalability provided by Azure

relational database

comprises a set of tables. A table can have zero (if the table is empty) or more rows. Each table has a fixed set of columns. You can define relationships between tables using primary and foreign keys, and you can access the data in tables using SQL

Data analytics

concerned with examining, transforming, and arranging data so that you can study it and extract useful information

container

content is projected as a JSON document, then converted into a tree representation.

Azure Data Factory

described as a data integration service

analytical system (OLAP)

designed to support business users who need to query data and gain a big picture view of the information held in a database

key influencer chart

displays the major contributors to a selected result or value

Data Ingestion

the process of capturing the raw data

Reporting

the process of organizing data into informational summaries to monitor how different areas of an organization are performing

Describe predictive analytics

the use of historical data, statistical models and machine learning techniques to identify the likelihood of future outcomes.

Data Transformation/Data Processing

to do some cleaning operations and remove any questionable or invalid data, or perform some aggregations such as calculating profit, margin, and other Key Performance Metrics (KPIs)

Describe ETL

type of data integration refers to extract, transform and load to blend data from multiple sources

Describe ELT

type of data integration, takes advantage of the target system to do the data transformation

graph database

used to store and query information about complex relationships, contains nodes (information about objects), and edges (information about the relationships between objects).

Describe prescriptive analytics

using machine learning techniques to help businesses decide on a course of action to taken

non-relational system

you store the information for entities in collections or containers rather than relational tables. Two entities in the same collection can have a different set of fields rather than a regular set of columns found in a relational table