Databricks Competition Study
What are the 4 main elements of data governance?
1) Availability of Data (where is it, who has access) 2) Usability of data (what format, what language is used in processing the data) 3) Integrity of data 4) Security of data
What is the Machine Learning Process?
1) Data Preparation 2) Exploratory Data Analysis 3) Feature Engineering 4) Model Training 5) Model Validation 6) Deployment 7) Monitoring
what are the existing problems of Data cleanrooms?
1) Data movement and replication - used to be in vendor lock-in and added storage cost 2) Restricted to SQL - can't run arbitrary workloads and restricted to simple SQL statements 3)Hard to Scale- can't expand collaboration (stuck with one vendor)
What are the motivations behind edge computing/fog computing?
1) Low Latency 2) Poor Connectivity 3) Reduced Server Load 4) Privacy Examples: Facial Recognition, Sensors in jet engines, Sensors in cells
Why Databricks over EMR?
1) Lower TCO (Total Cost of Ownership) 2) Reliable Analytics 3) Collaborative 4) Scalable Data Pipelines
How does Delta Lake differ Iceberg?
1) Overall performance - loading and querying data is 3.5x faster 2) Load Performance- load from Parquet to intended formats (delta is faster) 3) Query performance - Delta is 4.5x faster
What are the 4 components of MLFlow?
1) Tracking 2) Projects 3) Models 4) Model Registry
What 2 ways can you ingest Data into a lakehouse?
1) network of data ingestion through partners 2) integrating data into delta lake with Auto Loader
what are the 2 main risk in ML systems?
1) technical risk - inherent to the system itself 2) risk of noncompliance with external systems
How to know if it's a production workload?
1) workload is operationalized 2)new data is processed on a regular basis (batch/streaming) 3)key parts have been automated through workflows/jobs 4)drives business value
What were the 3 components of Hadoop?
1)HDFS- storage unit (Hadoop Distributed File System) stored on blocks and then stored on several data nodes in the cluster 2)MapReduce - data processing 3) YARN - resource negotiator (managed the resources)
Each node is how many Megabytes?
128 megabytes
What are the key features of Delta Lake
ACID Transactions, Time Travel, Schema Enforcement, Audit History, Iceberg to Delta Converter, Optimize ZOrder, Clones Etc
Sagemaker
AWS product that is a fully managed ML platform that enables data scientists and developers to create, train and deploy ML models
What was the first major solution to distributed storage?
Apache Hadoop - only built for local storage not for processing
What Databricks Products address Hadoop migration needs for data engineering?
Auto Loader, DLT, Databricks Workflows
What is the difference between batch processing and streaming processing?
Batch is data at rest. (allows to look into more depth) Stream - data is constantly arriving Streaming requires a different infrastructure, uses different methods and has different goals - looking for anomalies or quick trends
What is Edge Computing?
Central storage and processing and then have connected devices at the edges of the network. The devices can store and process the data
In MEDDPICC, what does the C's stand for?
Champion: the person who has power, influence, and credibility within the customers' organization Competition: what other vendors the company is evaluating
Why have data marketplaces seen limited use?
Closed platforms (one per vendor) Limited to just datasets
Databricks vs Sagemaker?
Comparing Databricks to Sagemaker is an unfair comparison because when comparing a complete end-to-end platform Sagemakeer requires 20+ other services Sagemaker- overhead and complexity, deteriorated team collaboration, data duplications higher TCO and tedious governance
Azure Synapse Weakness
Complex and has many different options for computing, storage, and required integrations have 4 different compute options and 3 storage options when it comes to streaming synapse relies on other services like Azure ML and Azure Stream Analytics Vendor Lock-In - can't be multi-cloud
What is Model Training in the ML lifecycle?
DS explore algorithms to find the best performing model
What are the major risk of storing data on the cloud?
Data Dissipation - Security Cost Containment Remote Possibility - servers could fail
How are data engineers used in the ML lifecycle?
Data Engineers prepare production data and make it available for consumption (raw data)
How does Databricks differentiate from Snowflake?
Data Science/Machine Learning Data Ingestion ETL Streaming Capabilities Data Sharing With Snowflake, you will have high costs, limited capabilities, limited unstructured data support, inefficient engineering and vendor lock in
What Databricks Products address Hadoop migration needs for Data Warehousing?
Databricks SQL
Streaming Capabilities- how does Databricks differ from Snowflake
Databricks reads in any streaming services while Snowflake only supports Kafka and was not designed for high velocity data but rather structured data at rest
In MEDDPICC, what does the D's stand for?
Decision Criteria: are the various criteria in which a decision to process your criteria will be judged Decision Process: series of steps that form a process of which the buyer will follow to make a decision
What is Delta Lake?
Delta Lake - bringing data warehousing functionality to data lakes It's a storage framework that takes data in Parquet format (everyone in Big Data knows this data format) and adds an abstraction layer to it (transaction log, metadata etc) which resolves the key issues that have been a pain for all practitioners. Data stays in their own cloud object store
What is Delta Live Tables?
Delta Live Tables is the first ETL framework that allows you to build reliable data pipelines (accelerate ETL development) - automatically manages your infrastructure at scale so data analysts and engineers can spend less time on tooling and focus on getting value from data. DLT- fully supports Python and SQL and works with both batch and streaming DLT - manages task oircehstration, cluster management, monitoring, data quality and error handling supports CDC (change data capture)
MLFlow Models
Deploy machine learning models in diverse serving environments
What companies have migrated from AWS EMR to Databricks?
Disney+ - 15.6 million infrastructure savings with reduction in compute costs for data processing, modeling and analytics Plume Wild Life Ancestry
Price difference between Databricks and EMR?
EMR is relatively fast and extremely cheap however it comes with high operational cost, operational overhead and risks They are giving away EMR for free to sell EC2 hours
EMR
ETL/stream processing many open sourced projects (Hadoop, Presto, Jupyter) are packaged and rolled into a single click which integrates well with AWS ecosystem cloud big data platform for running large scale distributed data processing jobs, interactive SQL queries EMR- EMR runtime which provides performance enhancements
In MEDDPICC, what does the E stand for?
Economic Buyer: the person with the overall authority in the buying decision
What Databricks features helps out with observability and governance?
Expectations- help prevent bad data from flowing into tables, track data quality over time and provide tools to troubleshoot bad data with granular pipeline observability so you get a high fidelity lineage diagram of your pipeline, track dependencies and aggregate data quality metrics across all of your pipeline.
What is Project Lightspeed?
Faster and simpler stream processing 1) predictable low latency 2) enhanced functionality 3) Operations and Troubleshooting 4) Connectors & Ecosystem
What Databricks Products address Hadoop migration needs for Data Science and ML?
Feature Store, MLFlow, ML Runtime, MLOps and AutoML
Databricks Workflows
Fully managed orchestration service Orchestration is the configuration of multiple tasks into one complete end-to-end process the goal is to streamline and optimize the execution of frequent, repeatable processes
At Summit, what was Snowflake's biggest announcements?
Iceberg Tables, Snowflake for Python, and Snowpipe Streaming -also support for transactional workloads with Unistore and Hybrid Tables
In MEDDPICC, what does the I stand for?
Implicating the Pain: identified the pain and identified the solution that your pain solves
What are Data cleanrooms?
It's a secure environment to run computations on joint data Run any computation on Python, SQL, R, or java No data replications Scalability
How is HDFS Fault Tolerant?
Makes copies of the data and stores across multiple systems -replication of 3 nodes
In MEDDPICC, what does the M stand for?
Metrics: are the quantifiable measures of value that your solution can provide
What are the different types of SQL Workloads?
Online Transaction Processing (OLTP) Workloads, Decision Support System or data warehouse (DSS or DW) Workloads, Online analytical processing (OLAP) Workloads
MLFlow Projects
Package data science code in a format to reproduce runs on any platform
In MEDDPICC, what does the P stand for?
Paper Process: how will you go from decision to signed contract
What are the 4 ways companies make decisions?
Revenue Growth Cost Reduction & Savings Strategic Initiatives Driving Efficiency
In MEDDPICC, what does the R stand for?
Risks: specific risks that will need to be monitored in order to get the deal done
What is Structured Data?
Rows and columns Variables are predefined Ex: Spreadsheets, rational databases
How do we position with Snowflake
Snowflake handles DW capabilities while Databricks handles the DS/ML capabilities
For ETL- how does Databricks differ from Snowflake
Snowflake has no ETL tools rely heavily on 3rd party vendors which increases cost and complexity Databricks - Delta Live Tables
For Data Science- how does Databricks differ from Snowflake?
Snowflake is built for SQL workloads. They either use Snowpark or rely on 3rd party tools for DS and ML-like Dataiku or DataRobot Databricks can allow your data scientist to use any libraries/languages in the same platform as their DE, and manage the end-to-end ML pipeline with MLflow
For Data Sharing- how does Databricks differ from Snowflake?
Snowflake's data format is proprietary, users can only share data with other Snowflake accounts (vendor lock in) -Snowflake would take the data from your cloud storage, conduct transformations and then push the data back so you are paying an egress tax to and from and as that data is going to be stored -pay for compute to send data Delta Sharing - an open standard for data sharing, no replication of datasets
Velocity
Speed with which the data arrives -data that comes in rapidly and changes frequently
What Databricks Products address Hadoop migration needs for Streaming workloads?
Structured Streaming, DLT, AutoLoader
What is artificial intelligence?
Techniques that allow computers to do things typically done by humans
How are text, audio, and video data measured?
Text is measured in Kilobytes Audio is measured in megabytes Video is measured in gigabytes
What does the Lakehouse consist of?
The lakehouse brings the scalability and cost effectiveness of data lakes with the reliability and performance of data warehouses
Why Databricks over Synapse?
Unified collaborative environment and a single compute engine Significantly better cost performance Multi cloud and open Best for 2 primary uses cases correlated to Lakehouse architecture Data sharing live tables without copying -Synapse - copy their data
What is hybrid cloud?
Using both public cloud providers and secure private cloud
What is Multi-cloud?
Using several cloud storage and computing providers simultaneously
What is Unstructured data?
Variables and fields are not labeled or identified Ex: text, photos, videos, audio
What is Semi-Structured data?
Variables that are marked with tags HTML, EXML, JSON (webpages and social media data)
When is anomaly detection important?
When identifying process failure, potential value (find outliers) and fraud detection
How do we position with AWS Sagemaker?
Win with Delta - allows the need for a data-centric ML platform and ease of integration through MLflow Sagemaker support Serverless Model Endpoints
For Data Ingestion- how does Databricks differ from Snowflake
With Snowflake ingestion go through various Snowflake stages, often needing to utilize limited SQL pipelines or Snowpipe which is inefficient and expensive 2) Snowflake tax - egress tax moving data to and from and as that data is being stored Databricks - ingestion is simple with AutoLoader where data is automatically transformed into Delta Tables
YARN consists of 4 components?
Yet another resource negotiator 1)Resource Manager 2)Node Manager 3)Application Master 4)Containers YARN - manages the resources
Databricks SQL
a
What is Databricks Workflows?
a fully managed orchestration service for all your data analytics and AI -allows users to build ETL pipelines that are automatically managed, including ingestion and lineage using Delta Live Tables -"repair and rerun" failed jobs -provides a deep monitoring capabilities and centralized observability across all your workflows
What do Workflows allow you to do?
allow users to build ETL pipelines that are automatically managed Orchestrating and managing production workflows are. bottleneck for many organizations and require external tools like Apache Airflow, Azure DataFactory, AWS Step Function, or GCP workflows
MLFlow Tracking
allows you to record and query experiments: code, data, configure results
What is Exploratory data analysis in the ML lifecycle?
analysis is conducted by data scientist to assess statistical properties of the data available and determine if they address the business question
What is AutoLoader?
automatically processes files landing in cloud storage
MLFlow Model Registry
collaborative hub for all ML models - use webhooks to automate and integrate your machine learning pipeline with existing CI/CD tools and workflows
What is Machine Learning?
collection of algorithms that can find patterns in data to predict outcomes -improve over time -can go from simple with linear regressions to more complex solutions with neural networks
What is Apache Spark?
data processing language engine for executing data engineering, data science and ML on a single node machine or clusters
What is Feature Engineering in the ML lifecycle?
data scientist clean data and apply business logic and specialized transformations to engineer features for model training
Variety
format of the data (Structured, Unstructured or semi-structured data)
What is a data lake?
hold data that is structured, semi-structured and unstructured -gets rid of data silos so all organizations can access the data
Volume
how much data you have - more data than fits in a computer's RAM (memory) -more data than fits on a hard drive
What is MEDDPICC?
is a qualification methodology
What is MLOps?
is a set of processes and automation for managing models, data, and code to improve performance stability and long-term efficiency in ML systems MLOps = ModelOps + DataOps + DevOps
What is MLFlow?
is an open source platform to manage the ML lifecycle including experimentation, reproducibility and a central model registry
When you talk about ETL, you also hear about Production Workloads. What are production workloads?
jobs that are scheduled, automated and critical to our customers running their everyday business
What does EMR lack?
lacks simplicity in management and configuration set up, has high dependency to multiple other AWS tools and performance wise it is not on par with Databricks
What is Unity Catalog?
offers a centralized governance solution for all data and AI assets with built in search and discovery newest announcements around data lineage- enhances performance and scalability and gives business a complete view of the entire data lifecycle
What is Databricks Marketplace?
open marketplace for data solutions, built on Delta Sharing Consist of Notebooks Data files, Data Tables Solution Accelerators ML Models Dashboards
What is Iceberg Tables?
open table format- pairs Snowflake's powerful performance engine and platform capabilities with open formats and storage managed by customers -competes with Delta Lake
What is Photon
query engine on Databricks which provides up to 12x better performance compared to other cloud data warehouses
Databricks SQL
serverless data warehouse that lets you run all of your SQL and BI applications at scale
What is MapReduce?
splits the data into parts and processes them separately on different data nodes and then the results are aggregated to give a final output
Hadoop
stored and processed vast amounts of data efficiently using cluster of commodity hardware (multiple storage units and processors framework that manages big data storage in a distributed way and processes it
What is Snowpipe Streaming?
supports streaming data (only kafka support) Databricks supports Kinesis, Event Hubs Kafka and many more
What is Snowpark with Python?
targets Application development workloads both with native Streamlit integrations and for ISV partners -can read files and tables and apply basic transformations Snowpark is a proprietary API that woks only on Snowflake compute
What is DevOps?
the concept of developer operations - deploy software applications
What is a Data Warehouse
unified place to keep an organization's data sets - handles structured data
What are the 3 V's of big data?
volume, velocity, variety
