Amazon Certified Cloud Practitioner
Computer Vision Uses
-Interpret an image and suggest an appropriate caption comes with associated confidence score -Suggest relevant tags that could be used to index an image -Categorize an image -Identify objects in an image and provide with a bounding box -Detect faces and people in an image -Recognize celebrities and landmarks in an image -Read text in an image
Azure Machine Learning
A cloud-based platform for creating, managing, and publishing machine learning models.
Object Detection
A computer vision solution in which ML models are trained to classify individual objects within an image, and identify their location with a bounding box.
Semantic Segmentation
A computer vision solution in which individual pixels in the image are classified according to the object to which they belong.
Image Classification
A computer vision solution that involves training a ML model to classify images based on their contents. Example training a model to classify images based on the type of vehicle they contain.
Inference pipeline
A second pipeline that performs the same data transformation for new data and uses the trained model to inference label values based on its features. There is a drop down list that you use to create this pipeline. Two ways to get new data usually entering data manually and a web service input. Also want to execute a python script module before web service output to return only the predicted label.
Conversational AI
AI agents known as bots to provide a first-line of automated support through the full range of channels that we use to communicate. You need a knowledge base of question and answer pairs usually with some built in natural language processing moel to enable quesztions that can be phrased in multiple ways to be understood with the same semantic meaning. And you need a bot service that provides an interface to the knowledge base through one or more chanels.
Privacy and Security
AI systems should be secure and respect privacy. One of the ways they approach this is to keep the data on the customers device and run the models on that device. Considerations for data are where and how the data is coming from, if its user submitted or public data source, how do you prevent that from being corrupted. Anomaly detection for changes in data.
Transparency
AI systems should be understandable. Users should be aware of the purpose of the system, how it works, and what limitations can be expected. Two sides to transparency. People who create AI systems should be open to how and why they are using AI and limitations. Also means that people should be able to understand the behaviour of AI systems.
Inclusiveness
AI systems should empower everyone and engage people. It should bring benefits to all parts of society regardless of physical ability, gender, etc. Ensuring minority groups are involved in earliest concepts and future design and planning and that they are part of testing.
Reliability and Safety
AI systems should perform reliably and safely or else human lifes can be put at risk. These systems must be subjected to rigorous testing and deploying management processes to ensure that they work as expected before release. Quantify risk and harms when releasing products.
Deploying a predictive service
After you've created and tested an inference pipeline for real-time inferencing, you can publish it as a service for client applications to use. To publish a real-time inference pipeline as a service, you must deploy it to an Azure Kubernetes Service (AKS) cluster. Need primary key and endpoint to connect with the inference pipeline as a service.
Fairness
All systems should treat all people fairly. No basis on gender, ethnicitiy, or other factors that might offer an unfair advantage. Relates to not just the technical component but also the societal context in which the system is deployed. Greater diversity of people developing and deploying AI systems. Decisions made by teams can introduce biases.
Azure Machine Learning features and capabilities
Automated machine learning - This feature enables non experts to quickly create an effective machine learning model from data. Azure Machine Learning designer - a graphical interface enabling no-code development of machine learning solutions Data and compute management - Cloud-based storage and compute resources that professional data scientists can use to run data experiment code at scale. Pipelines - DS, software engineers, and IT operations professionals can define pipelines to orchestrate model training, deployment, and management tasks
Translate Text and Speech
Automated translation sometimes known as machine translation is being employed to solve this problem. AI systems must be able to understand not only the words but also the semantic context in which they are used. Text translation can be used to translate documents from one language to another, translate email commmunications that come from foreign governments and even provide the ability to translate web pages on the internet. Speech translation is used to translate between spoken languages sometimes directly speech to speech and sometimes by using text speech to text Use translator text service which supports text to text Or speech service which enables speech to text and speech to speech translation Or create cognitive services resource
Deploying a predictive service
Can either deploy a service as an Azure Container Instance or to an Azure Kubernetes cluster. For production scenarios, AKS deployment is recommmended for which you must create an inference cluster compute target. Need the REST endpoint and the Primary key for your service.
Clustering Specifics
Clustering is an example of unsupervised ML in which you train a model to seperate items into clusters based purely on their characterisitcs of features. K-Means Clustering model- assigns data in clusters designated as K. Works by initializing K coordinates as randomly selected points called centoids in n-dimensional space. Plotting the feature vectors as points in the same space and assigning each point to its closest centroid. Moving centroid to the middle of the points allocated to it. Evaluating model -metrics for each cluster Average Distance to Other Center - indicates how close on average each point in the cluster is to the centroids of all other clusters Average distance to Cluster Center - this indicates how close on average each point in the cluster is to the center Number of points Maximal Distance to Cluster Center- max of the distances between each point and the centroid of that points clusters this combined with average distance to cluster center helps you determine the cluster's spread
Image Analysis
Comining ML models with advanced image analysis techniques to extract information from images including "tags" that can help catelog the image or even descriptive captions that summarize the scene shown in the image.
Different types of compute targets
Compute Instances - development workstations that data scientists can use to work with data and models Compute Clusters- Scalable clusters of virtual machines for on-demand processing of experiment code. Used for training models. Also used to normalize data. Inference Clusters - Deployment targets for predictive services that use your trained models Attached compute - links to existing compute resources like VM or Azure Databricks
Azure Resources for Computer Vision
Computer Vision - SPecific resource for the Computer Vision service. Use this resource type if you don't intend to use any other cognitive services or if you want to track utilization and costs for your Computer vision resource seperately. Cognitive Services - general cognitive services resource that includes computer vision along with many other cognitive services. Use this resource type if you plan to use multiple cognitive services and want to simplifiy administration and de
Face Analysis on Azure
Computer Vision - face detection and some basic face analysis such as determining age Video Indexer - use to detect and identify faces in a video Face - offers pre-built algorithms that can detect, recognize and analyze faces
Computer vision Services in Microsoft Azure
Computer Vision- use to analyze images, videos and extract descriptions, tags, objects, and text Custom Vision - use this service to train custom image classification and object detection models using your own images Face - enables you to build face detection and facial recognition solutions Form Recognizer - Use this service to extract information from scanned forms and invoices
Text Analytics
Contains pre-trained models that can determine the language of a document of text, perform sentiment analysis on text to determine a positive or negative sentiment, extract key phrases from text that might indicate its main talking points, identify and categorize entities in the text. Can use a text analytics resource or cognitive services resources. Language detection- submit multiple documents at a time and the service will detect language name, language code, score indicating confidence levels if mixed languages will return predominant language. Sentiment Analysis - returns in the range of 0 to 1 with values closer to 1 being a positive sentiment. Key Phrase Identifier Entity Recognition - item of a particular type or category and in some cases subtype. Also supports entity linking to help disambiguate entities by linking a specific reference
Text-to-speech API
Convert text to speech which can either be played directly through a microphone or written to an audio file. Can use pre defined voices with support for mulitple languages.
Evaluating a model
Drag a "Evaluate Model" tab into the designer pane and view outputs and logs and visualize. This will give you various statistics in a regression model. Mean Absolute Error - Average difference between predicted values and true values. Lower the better Root Mean Squared Error - square root of the mean squared difference between predicted and true values. When compared to the MAE larger difference indicates greater variance in individual errors. Relative Square Error - relative metric between 0 and 1 based on the square of the differences between true and predicted values. Closer to 0 the better. Used to compare models where the labels are in different units. Relative Absolute error - Relative metric between 0 and 1 based on the absolute differences between predicted and true values. Closer to 0 the better. used to compare models where labels are in different units. Coefficient of Determination - Metric referred as R squared and summarizes how much of the variance between predicted and true values is explained by the model. Closer to 1 the better.
Face Detection and Facial Analysis
Face Detection - involves identifying regions of an image that contain a human face, typically by returning a bounding box coordinate that forms a rectangle around the face Facial Analysis - can return facial landmarks like nose lips to potentially identify an age or emotional status. Facial Recognition - further application of facial analysis to identify known individuals from their facial features
AI software development Principles
Fairness, Reliability and safety, Privacy and Security, Inclusiveness, Transparency, Accountability
Residual Histogram
Frequency of residual value ranges.
Score Model
How to evaluate your model. If you select this and visualize on the outputs and logs tab. You can see a predicted scored labels which contains the predicted label values.
Authoring
Language Understanding provides a comprehensive collection of pre built domains that include pre defined intents and entities for common scenarios. Portal for this step. For each intent you should include a variety of utterances that provide examples of how a user might express the intent. Can be applied to multiple entities. 4 types of entities machine learned during training from context in sample utterances, list such as adevice list, regex, pattern
Automated ML capability of Azure ML
Leverages scalability of cloud compute to automatically try multiple pre-processing techniques and training algorithms in parallel to find the best performing supervised ML model for your data
Face detection, analysis, and recognition
Locates human faces in an image. This can be combined with classification and facial geometry analysis techniques to infer details such as age and emotional state and even recognize individuals based on their facial features.
MRC
Machine Reading Comprehension in which an AI system not only reads the text characters but can interpret what the text is about
Supervised ML models
Models for which the training data includes known label values. Classification (predicting categories or classes) Regression (predicting numeric values) Time series forecasting (regression with time-series element enabling you to predict numeric values at a future point in time)
OCR
Optical Character Recognition in which a model can be trained to recognize individual shapes such as letters, numerals, punctuation or other elements of text Use computer vision or cognitive services resource.
Read API
Optimized for images that have a lot of text or have considerable visual noise. Better option for scanned documents. WOrks asynchronously so to not block your application while it is reading the content and returning results to your application. 3 step process 1. submit an image and retreive an operation ID 2. use operation ID to check on the status of the operation 3. retrieve results of operation Returns pages, lines words
Accountability
People should be accountable for AI systems. Deisgners and developers of AI based solution should work within a framework of governance and organizatinoal principles that ensure the solution meets ethical and legal standards that are clearly defined. Principles on how they develop sell and advocate for facial recognition.
Speech-to-Text API
Perform real time or batch transcription of audio into a text format can be from a microphone or an audio file. Real time transcription - allows you to transcribe text in audio streams. Use for presentation demos or other scenario where a person is speaking. Batch transcription - audio recordings stored on a file share or Azure stroage.
Form Recognizer
Provides intelligent form processing capabilities that you can use to automate the processsing of data in documents such as forms, invoices and receipts. Combines OCR with predictive models that can interpret form data by matching field names to values, processing tables of data, identifying specific types of field. Can process through prebuilt receipt models or custom models. Use the form recognizer resource to process these things with its key and endpoint
Qna Maker and Azure Bot Service
QnA Maker - enables you to create and publish a knowledge base with built in NLP capabilities. Azure Bot Service - framework for developing, publishing, and managing bots on Azure QnA Maker portal to create, train, publish, and manage knowledge bases. Can generate question answer pairs from FAQ documentation, import from a pre defined chit chat data source, entered and edited manually. When you're satisfied with your trained knowledge base you can publish it so that client applications can use it over its REST interface with ID, endpoint, and authorization key. Can create a custom bot using MSFT Bot Framework SDK to write code or you can use automatic bot creation functionality of QnA Maker and publish it as an Azure Bot Service application. You can connect bot with multiple channels.
Conversational AI Services in Microsoft Azure
Qna Maker- Enables you to quickly build a knowledge base of questions and answers that can form the basis of a dialog between a human and an AI agent. Azure Bot Service - Provides a platform for creating, publishing, and managing bots. Developers can use the Bot Framework to create a bot and manage it with Azure Bot Service - integrating back-end services like QNA maker and LUIS and connecting to channels for web chat, email, etc.
Object Detection
Returns bounding box, probability, and class
Data Transformations for preparing data
Select columns in Dataset - used to select columns in which you are working with. Can use to exclude certain columns. Clean missing data - used to remove rows or edit rows that contain missing data in the dataset. Normalize Data - used to normalize data so that numerical values are all on the same scale. Usually use minMax setting Split Data - splits data into 2 datasets one used to train the model and another used for testing
Predicted vs True Chart
Shows a diagonal trend in which the predicted value correlates closely to the true value. A dotted line shows how a model should perform and the closer the line for your model's average predicted value is to this the better its performance.
Global Importance Chart
Shows how much each feature in the dataset influences the label prediction
Image Classification Details
Solution based on deep learning techniques that make use of convolutional neural networks to uncover patterns in the pixels. Consists of two main tasks. Using existing images to train the model and then publishing the model. Custom Vision can be a training, prediction, or both resource if you choose both 2 resources are created Cognitive Service can be used for training, prediction, or both Custom vision resource needs project ID, model name, prediction endpoint and primary key
Recognize and synthesize speech
Speech recognition - ability to identify and interpret spoken input. converting speech into data can be in the form of a recorded voice or live audio from a microphone. Uses acoustic models to convert audio signal into phonemes. Uses a language model to convert phonemes to words. Speech synthesis - ability to generate spoken output. Usually tokenizes the text to break it down into individual words and assigns phonetic sounds to each words. Offers both speech recognition and speech synthesis capabilities through the Speech cognitive service.
Face
Supports face detection, face verification, find similar faces, group faces based on similarities, identify people Need key and endoint
Optical character recognition (OCR
Technique used to detect and read text in images. For example reading text in photos or extracting information from scanned documents.
Cross Validation
Test the trained model with data it wasn't trained with and compare the predicted value with the actual known value. Difference between predicted and actual value is know as the residuals indicates the amount of error in the model. Performance metric is calculated by squaring the errors across all test cases, finding the mean of these squares, and then taking the square root.
Natural Language Processing Services in Microsoft Azure
Text Analytics - Use this service to analyze text documents and extract key phrases, detect entities, and evaluate sentiment Translator Text - use this service to translate text between more than 60 languages Speech - use this service to recognize and synthesize speech and to translate spoken languages Language Understanding Intelligent Service - use this service to train a language model that can understand spoken or text based commands
Natural language processing
The capability for a computer to interpret written or spoken language, and respond in kind.
Conversational AI
The capability of a software "agent" to participate in a conversation. Most commonly AI solution use bots to manage dialog with users.
Computer vision
The capability of software to interpret the world visually through cameras, video, and images. Most solutions are based on ML models that can be applied to visual input from cameras, videos, or images.
Anomaly Detection
The capability to automatically detect errors or unusual activity in a system. Analyzes data over time so that it can learn various fluctuations and can identify unusual patterns.
Machine Learning
This is often the foundation for an AI system, and is the way we "teach" a computer model to make prediction and draw conclusions from data.
Classification model specifics
Two-Class Logistic Regression - used when there are 2 possible classes a label can have. Evaluating a Classification Model- In outputs and logs tab you will be given a confusion matrix which shows false positives, positives, negatives, false negatives. Accuracy - Ratio of correct predictions to the total number of predictions Precision- fraction of positive cases correctly identified(true positives/true positives+false positives) Recall - fraction of cases classified as positive that are actually positive (true positives/true positives + false negatives) F1 score - overall metric measuring recall and precision Threshold slider - adjust what is classified as what ROC curve - plots true positive rate or recall against false positive rate. Plotting these metrics for every threshold value results in a curve. In an ideal model the curve would go up and left having an area of 1. If the AUC is higher than .5 it is higher than a random guess.
Speech Translation API
Used to translate speech in one language to text or speech in another. Source language needs to be specified using language and culture code. Target language only need the language code.
Language Understanding Intelligent Service
Used to understand what is being said. Need to take into account utterances, entities, and intents. Utterance is an example of something a user might say. Entities is an item to which an utterance refers to. Intents represents the purpose or goal expressed in an utterance. Use None intent to help handle utterances that do not map to any utterances you have already entered. Consists of two main tasks. Authoring the model to define entities, intents and utterances with which to train the language model. Then publishing the model so that client applications can use it for intent and entity prediction. Use Language Understanding resource which can either be an authoring or a prediction resource. IF you choose both 2 resources are created. Cognitive Services resource only for predictionl
Designer
Visual interface that you can use to train, test, and deploy ML models without writing any code
OCR API
quick extraction of small amounts of text in images. operates synchronously to provide immediate results and can recognize text in numerous languages. Returns regions in the image that contain text, lines of text in each region, words in each line of text, also returns bounding box coordinates
Translator Text service
uses a neural machine translation model for translation which analyzes semantic context of tex t and renders a more accurate translation. Need to specify what languages from and what languages to using ISO standard codes. You can specify multiple to languages. Translator Text API offers profanity filtering by marking as profane or ommitting it in results. Selective translation so you can tag something you don't need translated.