Tools Final

Gene Kim, The Phoenix Project

"Improving daily work is more important than doing daily work."

collecting metrics

- Pulls from HTTP endpoints 1) _targets must___________ exposes a /metrics endpoint 2) Data available and /metrics endpoint must be in the format that Prometheus understands. ___Exporter_____________________ - used by services that do not have native Prometheus endpoints Script or service that fetches metrics from your target and converts them into a format Prometheus understands Exposes these converted data in its own /metrics endpoint where Prometheus can spread them.

playbook

1 or more plays

traits: system facts

Ansible collects a series of detailed information about the managed nodes •Network interfaces •Operating system •Provides this information as global variables called _system facts_____________ •System Facts: •__used within playbooks/scripts_ to make your automation more versatile and adaptive

conatiner orchestration

Automates the deployment, management, scaling, and networking of containers

monitoring tool (prom wks)

Created to monitor ___highly dynamic container environment_____________________________ - Kubernetes, Docker Swarm mode, etc. Can be used in __a traditional non-container infrastructure____________________________________ with bare servers with apps deployed directly

containers

First, create a Manifest: describes the Container Ex. "Docker World" or "Cloud Foundry"

importance of prometheus

Modern DevOps: more and more complex for manual handling...need automation Typical: multiple servers that run containerized applications with hundreds of different processes running on that infrastructure. High degree of _interconnections_______________________ Challenge: Maintain such _set up to run smoothly_________ and without application downtimes Imagine tens or hundreds of widely distributed servers at numerous locations No insight of what is happening on the hardware level or on application level Ex. Errors, response latency, hardware down or overloaded, resouces exhausted __when services start to crash and you have so many moving pieces, that's_______ when application becomes unavailable

why is Ansible is agent-less:

Normally, have to install agent on each target server With Ansible just install on __one machine__________________________ and it manages machines remotely No need to _have any deployment effort________________________ in beginning When new version of Ansible, no upgrade effort

What if need to use a third-party service such as a Python app?

Not realistic if using a VM Containers - ...just ___deploy one of the resources ___ and combine with the Python app

wat happens w caling out to two additional copies

Potential to ___have troubles when pushing to production

First, how does Prometheus trigger the alerts that are defined by rules and who receives them.?

Prometheus has a component called _Alert Manager________________________. It is responsible for firing alerts via different channels__ such as email, select channel, etc. or some other notification client. and data storage

push metrics

Prometheus offers push gateway components so that these services can be _____push their metrics directly to Prometheus database_____________________________________.

what is used for handling complex apps

Requires _no need for learning____ in a specific sequence grouped together Need to use Playbooks Sequential modules are grouped into tasks Tasks have: names, arguments, descriptions

hypervisor?

Software that creates and manages virtual machines virtual machine

Continuous Monitoring

__Automated process__ that allows DevOps teams to ___detect compliance and security threats__ in their software development lifecycle and infrastructure

4 Ways Ansible makes life easier

a. Execute tasks from __your own machine__________________ b. Configuration and Installation with ___steps in a single YAML file___________________ c. Re-use same file multiple times and in different environments d. More reliable and less errors

contianers provide

a. Portability b. _scalability______ c. Agile DevOps

why use ansible?

a. Update __docker version_________________ b. Repetitive ____tasks______________

containers include the app as well as:

bins and libraries

host attribute

indicates on which server the task should be executed

YAML

no specific language to learn script that Ansible uses

Metric

specific unit of monitoring - Saved to Prometheus database component 1. Help attribute: basically description - just describe what the metrics is about 2. Type attribute: a) How many ___times something happens (number of exceptions) ________________ b) Counter type metric that can go both up and down Duration required to complete a request

kube scheudler

watches for newly created pods with no assigned node

comprehensive

yes

multiple choice, 100 qs

yes

what is containerization?

•*The packaging of software code with __just the operating system (OS) libraries and dependencies__ required to run the code to create a single lightweight executable (want them lightweight- for efficiency) •*__Runs consistently on any infrastructure (don't have to unload put anywhere and stays running)__ •More portable and resource-efficient than virtual machines (VMs) •Has become the de facto compute units of modern cloud-native applications. •Allows developers to __create and deploy applications faster and more securely__ •With traditional methods, code is developed in a specific computing environment which, when transferred to a new location, often results in bugs and errors

contaierization history / today

•61% of container adopters reported __using containers in half or more of the new applications__ they built during the previous two years •64% of adopters expected 50% or more of their existing applications to be put into containers during the next two years

Ansible provides...

•A consistent, reliable, and secure way to __manage the environment________

core components: dockerfile

•A script that consists of a set of instructions on how to build a Docker image •These instructions include: •Specifying the __Operating systems____ •Languages •Environment variables •_file locations_______________________ •Network ports •An image has _multiple layers___ •Once you run a Docker image to create a container, __a new read-write layer is added •This is sometimes referred to as the container layer

containers popularity

•Agile application creation and deployment •Increased ease and efficiency of container image creation compared to VM image use •__continuous developement_______, integration, and deployment •Potential for reliable and frequent container image build and deployment •Dev and Ops separation of concerns •Create application container images at build/release time instead of at deployment •Results in __decoupleing apps from infrastructure______________.

Depl. history #2 virtualized deployment era

•Allows applications to be isolated between VMs •___provides level of security____________ as the information of one application cannot be freely accessed by another application •Allows better utilization of resources in a physical server •Allows __better scalability__________ because an application can be added or updated easily •Reduces hardware costs

docker compose

•Allows you to start _creating and running containers___________________ on Windows and Mac within minutes • is a simple way of installing and setting up the entire Docker development environment •Includes: a.Docker Engine b.Docker Compose c.Docker CLI client d.Docker Content Trust e.Kubernetes f.Credential Helper •Used for building and sharing containerized applications and microservices in multiple languages and frameworks on any cloud platform

traits: templating system

•Ansible uses the Jinja2 Python templating system •Allows for dynamic expressions and access to variables •Templates can be _used to facilitate setting up configuration files and services.__ •Ex. •Use a template to set up a new virtual host within Apache •Then reuse the same template for multiple server installations

containerization continued

•Bundles the application code together with the related configuration files, libraries, and dependencies required for it to run •__Container is ABSTRACTED away from the host operating system •It stands alone and becomes portable__—able to run across any platform or cloud, free of issues.

container deployment era

•Containers - similar to VMs •Provide relaxed isolation properties •Share the Operating System (OS) among the applications •Considered lightweight •Similar to a VM •Container has its own: •Filesystem •Share of CPU •Memory, etc. •_______decoupled from the underlying infrastructure__ •Therefore, they are portable across clouds and OS distributions.

A.C. benefits continued

•Ease of management •Automates the installation, scaling, and management of containerized workloads and services •__Container orchestration platforms can ease management taxks such as a.Scaling containerized apps b.Rolling out new versions of apps c.Providing monitoring, logging and debugging •__Kubernetes - most popular container orchestration system available •Open source technology that automates Linux container functions originally •Works with any container system that conforms to the Open Container Initiative (OCI) standards for container image formats and runtimes.

A.C. benefits continued

•Fault isolation •Each containerized application is isolated and operates independently of others •Failure of one container does not affect the continued operation of any other containers •Efficiency •Software running in containerized environments shares the machine's OS kernel •__Application layers withing a container can be shared across containers •__Containers are inherently smaller in capacity than a VM and require less start-up time

concepts: handlers

•Handlers •Used to _perform actions on a service_ •Ex. Restarting or stopping a service that is actively running on the managed node's system

continuous monitoring best practices (scope your implementation)

•How you implement continuous monitoring will __depend on business focus, functions, and goals •Organizations in different industries will need to monitor different aspects of their infrastructure •__Limit your implementation to your critical business processes__, particularly areas involving sensitive or proprietary information •Conduct a __security risk analysis___ to assess and prioritize your risks to determine which processes should be monitored

continued: docker images

•Images are templates used for __building conatiners__________________________ •Docker images are: a.Immutable b.__Read-only files that consist of the source code c.__Libraries d.__Dependencies e.Tools •Each image is created from a Dockerfile which contains specific instructions for _building a particular Docker Image________________________________ •Useful for _increasing reusability_______________________ and essentially decreasing disk use •Since you want to keep your containers lightweight and fast, it is __vital to maintain small images__ which can be done by: •Utilizing a lighter image base •Avoiding unnecessary layers

concepts: inventories

•Inventory file(playbook and inventory) •Contains a list of the hosts to be managed using Ansible • •Static inventories •Usually created as .ini files

Anisbles traits (what is Idempotent Behavior)

•Keeps track of the state of resources in managed systems in order to avoid repeating tasks that were executed before •If a package was already installed, it won't try to install it again •Objective: •After each provisioning execution the ____system reaches or keeps the desired__ state, even if you run it multiple times •This is known as ___idempotent behavior •Regardless of how many times the procedure is executed, the result will be the same •When running a script, it displays the status of each task being executed and whether or not the task performed a change in the system.

docker containers

•Lightweight virtualized runtime environments for running applications •Each container _represents a package_____________ of software that contains a.Code b.System tools c.Runtime d.Libraries e.Dependencies f.Configuration files required for running a specific application •These are __independent and isolated from the host and other instances running on the host. •Containers are quick and ___easy to start up, configure, add, stop, and remove__ •Developers can work on the same application in different environments knowing this will not affect its performance •Additionally, they can __share data between containers__________ using data volumes.

components of main Prom server and alter manager

•Main Prometheus server__ •Scrapes and stores time series data •(Nodes are scraped) •Client libraries •For instrumenting application code •Push gateway (not-for long term thigns) •Supports short-lived jobs •Altertmanager__ Handles all alerts

What is ansible

•Modern configuration management tool • ___Facilitates setting up and maintaining _________remote servers •Provides a minimalist design so users are up and running quickly •Users write Ansible provisioning scripts in YAML •YAML •User-friendly data serialization standard •Not tied to any particular programming language •Enables users to create sophisticated provisioning scripts more intuitively •Doesn't require any special software to be installed on the nodes that will be managed •A ___control machine___________________ is set up with the Ansible software which then communicates with the nodes via standard SSH •Functions as a __configuration management tool________________ •____Encapsulates all of the common features___ present in other tools of the same category •Maintains a strong focus on simplicity and performance

features

•Multi-dimensional data model with __time series data identified by metric name and key/value pairs •__PromQL: a flexible query language to leverage this dimensionality •Single server nodes are autonomous •Time series __collection happens via a pull model over HTTP •Targets are __discovered via service discovery or static configuration

metrics

•Numeric measurements •Time series •Means that changes are __recorded over time__ •What users want to measure differs from application to application •For a web server it might be request times •For a database it might be number of active connections or number of active queries, etc.

what is promethues

•Open-source systems monitoring and alerting toolkit •A standalone open source project •Maintained independently of any company •Collects and ___stores its metrics as time series data__ •Metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called Labels.

concepts: playbook

•Playbook contains •An __ordered list of tasks_______ •A few other directives to indicate which hosts are the target of that automation

A.C. benefits

•Portability •A container creates an executable package of software that is abstracted away from (not tied to or dependent upon) the host operating system •Able to run uniformly and consistently across any platform or cloud •Agility •The ___open source Docker Engine for running containers started the industry standard for containers with a.Simple developer tools b.____Universal packaging approach that works on both Linux and Windows operating systems •The container ecosystem has shifted to engines managed by the ___Open Container Initiative (OCI). •Speed •Containers are often referred to as "lightweight" because __they share the machine's operating system (OS) kernel and are not bogged down with this extra overhead

basic architecture of prom server

•Prometheus Server ( the server which scrapes and stores the metrics data ) •___Client Library - ___Calculates and exposes the metrics (here are the metrics what do you want to do with them) •Alert manager - Raises alerts based on preset rules •Pushgateway__ - pushes metrics into Prometheus •Service Discovery - finds new metrics running inside supported environments •Grafana - queries data in Prometheus and displays it (data displayed graphically) •AlertManager__ - notifies admins •Prometheus scrapes metrics from instrumented jobs •Stores all scraped samples locally •Then __runs rules over this data__ to either aggregate and record new time series from existing data or generate alerts Grafana visualizes the collected data

purpose of Kubernetes

•Purpose: __manages containerized workloads__________________ and services

iterate the CM Processes

•Require __feedback and assessment are essential at every step. •Gathered information must be assessed on an ongoing basis so that operations analysts and developers can measure and track security, operations, and business-related issues •Accurate and actionable feedback enables DevOps teams to __produce products and services in accelerated development cycles •Information gathered from the assessment process can also benefit business and IT decision-makers as they choose how to invest resources as the business grows

kube controller manager

•Runs controller processes

benefits on container orchestration

•Security Isolation of applications as __containers prevent the invasion of malicious code from affecting other containers or the host system

concepts: tasks

•Task •An individual unit of _work to execute on a managed node___ •Each action to perform is defined as a task •Can be executed as a one-off action via ad-hoc commands... Or included in a playbook as part of an automation script.

when to use promethues

•Works well for __recording any purely numeric time series__ •Designed for reliability •The system you go to during an outage to allow you to quickly diagnose problems •Each __Prometheus server is standalone__, not depending on network storage or other remote services

how does Ansible make IT Automation Accessible?

•___configurations are simple data descriptions________________________________ of your infrastructure •Ensures everyone on your team will be able to understand the meaning of each configuration task •Requires only a password or SSH key in order to start managing systems •Managing can begin without installing any agent software No more "managing the management

docker

•_open-source containerization_ platform •Used for developing, deploying, and managing applications in lightweight virtualized environments called Containers •Mainly used as a software development platform for developing distributed applications that __work efficiently in different environments •The software system is __agnostic(aren't compatibility issues/ not influenced by environment)____________ •Developers don't have to __worry about compatibility issues •Docker containers are ___smaller________________________________ than virtual machines.

(Containerization wks) vms

- pay attention to the amount of ___resources consumed___ 1. Hardware 2. _host operating system______________ 3. Hypervisor 4. Linux VM a) OS (guest OS) b) Binaries and libraries c) node JS

steps

1. Create image Ex. Docker Image or Application Container Image (ACI) 2. Container Has all the _the run times___ required to run an application 3. Host OS Then create the Runtime Engine _-something that runs those containers._ Ex. Docker Engine Next, create multiple containers - (___libraries___) Uses less resources

targets are monitored for

1. Current _CPU status_____________________ 2. Memory usage 3. _Memory/Disk Space Usage________________________ usage 4. Counts: requests, exceptions

advantages

1.Consistency •Docker ensures reliability that your _app runs the same_ across multiple environments •Developers working on different machines and operating systems can work together on the same application without environment issues 2.Automation •The platform allows you to automate _ tedious, repetitive tasks______________________ and schedule jobs without manual intervention 3.Faster deployments •Since containers virtualize the OS, there is __no boot time when starting up containers instances__ •Therefore, you can do deployments in a matter of seconds •You can share existing containers to create new applications

what happens when editing the index.php file in the docker video?

11. Editing index.php file does not result in the updated text __image____ The docker build command made a copy of the original index.php file To see the edit, first __rebuild the container____, then spin up a new container

prometheus server

3 parts 1) Time series database _stores all the metrics data_______________ such as current CPU usage or number of exceptions in application. 2) ___data retrieval worker_______________________ Responsible for getting or pulling those metrics from applications, services servers and other target resources and storing them or pushing them into the database. 3) Web server or server API Accepts queries for that stored data This component is used to __display the data ________ or UI

what size businesses are using contianers?

37% large enterprises 20% midmarket businesses

How Ansible Works

Modules do the actual work Modules are __small programs that do the actual work When modules are done, they are removed Modules are granular Module = ___one small specific task____________________________

pull mechanism

Prometheus ___pulls data from end points_______ - an important characteristic of Prometheus. When you're working with many microservices and you have each service pushing their metrics to the monitoring system, this __creates a high load of network traffic____ within your infrastructure and your monitoring can actually become a bottleneck. You have monitoring, but you pay the price of overloading your infrastructure with constant push requests from all the services. This floods the network . Plus you also have to install daemons on each of these targets to push the metrics to monitoring servers. In comparison Prometheus requires just scraping the end point. Therefore, metrics can be __moved by multiple Prometheus instances___. Another advantage of using pull Prometheus can easily detect whether services are running.

how does prometheus know what to scrape and when?

That is configured in the __prometheus.yml file__________________________________. The file identifies targets Prometheus should scrape and at what interval. Prometheus uses a service discovery mechanism to find those target endpoints when you first download and install. You will see the sample config file with some default values. Since Prometheus has its own metrics endpoint to expose its own data, it ___can monitor its own health__________________________.

Dockerfile is just a text file of steps used to

__create an image and run it_____

Continuous Security Monitoring

a)Allows security and operations analysts to use __automated processes to aggregate and analyze data__ throughout an entire system b)Provides increased visibility into all areas, ensuring that important trends, events, and security risks are detected c)Admins can ___discover and report incidents in shorter amount of time__ and immediately respond to security threats Real-time reports and alerts help response teams prevent attack attempts and minimize the impact of a breac

how does continuous monitoring catch performance issues sooner

a)Continuous monitoring can __expose performance issues early, ensuring that they never reach production b)Promotes more stable and error-free application releases c)Operations teams can manage and __monitor new application releases to ensure that they're performing as intended. d)Critical data gathered from monitoring tools __allow analysts to identify performance issues caused by low memory and network latency and implement a solution before unplanned downtime occurs

how does dev ops avoid downtime with C.M.

a)Using continuous monitoring tools, DevOps analysts can monitor the network, database, and applications for performance issues b)__Respond before downtime occurs (proactive) or customers are affected c)Aggregated statistics and reports are useful for __analyzing past issues to prevent them in the future

benefits (docker wks)

a. __same environment__________________ b. __sandbox project_____________________ c. It works!

containers require less

a. start up time b. ___resources__________ c. disk space d. _memory______________

containers provide

a.Cloud and OS __distribution portability_____________________ •Ubuntu, RHEL (RedHat), on-premises, on major public clouds...anywhere b.Application-centric management ___raises the level of abstraction___from running an OS on virtual hardware to running an application on an OS using logical (not physical) resources

container popularity...

a.Loosely coupled b.__Distributed c.Elastic d.Liberated micro-services: - Applications are broken into smaller, independent pieces-Can be _deployed and managed dynamically__________ - not a monolithic stack running on one big single-purpose machine

Kubernetes services

a.Service discovery and load balancing •Kubernetes can expose a container using the DNS name or IP address •When traffic is high, Kubernetes ___load balances__ so that the deployment is stable b.Storage orchestration •Kubernetes provides automatic mounting of a storage system (local, public cloud, etc... your choice) c.Automated rollouts and rollbacks •You can automate Kubernetes to: •__Create new containers_________________________ for your deployment •___Remove existing containers______________________________ •Adopt all their resources to the new container. run containerized tasks restart containers that fail kills containers

ansible playbooks

group of tasks to be excecuted

deployment history

had physical apps on servers •Result = __recourse allocation issues_____________ •Issue •Multiple applications running on a physical server can result in one app ___consuming majority of resources_____ •Other apps will underperform •Solution •Run each application on a different physical server •Problem •Did not scale as resources were underutilized •Expensive for organizations to maintain many physical servers

inventory list

keeps a list _of all the machines involved___ in the task executions identify using IP addresses or host names

In the past why would users take notes?

record changes they had made; so they can remember it when they are doing it again on the same server.

A.C. continued

•Abstraction from the host operating system makes containerized applications a.Portable b.___Able to run uniformly and consistently across any platform or cloud •Containers can be easily transported from a.A desktop computer to a virtual machine (VM) b.From a Linux to a Windows operating system •They will ___run consistently on virtualized infrastructures or on traditional "bare metal" servers, either on-premises or in the cloud •__Existing applications can be re-packaged into containers that use compute resources more efficiently.

the past for continuous monitoring

•Businesses have relied on periodic manual or computer-assisted assessments to provide snapshots of the overall health of their IT environment •This method provided information that was outdated and __could result in undetected security threats___, exposing the business to liability or compliance fines •Provides support for __critical processes like threat detection

•Key features of a good continuous monitoring tool

•Consider the extent to which the system can be customized to handle growth and changing needs in your DevOps environment • •Key features of a good continuous monitoring tool a.Easy-to-use dashboard b.____Full-stack application monitoring (Includes client and server software) c.In-depth analysis d.Short learning curve (is it intuitive) e.___Real-time performance metrics f.Decision-making tools

explain application containerization

•Containers encapsulate an application as a __single executable package of software •Bundles application code together with all of the related configuration files, libraries, and dependencies required for it to run •Containerized applications are "isolated" - ___they do not bundle in a copy of the operating system •Instead, an ___open source runtime engine (such as the Docker runtime engine) is installed on the host's operating system •It becomes the conduit for containers to share an operating system with other containers on the same computing system.

configuration management and monitorying

•Continual assessment •Ensures changes to software and network configurations __don't create security gaps__ and cause noncompliance • •Monitoring tools offer built-in mechanisms for: •__Setting baseline security controls__ •Customizing security policy assessments •Automated reports___ that DevOps teams can use to review configuration changes across the organization

Ansible concepts (control node)

•Control Node •_system where Ansible is installed_ and set up to connect to your server •You can have multiple Control Nodes •Any system capable of running Ansible can be set up as a control node •Includes: personal computers or laptops running a Linux or Unix-based OS

database monitoring and •__A good database monitoring tool will provide useful metrics on:______

•Database monitoring mechanisms help operations analysts quickly __determine the exact time an issue occurred and what caused it •In development and staging environments, teams can test how specific changes might affect application performance, resource usage, or quality of service to customers •__A good database monitoring tool will provide useful metrics on:______ •SQL query performance •Session details •Deadlocks Transactions per minute

kubernetes components

•Deploying Kubernetes _results in a cluster___________________________ •Kubernetes cluster •Consists of a set of worker machines called nodes__ •Nodes run containerized applications___ •Every cluster has at least one worker node •Worker node(s) •Host the Pods that are the components of the application workload •Control plane__ •Manages the worker nodes and the Pods in the cluster

Kub History

•Developed by engineers at Google •Transitioning today into a __general purpose computing platform___________________________________________ •Becoming basic building blocks of modern cloud infrastructure and applications •Enables organizations to deliver a high-productivity __Platform-as-a-Service (PaaS) •Addresses multiple infrastructure-related and operations-related tasks in __cloud-native development •Development teams can focus solely on coding.

core components docker engine

•Installed on the host machine •Represents the core of the Docker system •Lightweight runtime system and the underlying client-server technology that creates and manages containers •Docker Engine consists of 3 components 1.Server •The Docker daemon (dockerd) is responsible for __creating and managing containers____________________ 2.Rest API •Establishes __communication between programs and Docker__ •Instructs dockerd what to do 3.Command Line Interface (CLI) •Used for running Docker commands

implementing continuous monitoring three phases of application performance monitoring

•Involves monitoring the backend of an application to track and resolve critical performance and availability issues •___The three phases__ of application performance monitoring a)Proactively monitoring an application__ to identify issues before a problem occurs b)Isolating the problem___ so it doesn't affect the entire environment c)Diagnosing the Problem__ at the code level and implementing a solution •Monitoring based on metrics such as: •Uptime •Resource use •System response User experience

concepts: managed nodes

•Managed Nodes •These are the systems you control using Ansible • •Ansible ___requires__ that managed nodes are reachable via SSH, and have Python 3 (version 3.5 or higher) installed • •Managed Nodes can be a variety of operating systems including Windows servers

containerization design

•Often referred to as "lightweight," •They share the machine's operating system kernel •Do not require the overhead of associating an operating system within each application •____Smaller in capacity than a VM and require less start-up time___, allowing far more containers to run on the same compute capacity as a single VM •Result is higher server efficiencies and reduces server and licensing costs. __•Allows applications to be "written one and run anywhere"___

docker volumes

•Rather than adding new layers to an image, you can preserve data produced by a running container by using Docker volumes •Volumes allows users to: •Save data__ •_share it_________________ between containers •Mount it to new ones

what is docker used for

•Running __multiple workloads on fewer resources__ •__Isolating and segregating applications_________________________ across development and release cycles. •Streamlining the development lifecycle and supporting CI/CD workflows •Developing __highly portable workloads__ that can run on multi-cloud platforms •Cost-effective alternative to virtual machines •Version control system for an application

A.C. continued

•This __eliminates the overhead of running an operating system within each application •Makes containers smaller in capacity and faster to start up, driving higher server performance •Isolation of applications as containers •____Reduces the chance that malicious code present in one container will impact other containers or invade the host system.

docker hub

•This is the largest cloud-based repository of container images provided by Docker •It supplies over 100,000 images available for use created by open-source projects, software vendors, and the Docker community •The platform allows you to: •_ship your applications________ anywhere quickly •Collaborate with teammates •__Automate builds___________________ for faster integration to a development pipeline •__Like GitHub, developers push and pull container images from Docker Hub and decide whether to keep them public or make them private