DevOps
Maven
A build automation tool used primarily for Java projects.
HAproxy
High Performance TCP/HTTP Load Balancer
Netflix Open Source Software
* Big Data - Tools and services to get the most out of your (big) data * Build and Delivery Tools - Taking code from desktop to the cloud * Common Runtime Services & Libraries - Runtime containers, libraries and services that power microservices * Content Encoding - Automated Scalable Multimedia Ingest and Encoding * Data Persistence - Storing and Serving data in the Cloud. * Insight, Reliability and Performance - Providing Actionable Insight at Massive Scale * Security - Defending at Scale * User Interface - Libraries to help you build rich client applications
Continuous Deployment vs. Continuous Delivery
* Continuous Deployment is to deliver every successful build to production. * Continuous Delivery is that every build can potentially be delivered to production, typically by the click of a button
Deploy vs. Release
* You can deploy code to production without releasing * Release is to make functionality available to end users
Salt
A configuration management application by SaltStack that handles remote execution of applications. Its purpose is to provide central system management. Salt provides a dynamic communication bus for infrastructures that can be used for orchestration, remote execution, and configuration management. It's written in Python.
Test-Driven Development (TDD)
A development practice in which small tests to verify the behavior of a piece of code are written before the code itself. The tests initially fail, and the aim of the developer(s) is then to add code to make them succeed. Check out Xebia Labs' testing dashboard and analytics tool here.
Consul
A distributed, highly-available, and multi-datacenter aware tool for service discovery, configuration, and orchestration. Consul enables rapid deployment, configuration, and maintenance of service-oriented architectures at massive scale.
Canary Release
A go-live strategy in which a new application version is released to a small subset of production servers and heavily monitored to determine whether it behaves as expected. If everything seems stable, the new version is rolled out to the entire production environment.
Dark Launch
A go-live strategy in which code implementing new features is released to a subset of the production environment but is not visibly, or only partially, activated. The code is exercised, however, in a production setting without users being aware of it.
Agile
A precursor to Devops; Agile is a software development and, more broadly, business methodology that emphasizes short, iterative planning and development cycles to provide better control and predictability and support changing requirements as projects evolve.
Configuration Management
A term for establishing and maintaining consistent settings and functional attributes for a system. It includes tools for system administration tasks such as IT infrastructure automation.
CloudFormation
AWS service that helps create and manage AWS resources. It also supports resource provisioning and orderly updating. CloudFormation creates a template that helps describe AWS resources and all related dependencies needed to run an application. It manages the order of AWS resource provisioning .
AWS
Amazon Web Services (AWS) is a collection of remote computing services, also called web services, that make up a cloud computing platform offered by Amazon.com. These services are based out of 11 geographical regions across the world. The most central and well-known of these services are Amazon Elastic Compute Cloud and Amazon S3.
Ansible
An orchestration engine by AnsibleWorks that provides a framework for system automation. It aids multi-node deployment, ad hoc task execution and configuration management. It requires no special software, either for deployment or operation. Ansible allows use of customized modules written in any programming language, and allows integration points for numerous plugins. For security, it uses the authentication system of the operating system. It is written in Python.
Deployment Pipeline
At an abstract level, a deployment pipeline is an automated manifestation of your process for getting software from version control into the hands of your users.
Top Netflix Open Source Projects - Insight, Reliability, and Performance
Atlas Atlas is used for managing dimensional time series data for near real-time operational insight. Atlas features in-memory data storage, allowing it to gather and report very large numbers of metrics, very quickly. It was primarily created to address issues with scale and query capability in the previous system. Ice Ice provides a birds-eye view of our large and complex cloud landscape from a usage and cost perspective. Cloud resources are dynamically provisioned by dozens of service teams within the organization and any static snapshot of resource allocation has limited value. The ability to trend usage patterns on a global scale, yet decompose them down to a region, availability zone, or service team provides incredible flexibility. Ice allows us to quantify our AWS footprint and to make educated decisions regarding reservation purchases and reallocation of resources. Simian Army The Simian Army is a suite of tools for keeping your cloud operating in top form. Simian Army consists of services (Monkeys) in the cloud for generating various kinds of failures, detecting abnormal conditions, and testing our ability to survive them. The goal is to keep cloud safe, secure, and highly available. Currently the simians include Chaos Monkey, Janitor Monkey, and Conformity Monkey. After analysis and performance related projects, let's talk about top Netflix open source projects related to security:
Docker
Automation tool by Docker, Inc., that automates application deployment inside software containers. It's a tool that helps package an application and its dependencies as a virtual container. Docker is written in Go programming language.
Unit Testing
Code-level (i.e., does not require a fully installed end-to-end system to run) testing to verify the behavior of individual pieces of code. Test-driven deployment makes extensive use of unit tests to describe and verify intended behavior.
Chef
Configuration management tool by Opscode, is an automation platform that revamps infrastructure into simple code. It helps to automate configuration, deployment and scaling of servers and applications, irrespective of whether the server or application is in the cloud, on site or a combination of both. Chef runs in two modes: Client/Server and standalone configuration. It is written in Ruby and Erlang.
Deployment Management
Deployment Management aims to plan, schedule and control the movement of releases to test and live environments.
DevOps
DevOps (development and operations) is a software development phrase used to describe a type of agile relationship between Development and IT Operations. The goal of DevOps is to improve communication, collaboration, and processes between the various roles in the software development cycle, in order to improve and speed up software delivery.
Top Netflix Open Source Projects - Data Persistence
EVCache EVCache is a memcached and spymemcached based caching solution that is mainly used for AWS EC2 infrastructure for caching frequently used data. Dynomite Inspired by Amazon's Dynamo whitepaper, Dynomite is a generic dynamo implementation for different k-v storage engines. Astyanax Astyanax is a high-level Java client for Apache Cassandra. Apache Cassandra is a highly available column oriented database. It borrows many concepts from Hector but diverges in the connection pool implementation as well as the client API. One of the main design considerations was to provide a clean abstraction between the connection pool and Cassandra API so that each may be customized and improved separately. Astyanax provides a fluent style API which guides the caller to narrow the query from key to column as well as providing queries for more complex use cases that Netflix have encountered. The operational benefits of Astyanax over Hector include lower latency, reduced latency variance, and better error handling. This was all about the data persistance related top Netflix open source projects. Let's discuss imsight, reliability, and performance tools:
Puppet
IT automation software by Puppet Labs that aids management of an infrastructure throughout its lifecycle. It provides automation and management, right from provisioning and configuration to start with until orchestration and reporting. Puppet helps automation, deployment and scaling of applications in the cloud or on site and follows the Client/Server model. It is written in Ruby.
Netflix Big Data
Genie Genie is a federated job execution engine which provides REST-ful APIs to run a variety of big data jobs like Hadoop, Pig, Hive, Presto, Sqoop, and more. It also provides APIs for managing many distributed processing cluster configurations and the commands and applications which run on them. Inviso Inviso is an interface to search and visualize Hadoop jobs, Job performance, and cluster utilization data. Lipstick Lipstick is a God-project for pig developers. It combines a graphical depiction of a Pig workflow with information about the job as it executes, giving developers insight that previously required a lot of sifting through logs (or, a Pig expert) to piece together. Aegisthus Aegisthus enables the bulk abstraction of data out of Cassandra for downstream analytic processing. It does so by implementing a reader for the SSTable format and provides a map/reduce program to create a compacted snapshot of the data contained in a column family.
git
Git is a distributed revision control system with an emphasis on speed, data integrity, and support for distributed, non-linear workflows. Git was initially designed and developed by Linus Torvalds for Linux kernel development in 2005, and has since become the most widely adopted version control system for software development.
Top Netflix Open Source Projects - Common Runtime Services & Libraries
Hystrix Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services, and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable. In a distributed environment, inevitably some of the many service dependencies will fail. Hystrix is a library that helps you control the interactions between these distributed services by adding latency tolerance and fault tolerance logic. Hystrix does this by isolating points of access between the services, stopping cascading failures across them, and providing fallback options, all of which improve your system's overall resiliency. Karyon Karyon is a framework and library that essentially contains the blueprint of what it means to implement a cloud-ready web service. All the other fine-grained web services and applications that form Netflix's SOA graph can essentially be thought as being cloned from this basic blueprint. Turbine Turbine is a tool for aggregating streams of Server-Sent Event (SSE) JSON data into a single stream. The targeted use case is metrics streams from instances in an SOA being aggregated for dashboards. Netflix uses Hystrix which has a real-time dashboard that uses Turbine to aggregate data from 100s or 1000s of machines.
Bluemix
IBM Bluemix is a cloud platform as a service (PaaS) developed by IBM. It supports several programming languages and services as well as integrated DevOps to build, run, deploy and manage applications on the cloud. Bluemix is based on Cloud Foundry open technology and runs on SoftLayer infrastructure. Bluemix supports several programming languages including Java, Node.js, Go, PHP, Python, Ruby Sinatra, Ruby on Rails and can be extended to support other languages such as Scala through the use of buildpacks.
LXC
LXC - (LinuX Container) is an operating system-level virtualization method that allows multiple isolated Linux systems to run as a single host that acts as the controller. Virtualization is not provided through Virtual machines, but rather through a virtual environment with a process set and network space. Each isolated system gets its own directory structure, network devices, IP addresses and process table. It's written in C, Python, Shell and LUA.
Nexus
Manages software "artifacts" required for development. If you develop software, your builds can download dependencies from Nexus and can publish artifacts to Nexus creating a new way to share artifacts within an organization.
Microservices
Microservices are a software architecture design pattern, in which complex applications are composed of small, independent processes communicating with each other using language-agnostic APIs. These services are small, highly decoupled and focus on doing a small task.
Nginx
NGINX is open source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. ... In addition to its HTTP server capabilities, NGINX can also function as a proxy server for email (IMAP, POP3, and SMTP) and a reverse proxy and load balancer for HTTP, TCP, and UDP servers.
Top Netflix Open Source Projects - Build and Delivery Tools
Nebula Nebula is a collection of Gradle plugins that Netflix has open sourced to share its internal build infrastructure to the public. The nebula-plugins organization was set up to facilitate the generation, governance and releasing of Gradle plugins. It is done by providing a space to host plugins, in SCM, CI, and a Repository. Asgard Asgard is a web-based tool for managing cloud-based applications and infrastructure. Asgard helps Netflix to build and deploy hundreds of applications and services to the Amazon cloud. Asgard is released under the Apache License, Version 2.0. Please feel free to fork the project and make improvements to it.
Jenkins
Open source continuous integration server. It features numerous plugins that support project building and testing. Jenkins monitors a version control system by maintaining a build system, monitoring it for changes and providing appropriate notifications of those changes. It is written in Java.
Ansible
Open-source software platform for configuring and managing computers. It combines multi-node software deployment, ad hoc task execution, and configuration management. It manages nodes over SSH or PowerShell and requires Python (2.4 or later) to be installed on them. Modules work over JSON and standard output and can be written in any programming language. The system uses YAML to express reusable descriptions of systems.
Provisioning
Provisioning is the process of preparing new systems for users (in a Continuous Delivery scenario, typically development or test teams). The systems are generally virtualized and instantiated on demand. Configuration of the machines to install operating systems, middleware, etc. is handled by automated system configuration management tools, which also verify that the desired configuration is maintained.
Redis
Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster
Release Orchestration
Release Orchestration is the use of tools like XL Release which manage software releases from the development stage to the actual software release itself.
Top Netflix Open Source Projects - Security
Security Monkey Security Monkey monitors policy changes and alerts on insecure configurations in an AWS account. It also proves to be a useful tool for tracking down potential problems as it is essentially a change tracking system. Scumblr Scumblr is a web application that allows performing periodic searches and storing / taking actions on the identified results. Scumblr searches utilize plugins called Search Providers. Each Search Provider knows how to perform a search via a certain site or API (Google, Bing, eBay, Pastebin, Twitter, etc.). Searches can be configured from within Scumblr based on the options available by the Search Provider. Message Security Layer Message Security Layer (MSL) is an extensible and flexible secure messaging framework that can be used to transport data between two or more communicating entities. Data may also be associated with specific users, and treated as confidential or non-replayable if so desired. The above list of top Netflix open source projects was prepared with inputs from Netflix OSS. Use our comment section below to share your views. Let us know if we missed out any popular open source project from Netflix in our list.
Functional•Testing
Testing of the end-to-end system to validate (new) functionality. With executable specifications, Functional Testing is carried out by running the specifications against the application.
Regression Testing
Testing of the end-to-end system to verify that existing functionality has not been negatively impacted by changes to the application.
Build Automation
Tools or frameworks that allow source code to be automatically compiled into releasable binaries. Usually includes code-level unit testing to ensure individual pieces of code behave as expected.
Application Release Automation (ARA)
Tools, scripts or products that automatically install and correctly configure a given version of an application in a target environment, ready for use.
Acceptance•Testing
Typically high-level testing of the entire system carried out to determine whether the overall quality of both new and existing features is good enough for the system to go to production.
Top Netflix Open Source Projects - UI
We created Falcor for efficient data fetching. We help maintain Restify to enable us to scale Node.js applications with full observability. We're helping to build the next version of RxJS to improve its performance and debuggability.
Juju
cloud infrastructure automation tool by Canonical. It helps configure, deploy, manage and scale applications that are on a public, private or hybrid cloud. It comes in two versions—command-line interface and Webapp. Juju is written in Python.
Vagrant
open source tool used for creating and configuring Virtual Development Environments. It enables sharing of pre-configured virtual machines among users, thereby allowing use of a common development environment. Vagrant supports projects written in PHP, Python, Java, C# and Javascript. It's written in Ruby.