System Design

data replication disadvantage

- database replicating each other takes some time (weak consistency) OR - takes more time to synchronize between multiple databases (strong consistency)

load balancer patterns

- random - round robin (in order) - least busy - stick session/cookies

Representational State Transfer (REST)

-Architectural style with client/server model where client acts on a set of resources managed by the server - Server provides object of resource and actions that can either manipulate or get a new representation of the resource -All communication must be stateless and cacheable -4 qualities of RESTful interface: 1. Identity resources (URI) - use the same URI regardless of any operation 2. Change with representation (verb) - use verb, header, body 3. Self-descriptive error msg (status response) - use status code, don't reinvent the wheel 4. HATEOAS (HTML interface) - your web service should be fully accessible in browser

Pull Content Delivery Network

1. CDN will get the content from the server when the first user requests the content. Following requests will get content from the CDN 2. Includes Time-to-live (TTL), which determines how long content is cached/stored on CDN. 3. Pull CDN minimizes storage on the CDN, and when content TTL expires, the next request will get data from the server.

How does DNS work

1. Client opens browser and enters "www.google.com" 2. Check browser cache and if it has it, go to 7, else continue 3. Check OS cache (Windows/Mac/Linux) if it has the IP address 4. Internet service provider (ISP) checks its cache, and if it doesn't have it, it checks the DNS server 5. The DNS server provides the IP address and sends it back to the ISP 6. ISP caches the IP address and sends the IP address back to the user 7. Browser now makes a request to that IP address and connects

4 Main Steps for System Design

1. Gathering information (use case + constraints / requirements) 2. Sketch main components (draw boxes and arrows) 3. More details about each component (specific like DB type, API) 4. Scaling: find bottlenecks (slowest part of system) and make it better.

Key-value store

A dictionary data structure, where we have O(1) reads and writes. It will store data in memory or SSD for high performance (fast latency). Often used for simple data models or rapidly changing data, such as an in-memory cache layer Example: AWS DynamoDb, memcached

BASE

Basically Available - system is always available Soft State Eventual Consistency - data will change over time and eventually be consistent

CDN caching

CDN is a type of cache since it caches content

NoSQL

Collection of items (data) in a key-value store, document store, wide column store, or graph database. Data is denormalized and joins are generally done in the application code. Most noSQL doesn't support ACID (Atomicity, Consistency, Isolation, Durability from SQL characteristics) transaction and eventual consitency. Example: leaderboard scoring data Temporary data) shopping cart Frequently accessed meta/lookup tables

Denormalization

Combining normalized tables based on how they are used to avoid expensive joins. Once data becomes distributed with sharding and federation, managing table joins across data centers will be complex. To avoid this, combine tables together as needed. Pros: faster read performance Cons: many redundant writes (redundant copies of data are written in multiple tables to avoid expensive joins).

refresh-ahead caching

Configure the cache to automatically refresh any recently accessed data before it will expire. Lower latency compared to read-through if the cache can accurately predict which items are likely to be needed in the future. Disadvantage: not accurately predicting which items are likely to be needed in the future can lead to reduced performance

UDP (User Datagram Protocol)

Connection less protocol. Datagrams (similar to packet) are guarantee only at the datagram level. Datagrams might reach their destination out of order, or not reach it at all. - Doesn't support congestion control - Can broadcast, sending datagrams to all devices on the subnet. It's useful with Dynamic Host Configuration Protocol (DHCP) because the client has not yet received the IP address, thus preventing a way for TCP to stream w/o the IP address. - Less reliable but works well in real time use cases (VoIP, video chat, streaming, realtime multiplayer game) -Use this protocol over TCP when user needs the lowest latency, late data is worse than loss of data, and want to implement your own error correction - This is in OSI Layer 4 (network layer)

TCP (Transmission Control protocol)

Connection protocol over IP address (network). It's established and terminated using a 3-way handshake (SYN, SYNACK, ACK). All packets sent are guaranteed to reach the destination in the original order without corruption. If the sender didn't get the right response, TCP will resend the packet. If there's multiple timeout, the connection is dropped. Implements flow control (data communication) and congestion control (If there's network layer issue, then packet delay). This guarantee can cause delays and generally TCP is less efficient (slower) than UDP. -To ensure high throughput (# of actions / sec), web servers keep large number of TCP connections open, resulting in high memory usage. Having many TCP connections open between web server can be expensive. -Connection pooling helps to switch to UDP, if applicable. -This protocol is useful for apps that need high reliability, but are less time critical. Ex. Web server, DB info, SMTP (email), FTP (file transfer), and SSH -Use this protocol over UDP when user needs all data to arrive safely and wants to automatically make a best estimate use of the network throughput -This is in OSI layer 4 (network layer)

database caching

DB has some level of caching with a default configuration that is good for general usage but can be adjusted for specific patterns for better performance

Wide column store

Dict within a dict within a dict (nested maps). It offers high availability and high scalability. Used for very large data sets. Example: Apache Cassandra, Azure Tables, Google BigTable

Sharding

Distribute data across different database such that each database only manages a subset of the data. Pro: Less read/write traffic, less replication, and more cache hits. Index size will also be reduced, which means faster queries. Cons: Data distribution can be uneven, need to update app logic which can mean complex sql queries.

Azure Service Fabric

Distributed system platform that makes it easy to package, deploy, and manage scalable and reliable microservices that can be stateful or stateless.

OSI Session, Presentation, Application Layer (5, 6, 7)

In reality, session, presentation, and application are usually just part of your overall program. Session is about creating a session on the port. Presentation is about encrypt / decrypt / compress data. End user layer

Document store

Key-value store with documents (JSON/XML) stored as values. Document store provides APIs or query language. It organizes based on collections, tags, metadata, directories. It provides high flexibility. Example: MongoDb, AWS DocumentDB

Load balancer vs reverse proxy

Load balancer is useful for multiple servers. It spreads traffic to a set of servers that service the same function. Reverse proxy can be used even with only one web/application servers. It can spread the traffic out to various types of resources with different function, like DB and other services.

HTTP (hypertext transfer protocol)

Method for encoding and transporting data between client and server. It is a request/response protocol, so clients issue requests and server issues a response with relevant content and completion status info about the request. HTTP, itself contained, allowing request/response to be sent to routers, servers that perform load balancing, caching, encryption, and compression. Consists of verb (method) and resource (endpoint) GET, POST, PUT, PATCH, DELETE It is an application layer protocol relying on lower-level (TCP) protocols.

message queues

Receive, hold, deliver messages. If the operation is too slow to perform real-time, you can use message queue: 1. app publishes job message to queue 2. app notifies the user of job status 3. worker picks up the job from the queue, processes it, and then signals that the job is complete - User is not blocked and the job processes in the background. During this time, the client can do small amount of processing to make it look like the task is done. Example: if posting a tweet, tweet could be instantly posted to your timeline (processed by client), but it could take time before your tweet is actually delivered to all of your followers. Examples: Redis, RabbitMQ, Amazon SQS Disadvantage: Bad for inexpensive calculation (doesnt take long) and realtime workflows (chat messages, output need to show up quickly, not wait in queue) are better for synchronous operations. Queues add delays and complexity.

Task queue

Receives tasks and their related data, runs them, then delivers their results. Supports scheduling and run computationally-intensive jobs in the background.

CAP Theorem

Since internet networks are not reliable (cannot guarantee internet is working), we need partition tolerance. We will trade-off between consistency and availability (CP vs AP)

Microservices

Software architecture where each service is responsible for one thing.

Federation

Splitting up the database by *function*. This results in less read/write traffic to each database, so less replication lag (delay). Each DB has an internal cache, so if the database is federated, then we will have more room for cache also. Ex. Database -> Forums DB, Users DB, Products DB. Each will have its own internal (inside) cache. There is no central master database that controls the 3 federated databases, so writes can be in parallel. Each individual database can have a master-slave setup with replicas though.

Service discovery

System that helps to manage many services by keeping track of registered names, addresses, and ports

Content Delivery Network (CDN)

a globally distributed network of proxy servers (servers in the middle) that serve content from locations *closer to the user*. It stores static files (html, css, JS, photos, videos). App/web servers do not have to serve request that the CDN handles

load balancer

a system that distributes incoming client requests to computing resources (web server or DB) - it prevents requests from going to unhealthy servers, overloading (give too much work to) resources, and removes single point of failure (if one server dies, then system can keep working) - it can also do SSL (Secure Socket Layer) termination: LB will do both encrypt server responses and decrypt client requests - session persistence: client's cookie will tell the LB which server to send the request to because that server may hold the user's data (for a stateful application) Client<->LB: encrypted LB<->Server: decrypted

throughput

action per time example: 10 requests completed in 1 second

write-behind caching

add/update data in the cache, asynchronously write data to the data store, improving write performance (compared to write-through). Disadvantage: There could be data loss if the cache goes down before the data is stored in the database. It's more complex to implement than cache-aside/write-through.

Eventual consistency

after a write, reads will eventually (within a few millisecond) see it example: Domain name system - when you update a website to point to a new IP address, it might not update immediately, but will eventually Email - when you send email, other people will see it eventually (few second later)

strong consistency

after a write, reads will see it right away (synchronous). example: bank transaction - if you store or take out money from bank, it should show that right away

stateful application

application / service that stores data and uses data from the current server to handle requests example: memcached or shopping application that stores cart on the server Usually don't do stateful because it's hard to scale (can't do horizontal scaling) and easy to lose data.

cache-aside caching

application is responsible for read/write from storage. The cache doesn't interact with storage directly. The app does 1). look for record/entry in cache, resulting in a cache miss 2.)load/get data/entry from the DB 3.) add entry to cache 4.) return entry. It is referred to as lazy loading (when cache doesn't have that object, get from DB and store to cache). Disadvantage: -each cache miss (data not in cache) results in three trips, causing noticeable delay -data could be old if it's updated in the DB. This issue is mitigated by setting TTL which forces an update of the cache data. When a node (computer holding cache) fails, it's replaced by a new, empty node, increasing latency.

SQL tuning

benchmark and profile to simulate performance and find bottlenecks benchmark: find the SQL problem and get the throughput Profile: find what part will be slow and help track the performance issue

master-master replication

both masters read and write and coordinate with each other on writes (they will copy the writes to each other so data is consistent). If either one of the master goes down, then the system can still handle all of the traffic.

client caching

caching on the client side, such as application, browser, or operating system

OSI Network Layer (3)

controls the operations of the subnet, deciding which physical path the data takes. Data packets are used here with IP addresses. Routers look at the IP addresses to control the data's path.

Application layer load balancing (Layer 7)

distributes requests by looking at the header (ex: HTTP GET/POST API calls), message, cookies. It finishes network traffic, reads the message, selects a server to handle the request, then opens the connection to the selected server.

Transport layer load balancer (Layer 4)

distributes requests by looking at the source IP address (from), destination (to) IP address, and ports in the headers, but doesn't look at the actual contents of the packet example: when delivering mail, only look at address, but doesn't look at the letter

OSI Transport Layer (4)

ensures that messages are delivered error-free, in-sequence, and with no losses or duplications. This is where TCP and UDP are used. TCP gives flow control.

Consistency

every read gets the most recent write OR an error

availability

every request gets a response but not guaranteed to be the most recent data

When to use NoSQL

for semi-structured data, dynamic/flexible schema, non-relational data, no need for complex joins, store many (TB) data, very data intensive workload, very high throughput for IOPS (input/output per second) = fast

When to use SQL

for structured data, strict schema (name of column & type), relational data, need for complex joins, transactions needed, clear pattern for scaling (read replicas, master-slave, federated, sharding), more established, lookups by index are very fast

Graph Database

high performance and optimized to represent complex relationships with many foreign keys or many-many relationships. example: social network. Many graphs can only be accessed with REST APIs.

Back pressure

if queue grows too much, the size of the queue could be bigger than memory size, resulting in cache misses, disk reads, and even slower performance. Back pressure limits the queue size, maintaining a high throughput rate, and good response times for jobs in the queue. Once the queue is full, clients get a server busy or HTTP 503 status code to try again later, and clients can retry later

cache

improve page load times, can reduce the load on server and database. Dispatcher will first lookup if the request has been made before and try to find the previous result in the cache, in order to save on the actual execution. Putting a cache in front of a DB can help absorb uneven loads/spikes (sudden increase) in traffic.

application caching

in-memory cache (memcached, redis) are key-value stores between the app and database. Since the data is held in RAM (memory) on the cache, it is faster than a typical database. There is less RAM than disk, so we need cache invalidation algorithms (LRU: least recently used) to help invalidate (remove) cold (not used) entries and keep hot (recently used) data in RAM. ex: user session (customer info), fully rendered web pages, activity streams, user graph data

vertical scaling

increasing the processing power of individual components (getting a special or more expensive hardware) This is more likely a "stateful service"

Weak consistency

low chance to get the most recent data example: livestream, voice over IP (VoIP, ex. skype), video chat, and video game

master-slave replication

master replicates data to slaves, then slaves copy to other slaves. When master is offline, the system will be read-only (no writes) until one of the slaves promotes to master, or a new master DB node is created.

caching disadvantage

need to maintain consistency between cache and source of truth (like database) through cache invalidation

active-passive failover

passive server will check active server's heartbeat (health condition). When the heartbeat is interrupted (has problem), then the passive will take over the active server's IP address and resume service also known as "master-slave"

OSI Physical Layer (1)

physical structures like cables, hubs, etc for sending and receiving unstructured raw bit stream

Remote procedure call (RPC)

protocol that one program can use to request a service from a program located in another computer on a network. This uses client-server model, and it is synchronous operations meaning the requesting program will wait until results from remote procedure are returned.

OSI Data Link Layer (2)

provides error-free transfer of data frames from one node to another over the physical layer

asynchronism

reduces request times for expensive operations that would be performed at runtime. It helps by doing time-consuming work in advance, like periodic aggregation of data (for a leaderboard once an hour update or once an hour update of permissions).

web server caching

reverse proxy and caches can serve static/dynamic content directly. It can also cache requests, and return responses without having to contact the app server.

Push Content Delivery Network

server sends data to CDN when it's new or changed. This minimizes traffic (CDN will handle all content requests and server won't handle any), but it maximizes storage (server will send all content to CDN, and CDN will store all of it, even if it's never used).

reverse proxy (gateway)

server that combines internal services and provides APIs to the public internet (the internal services are hidden in our private cloud network) Benefits: - increased security (hide info of backend servers, blacklist IPs, limit number of connections per client) - increased scalability and flexibility (client only sees the reverse proxy IP address, so you can scale/change backend servers or change their configuration without making a client side change) - SSL termination: decrypt incoming requests from client and encrypt server responses - compression: make smaller server responses/content (shrink an image) - caching: return the responses for cached requests

partition tolerance

system can keep working even if some of the system (servers) go down from a network issue (internet is slow or broken) partition means splitting tolerance means can handle

Domain Name System (DNS)

system for converting alphabetic names (www.google.com) into numeric IP addresses (167.183.17.5)

Scalability problem is...

system is fast for one user, but slow under heavy load (many requests)

Performance problem is...

system is slow for one user

write-through cache

the app uses the cache as the main data store for reads and writes. The app adds/updates data to the cache, cache synchronously writes the entry to the data store, return data. Write is slow, but read is faster. When a new cache is created due to failure or scaling, the new node will not cache entry until the entry is updated in the database. Cache-aside combined with write-through can mitigate (fix) this issue.

latency

time per action example: 3 seconds to complete 1 request

active-active failover

two active servers both handling traffic and splitting the load. if one server goes down, the other will continue to handle requests. also known as master-master

horizontal scaling

using more commodity (normal) machines to split the load/traffic among them - services should be stateless, which means they don't contain any user-related data like session. Session can be stored in a centralized data store, like database or cache.

Caching at database query level

when you query the database, hash the query as a key, and store the result to the cache