Distributed Systems
(total elapsed time - downtime) / Number of Failures
How is MTBF calculated?
Maintenance Time / Number of repairs
How is MTTR calculated
scrubbing server
In a DNS system, after the routing system sends the client the IP address of the nearest proxy server, what component handles the request next
the routing system
In a DNS system, when a client first wants to send a request, what is the first component it is sent to?
for any connection, all incoming packets are forwarded to the same tier 3 load balancer
In a typical data center load balancing system, there are three tiers of load balancers. What is the role of the second tier which sits at layer 4 of OSI
less hardware infrastructure/layers are required to do load balancing, network latency will be reduced because of no intermediate hop, with a dedicated load balancing layer there can be traffic issues
In designing twitter, client side load balancing is utilized. What are the advantages of this approach?
uniqueness and scalability
In the context of a sequencer, what is the problem with using multiple databases that all start at different values and increment by the number of databases? What requirements are violated?
graph databases can be used in social applications
What are popular applications of graph databases?
Amazon DynamoDB, Redis, and memcached DB
What are some popular key-value databases
they are quite expensive, the configuration requires additional human resources, availability can be an issue as additional hardware is required in the case of failure
What are the downsides of hardware based load balancers
lack of standardization and consistency
What are the drawbacks of noSQL databases?
DNS resolver, root-level name servers, top-level domain name servers, authoritative nam servers
What are the four types of servers in the DNS hierarchy?
user requests a domain name to browser, browser sends DNS query to ISP, ISP forwards DNS query to DNS infrastructure, DNS infrastructure sends list of IP addresses to ISP, ISP forwards list to browser, browswer responds with HTTP request to ISP, ISP forwards HTTP request to web server
What are the steps in the high level flow of a dns
caching, server replication, and protocol
What are the three main things that make DNS a reliable system
MTBF and MTTR
What are the two ways of measuring system reliability
DNS uses caching at different layers and DNS name servers are in a hierarchical form
What are two aspects of domain name systems that allow them to scale well and cater to the requests of the entire internet?
load balancers are typically deployed in pairs. To maintain availability, enterprises will also use clusters of load balancers that use heartbeat communication. If the entire cluster fails, manual rerouting must occur
What happens if a load balancer fails? Are they not a single point of failure?
128-bit numbers makes the primary-key indexing slower, we cannot guarantee uniqueness, and they are not monotonically increasing
What is the problem with using a UUID for a sequencer?
multiple queues based on task categories
What is the solution to the fact that in a task scheduler, some tasks need to be processed with more urgency than others further ahead in a FIFO queue
UDP has no handshaking dialogues and therefore exposes the user to the unreliability of the underlying network
What makes UDP a more unreliable communication protocol as compared with TCP?
pull
What type of CDN strategy (push or pull) is appropriate when dynamic content is being requested?
applications that require a large volume of semi-structured or unstructured data, low latency, and flexible data models
What type of applications require noSQL databases?
HTTP responses
What type of content do web servers serve?
in the browser, the operating system, and the local name server. Some frequently visited services are also cached in the ISP DNS resolvers themselves
Where are DNS requests typically cached?
application server
Which server layer deals with longer running processes that are more resource intensive?
small additional costs are required to implement shadow load balancers on commodity hardware
Why isn't availability an issue when it comes to software based load balancers
99%
Youtube: What uptime % would be considered good as we consider availability?
the client could be malicious, if the uploaded video is a duplicate it can be filtered out, encoders will only be available on a private IP address within Youtube's network
Youtube: when a user uploads a video, it is passed through a server before hitting the encoder. Why is this the case?
simple design that does not require dealing with impedance mismatch, easy horizontal scaling, highly available, cost because you do not need to pay for a RDMS and many noSQL databases are open source
advantages of NoSQL databases
a document-term index
an inverted index is an improvement ona document level index in a distributed search system. What does an inverted index make use of?
health checking of servers, TLS termination, predictive analytics on traffic patterns, reduced human intervention in the case of failure handling, increased security
aside from ensuring that services will be scalable, available, and highly performant, what are key benefits and services of load balancers?
99.9%
at what level do cloud providers like google, microsoft, and amazon set their availabilities in their SLAs?
web layer, application layer, and database layer
at what three levels will a cache generally be present in a system?
layer 4 (network/trasnport) or layer 7 (application)
at what two layers of the OSI can load balancers be placed?
a term is a frequently occuring word that can be thrown away
define "term" in the context of a document-term matrix in distributed search
DNS redirection is a method that can be used to route users to the nearest proxy servers in a content delivery network. In step one, a client is mapped to the appropriate nework location. In step two, that location distributes the load over proxy servers.
define DNS redirection
the network time protocol is a networking protocol for clock synchronization between computer systems over packet-switched, variable latency data networks
define NTP
the broker server is the core component of the pub-sub system. It handles read and write requests. A broker will have multiple topics where each topic can have multiple partitions associated with it.
define a broker in the context of a pub-sub system
a cache that embeds cache and service functionality within the same host
define a co-located cache
a report collector independent of the primary service. It is made independent to avoid the situations where client agents want to report an error to a failed service
define a collector in the context of client side monitoring
columnar databases store data in columns instead of rows which enable quick and efficient access to all entires in a database column
define a columnar database
a group of geographically distributed proxy servers where a proxy server is an intermediate server between a client and the origin server
define a content delivery network
a mathematical matrix that represents the frequency of terms in a list of documents
define a document-term matrix in the context of distributed search
the internet's naming service that maps human-friendly domain names to machine-readable IP addresses
define a domain name system
key value stores are distributed hash tables where a key binds to a specific value and does not assume anything about the structure of the value
define a key value store as a distributed systems building block
an agent that collects logs from each node and dumps them into storage. This way we do not need to visit each node if we want to know about a particular event
define a log accumulator
an intermediate component between the interacting entities known as producers and consumers
define a messaging queue
rate limiters put a limit on the number of requests a service will fulfill and throttles requests that cross the limit
define a rate limiter
each counter has a specific number of shards as needed. These shards are run on different computational units in parallel.
define a sharded counter
a persistent sequence of messages stored in the local storage of a broker
define a topic in the context of publisher-subscriber systems
the data is first written to the cache and asynchronously written to the database. Inconsistency is inevitable when the client reads stale data from the database
define a write back cache and its considerations
writes on the cache as well as the database. This increases write latency but ensures strong consistency between the database and the cache
define a write through cache and its considerations
a prober embedded in the client application that sends the appropriate service reports about any failures
define an agent in the context of client side monitoring
encoders and transcoders compress videos and transform them into different formats and qualities to support varying numbers of devices according to their screen resolution and bandwidth
define an encoder
apache kafka is a framework for distributed real-time event processing and storage
define apache kafka
a database transaction is considered an atomic unit and if one statement fails, the whole transaction is aborted and rolled back
define atomicity in regards to databases
messages are placed in the queue in the order they are received which may not necessarily be the order they were sent
define best effort ordering
a storage solution for unstructured data where every type of data is stored as a binary large object
define blob storage
consistent hashing assigns each server or item in a distributed hash table a place on an abstact circle called a ring
define consistent hashing
the system should guarantee that completed transactions will survive permanently in the databse even in system failure events
define durability in regards to databases
edge side includes is a popular dynamic content caching strategy in CDNs. Since many dynamic content web pages only change in small portions, ESI caches the portions that do not change and only notifies of the portions that do.
define edge side includes markup language
the number of requests can cross the predefined limit if the system has excess resources available
define elastic throttling
the distribution of traffic load across multiple geographic regions
define global server load balancing
there is a hard limit on the number of API reqeusts. So whenever a request crosses the limit, it is discarded
define hard throttling
scaling by increasing the number of machines in the network
define horizontal scaling
horizontal sharding is used to divide a table into multiple tables row wise
define horizontal sharding
impedence mismatch occurs in relational databases when there is a difference between the relational model and the in-memory data structures
define impedence mismatch
a characteristic of operations that allows them to be applied several times without influencing the outcome
define indempotence
the organization and manipulation of data that's done to facilitate fast and accurate information retrieval
define indexing in the context of a search system.
in the case of multiple concurrent transactions, it should give the same result as if they were run in isolation
define isolation in regards to databases
load balancing achieved within a data center
define local load balancing
all secondary nodes replicate the actual data changes.
define logical log replication
the probability that the service will restore its functions within a specified time of fault occurrence
define maintainability
in power of two load balancing, two nodes are selected at random and the one with less load is chosen. This simple technique is exponentially better than random selection
define power of two client side load balancing
an asynchronous service-to-service communication method that is popular in serverless and microservices architecture
define pub-sub messaging
the probability that the service will perform its functions for a specified time
define reliability
refers to keeping multiple copies of data at various nodes which are preferably geographically distributed to achieve availability, scalability, and performance
define replication in regards to databases
each request is forwarded to a server in the pool in a repeating sequential manner
define round robin load balancer scheduling
the ability of a system to handle an increasing amount of workload without compromising performance
define scalability
the number of requests can exceed the predefined limit by a certain percentage
define soft throttling
the primary node saves all statements that it executes and sends them to the secondary nodes to perform
define statement-based replication which is a type of primary-secondary replication
in strict ordering messages are placed in a messaging queue in the order they were produced usually based on an attached timestamp
define strict ordering in the context of messaging queues
comprised of clearly defined data types with patterns that make them easily searchable. Typically quantitative data
define structured data
the primary node waits for acknowledgements from the seconday nodes about updating the data. After receiving acknowledgement from all secondary nodes, the primary node reports success to the client
define synchronous replication
some applications provide a different level of service to users based on their IP addresses. In that case, hashing the IP address is performed to assign users' requests to servers
define the IP hash method of load balancing
DNS is a routing methodology in which all edge servers location in multiple locations share the same sinlge IP address. THe border gateway protocol is then used to route clients based on the internet's natural network flow
define the anycast method for routing clients to the nearest proxy server in DNS
they adhere to a particular schema before storing data. Instances are stored in rows, each of which has a unique key identifying that tuple which can be linked to other tables
define the key factors of a relational database
content gets sent automatically to the CDN proxy servers from the origin server in the push model
define the push CDN content caching strategy
servers are unaware of each other, there is no synchronization, data sharing, and communication between the servers
define the shared-nothing architectured utilzied by memcache
scaling by providing additional capabilities such as CPUs or RAM to an existing device
define vertical scaling
memory hardware that fetches and stores data at high speed but is lost once the system is turned off
define volatile computer memory
if some servers have a higher capability of serving a clients' requests, then it is given a higher weight. The load balancer forwards requests according to the weight of each node
define weighted round-robin load balancing
data is written only to the database and written to the cache when there is a cache miss later. Not a favorable strategy for reading recently updated data
define write around cache and its considerations
the primary node saves the query before executing it in a log file known as a write ahead log file. It then uses these logs to write data onto the secondary nodes
define write-ahead log shipping
cassandra is PA/EL
describe cassandra DB in terms of the PACELC theorem
mongoDB is PA/EC
describe mongoDB in terms of the PACELC theorem
redis does not have strong consistency due to its use of asynchronous replication
describe the consistency level of redis and why
a client sends a task to a rate limiter which accepts. The task is sent to the task submitter which gets a UUID from a generator. The task is then sent ot a databse. The task is then batched and prioritized before being placed in a distributed queue. The task is then sent to the cloud resources for processing
describe the flow of a task scheduler
the local server itself requests the root the the top level domain then the authoritatitve server
describe the iterative process of DNS resolution
a burst of requests can fill the bucket and if not processed in the specified time, recent requests can take a hit. determining the optimal bucket size and outflow rate is a challenge
disadvantages of the leaking bucket algorithm for rate limiting
no. Usually they use the services of a CDN provider like Akamai or Cloudflare
do most companies build their own CDNs?
most content providers use both to get the benefits of each approach
do most content providers use a push or pull CDN caching approach?
static content
do web servers typically serve static or dynamic content?
read throughput
does MongoDB have a higher read throughput or a higher write throughput?
a system's ability to execute persistently even if one or more of its components fail
fault tolerance
availability
for key-value stores do we prefer consistency or availability?
we introduce secondary monitoring systems which pull from the application. These then push to a datacenter monitoring system which pushes to a global monitoring system
how can we utilize a hybrid push/pull approach to improve on scalability in a monitoring service
CDNs cache content that serves as a backup whenever the origin server fails. If a proxy server stops working, other operational proxy servers step in and continue to drive traffic
how do CDNs guarantee availability
we partition the blobs based on the complete path of the blob which is the combination of the account ID, container ID, and blob ID
how do we ensure that data with the same user or container is stored in the same partition in blob store systems?
we can periodically read all the shards of a counter and cache the results
how do we improve on the low read throughput and high read latency that is inherent to sharded counters?
as a string which means all non string data must be serialized
how does memcache store data?
13 (managed by 12 different organizations)
how many DNS logical root name servers are there spread strategically around the world?
1000
how many QPS can mySQL handle?
8000
how many requests per second can a single server handle
8000
how many users per day should we estimate that a server can handle
netflix might be able to store more than 90% of its content in the CDN while this is not feasible for Youtube due to the volume of content
how might netflix and youtube have different strategies for caching their content in a content delivery network?
a few MBs
how much data can we usually store in the value component of a key-value store
the length of the network path and the capacity (bandidth) limits along the network path
network distance between a user and a proxy server in a CDN is a function of what two variables?
pagination
often a user will want a list of blobs matching some condition. However, when these lists become very large this can affect performance. What is a solution to this problem?
certain content requires the execution of scripts that can be executed at proxy servers instead of the origin server
other than edge side includes, what is a popular optimization technique for dynamic content caching in CDNs?
CPU register, CPU cache, RAM, SSD, Magnetic Disk
rank the five types of memory from fastest to slowest
No because it cannot possible meet the data storage and query requirements and failure of the one mega server would result in downtime for everyone
should key value servers be located on a single node and why?
pipelining
since redis uses a client-server model, a request cannot be handled until the server responds to the prior request. What is the method that Redis uses to get around this?
true
true or false, DNS can be considered a global server load balancer?
true
true or false, blobs are broken up into a constant amount of chunks?
false
true or false, in nonvolatile memory the processor has direct access to the data
true
true or false, weighted round robin is a static load balancing algorithm?
within the premise of a users network
typically, where are DNS resolvers located?
throttle rule retriever
under a rate limiter system, assume that the service owner adds some extra rules to the database. Consequently data in the cache will become outdated. Which components are responsible for updating the cache
right after it finishes updating itself
under asynchronous replication, when does a primary node report success to the client?
what the workload is read-heavy
under what circumstances is primary-secondary replication appropriate?
root-level name servers
what DNS name servers hold mappings for the top-level domains such as .com, .edu, .us, etc.
receiving logs, storing those logs locally, pushing the logs to a pub-sub system
what actions is a log accumulator responsible for
the sliding window log algorithm
what algorithm proposes a solution to the window edge problem that occurs in the fixed window counter algorithm
redundant cache servers
what allows for the high availability of distributed caches?
sharding of the cached data
what allows for the scalability of distributed caches?
Google Cloud firestorm and MongoDB
what are examples of document databases?
dynamic algorithms are far better because they maintain the state of serving hosts and therefore are worth the extra effort and complexity
what are more effective, dynamic or static load balancing algorithms?
local or default servers
what are other names for DNS resolvers?
least recently used, most recently used, least frequently used, most frequently used
what are some cache eviction strategies
round-robin scheduling, weighted round-robin, least connections, least response time, IP hash, URL hash
what are some examples of popular load balancing algorithms?
conflict avoidance where all writes for a given record go through the same leader, last-write-wins, custom logic
what are the three methods of conflict resolution in a multi-leader database?
account metadata, container metadata, blob metadata
what are the three types of metadata that the metadata storage in a blob store holds?
operability, lucidity, modifiability
what are the three underlying aspects of maintainability
a primary-secondary model and a cluster of independent hosts model
what are the two approaches for performing message replication in a cluster of different hosts in a messaging queue
service-side erros and client-side errors
what are the two broad categories of monitoring focus for distributed systems?
synchronous replication within a storage cluster and asynchronous replication across data centers and regions
what are the two levels of replication that occur in blob storage?
document partitioning and term partitioning
what are the two methods for partitioning in a distributed indexing/search system?
dedicated cache servers and co-located cache
what are the two methods for sharding in cache clusters
separation of search and indexing nodes and not recomputing indexes when making replicas
what are the two things that need to be fixed in order to make a distributed search system scalable
a relational database and a graph database
what are the two types of databses necessary in a task scheduling system
key-range based sharding, hash based sharding, and consistent hashing
what are the types of horizontal sharding?
very efficient for a large number of aggregation and data analytics queries by drastically reducing the disk I/O requirements and the amount of data required to load from the disk
what are the use cases for columnar databases?
the data model maps directly to objects in code, support for ubiquitous JSON documents, the high flexibility of the schema where not all documents have to have the same fields
what are three differentiators (advantages) that document databases have over relational databases?
round-robin selection, random selection, metrics-based selection
what are three methods for assigning write requests to shards in a sharded counter?
data structure storage (memcached stores all data as strings), database for persistent storage, and a message broker that can translate millions of messages per second
what are three of the key features of redis
use appropriate authentication and resource authorization, consider code sandboxing using dockers or virtual machines, use performance isolation between tasks by monitoring resource utilization
what are three ways to deal with untrusted tasks in a task scheduler
best effort ordering and strict ordering
what are two approaches to message orderig in messaging queues
low latency and high availability
what are two non functional requirements that are needed for both youtube, netflix and also any other streaming service?
a locking mechanism (suboptimal due to performance issues) and the set then get approach
what are two potential solutions to the race condition problem of rate limiters
when we have to maintain changes in the replicated data over time
what causes the main problem in data replication?
blob indexing
what component in blob store solves the problem of blob management and querying (finding a blob in a sea of blobs)?
segments help identify the start and end of a message using an offset address
what do segments do within partitions in the context of a pub-sub system
clients can still read from secondary nodes if the primary node has failed
what does it mean that primary-secondary replication is read resilient?
a cache client
what entity performs the hash calculations for caching?
proxy servers usually serve content from RAM, CDN proxy servers are placed near the users to provide faster access to content, request routing ensures users are directed to the nearest proxy server, long-tailed content is stored in nonvolatile storage instead of directing to the origin server
what factors of a CDN help it achieve high performance?
the system needs to detect such cases where a counter unexpectedly starts getting very high write traffic. We'll dynamically increase the number of shards of the affected counter to mitigate the situation
what happens when a user with just a few followers has a post go viral on twitter (in the context of counters)
IDs that would have been generated in a dead period are wasted
what is a con of using the twitter snowflake method to encode causality in unique identifiers
randomly assigning nodes in the ring may cause non-uniform distribution
what is a disadvantage of consistent hashing?
session oriented applications like web applications
what is a good application of a key-value database?
LRU cache because the most recently uploaded data is the most likely to get views
what is a good cache eviction methodology for social application
bigtable because of its high throughput and scalability for storing key-value data
what is a good databse for storing thumbnails and why
bounded waiting time for users
what is a non functional requirement that is unique to distributed task schedulers?
quorums
what is a popular approach for solving the concurrent write problem in leaderless replication database systems?
a single point of failure
what is a problem with using a central database's auto-increment feature to generate unique IDs for a sequencer?
when message size exceeds the original packet size of 512 bytes
what is a time where a DNS may use the TCP communication protocol over the UDP (user datagram protocol)?
vertical sharding is used to increase the speed of data retrievals from a table consisting of columns with very wide text or a binary large object
what is a time where vertical sharding may be useful?
Google's trutime API which return confidence intervals of time rather than points
what is a unique identifier for sequencers that guarantees all of the required properties including causality
if we use a push model to push metrics from the services to the data collector service
what is a way a service side monitoring service can be overwhelmed and become a bottleneck
mongoDB ensures atomicity in concurrent write operations and avoids collisions by returning duplcate-key errors fro record-duplication issues
what is an advantage of mongoDB that makes it a good use case for URL shortening?
cassandra
what is an example of a columnar database
mapreduce
what is an example of a distributed data processing system that an indexer will use for index construction in a distributed search system?
mongoDB
what is an example of a document database?
read only memory
what is an example of nonvolatile memory?
a system that processes financial ATM transactions
what is an example where a sampler service is not viable in distributed logging
durability
what is an important non functional requirement of messaging queues
a failover server is activated
what is done in the case of a failure of a range handler in a sequencer
volatile memory
what is more costly per unit size- volatile memory or nonvolatile memory
creating containers to group blobs
what is one of the key functional requirements of blob storage?
replication
what is one of the most commonly used methods of providing fault tolerance for a system?
rules for how to operate the rate limiter
what is stored in the database/cache associated with a database?
reduced capex and opex for extra hardware
what is the advantage of a co-located cache
using time stamps based on synchronized clocks
what is the best approach for ordering incoming messages in a messaging queue
a range handler
what is the best way of generating unique IDs for a sequencer
when the data is write once read many
what is the circumstance where blob storage is useful
client then load balancer then a layer of services with cache clients then a layer of cache servers that feed into a persisitent storage layer
what is the diagram of a distributed cache in simple terms
maintainability is measured by mean time to repair while reliability is measured by both mean time to repair as well as mean time between failure
what is the difference between maintainability and reliability
the client lacks the overall information to choose the most suitable server for its request which may result in requesting an already over-loaded server
what is the disadvantage of using client multiplexing to route a client to the nearest proxy server in DNS?
any non-deterministic function such as now() might result in distinct writes on the follower and leader
what is the disadvantage of using statement-based replication
if the primary node fails, the writes that weren't copied to the secondary nodes will be lost
what is the downside of asynchronous replication?
we need to build a system such that many nodes could collectively work as if we had a single huge server
what is the downside of horizontal scaling
high latency
what is the downside of synchronous replication
when content changes infrequently, the polling approach consumes unnecessary bandwidth
what is the downside of using period polling in a DNS to ensure content consistency?
We can only grow to the limitations of our server and the dollar cost of vertical scaling is usually high
what is the downside of vertical scaling
if two intervals overalap, we are unsure in which order the events occurred
what is the downside to using Google's truetime API as a sequencer?
each microservice sends its metrics to the monitoring system, resulting in a heavy traffic load on the infrastructure which can cause a bottleneck for business operations
what is the drawback to using a push based approach in a monitoring system
a load balancer
what is the first point of contact within a data center after the firewall?
DAU / RPS of a server
what is the formula for calculating the number of servers required
we lose a significant range any time a server dies
what is the major con of a range handler?
in a queue, only one consumer consumes a message as opposed to multiple consumers in a pub-sub system
what is the major difference between pub-sub messaging and queue based messaging?
a consistent burst of traffic at the window edges could cause a potential decrease in performance
what is the major disadvantage of the fixed window counter algorithm for rate limiting
conflict
what is the major drawback of multi-leader replication?
minimizing latency
what is the most important non functional requirement of a CDN?
setting up a configuration service that sits between the cache client and the cache servers that continuously monitors the health of the cache servers
what is the most robust (yet costly) method for keeping cache clients up to date on the presence and health of each cache server?
in a distributed system where there can be millions of events ocurring per second, we need a mechanism to distinguish these events from each other
what is the motivation for using sequencers?
continuation token
what is the name of the indicator used in pagination under a blob store system?
as nodes join or leave the system, a minimal number of keys need to move
what is the primary benefit of consistent hashing
unstructured catalog data like JSON files or other complex hierarchical data. It is also a good option for content management applications such as blogs and video platforms
what is the primary use case for document databases?
index recomputation is a waste of resources
what is the problem that arises in search systems when each replica of a primary node stores the index
kilobyte, megabyte, gigabyte, terabyte, petabyte
what is the progression of bit sizes?
increase the availability of the queue
what is the purpose of replicating queues on multiple servers?
memcahed has a nearly deterministic O(1) speed serving millions of keys per second
what is the query speed of memcached
verify the consumer, retention time management, message receiving options management, allow multiple reads
what is the role of a consumer manager in a pub-sub system
to fetch content and create documents
what is the role of a crawler in a search system?
resolvers initaite the querying sequence and forward requests to the other DNS name servers. They can also cater to user's DNS queries through caching techniques
what is the role of a dns resolver in the dns infrastructure hierarchy?
to fairly divide all clients' requests among the pool of available servers
what is the role of a load balancer
responds to search queries by running the query on the index created by the indexer
what is the role of a searcher in a search system
broker/topics registry and managing replication by assigning the new lead broker in the case of failure
what is the role of the cluster manager in a pub-sub system
a temporary data storage that can serve data faster by keeping data entries in memory
what is the simple definition of a cache
we compute the index on the primary node only and then communicate the inverted index (binary blob/file) to the replicas
what is the solution ot index recomputation when replicating nodes in a distributed search system?
storing metadata indicating a time to live
what is the solution to having stale data stored in a distributed cache
if the number is too small, we face high contention for writes. If the number is too large we encounter higher overhead on the read operation
what is the tradeoff in determining the number of shards for a sharded counter?
web server
what is typically the first point of contact after a load balancer?
mean time to repair
what is used to measure maintainability
all of the writes must go through the leader node
what limits the performance of primary-secondary replication?
multiword queries necessitate sending long mapping lists between groups of nodes for merging
what makes term partitioning difficult in distributed search indexers
scalability, availability, performance
what non functional characteristics do load balancers improve
the write reqeuest queue increases, maximum shard utilization decreases, the user will not get quick responses
what problems might arise if shards are selected in an order (sequentially) rather than randomly?
iterative
what type of DNS resolution reduces query load on the DNS infrastructure- iterative or recursive?
the metadata for a blob's chunks is cached on the client side when its read for the firs time. The client can then go directly to the data nodes without communicating to the master node.
what type of caching can occur on the client side in a blob store system?
asynchronous communication
what type of communication do messaging queues enable?
universal datagram protocol (UDP)
what type of communication protocol do many clients of DNS use?
eventual consistency
what type of consistency level do domain name systems guarantee?
HTTP responses as well as other protocols
what type of content do applications servers serve
a flat data organization pattern with no hierarchies or sub directories
what type of data organization pattern does blob storage follow?
propogation delay by bring the data closer to its users
what type of delay does a CDN primarily try to alleviate?
a stateless load balancer
what type of load balancer does not keep track of any session information?
messages that failed to be consumed and have reached the maximum attempts limit
what type of messages do dead-letter queues contain?
leader-follower protocol
what type of replication does MongoDB use?
leaderless
what type of replication does the cassandra DB use?
vertical sharding
what type of sharding (vertical or horizontal) is more amendable to manual partitioning?
Different ISPs have different numbers of users resulting in uneven load distribution. It does not consider end-server crashes
when a DNS infrastructure responds with a reordered list of IP addresses, it is essentially performing global server load balancing in a round robin format. What are the two disadvantages to this approach?
static content delivery where the origin server decides which content to deliver to users of the CDN
when is the push CDN content strategy appropriate?
when we have incoming requests with extensive service time or requests with widely differing length of service time
when is the weighted round robin algorithm inappropriate for load balancing?
a sampler service
when it is not viable to log every single event in a distributed logger, what can we use instead?
if we are operating an application where we need to be able to make changes even if we are offline such as a calendar application
when might a multi-leader replication strategy be useful?
when a firewall prevents the monitoring system from accessing the servers directly
when might a push metrics model be appropriate in a monitoring system?
if the data is unstructured, if there is a need to serialize and deserialize data, if the size of the data to be stored is large
when should a non-relational database be chosen?
if the data to be stored is structured, if acid properties are required, if the size of the data is relatively small and can fit on one node
when should a relational database be chosed
non relational
when strong consistency is not required, what might be a good type of database to use?
highly available but weak consistency
where do noSQL databases fall on the spectrum of consistency and availability?
in RAM to support low latency for the search
where do we store the index in distributed search and why?
web server
which server layer deals with HTTP requests and responses only?
application server
which server layer deals with several communication protocols one of which is HTTP?
there is no indication of the severity of the issues, no structure, and they are hard to track
why can't we just use print statements instead of distributed logging systems?
there is a cost to queue storage and some tasks need to be repeatedly scheduled at regular intervals and so need to be stored for long periods
why do we use a database in task scheduling instead of just storing everything in the queue
because it makes a great tradeoff between speed, storage size, and cost
why is RAM a good choice for serving cached data as opposed to CPU register or Magnetic Disk
all API requests pass through the rate limiter. Therefore it needs minimum latency so as not to affect the user experience
why is low latency critical for rate limiters
it provides a much shorter delay
why might a DNS system favor the unreliable UDP communication protocol over something like TCP that uses a handshaking dialogue?
statement-based replication, write-ahead log shipping, logical (row-based) log replication
name three different types of primary-secondary replication
kilobyte
2 ^ 10 bits
gigabyte
2 ^ 30 bits
petabyte
2 ^ 50 bits
megabyte
2^20 bits
terabyte
2^40 bits
the percentage of time some system is accessible to clients is is operated under normal conditions
Define availability
dependent and independent tasks
broadly, tasks can be of two types. What are they?
a fault tolerance technique that saves a system's state in stable storage when the system state is consistent
checkpointing
what are the disadvantages of the token bucket algorithm for rate limiting
choosing an optimal value for the parameters is a difficult task. A lock might require taking a token from the bucket that can increase the request's processing delayt if contention on the lock increases
they can perform global traffic management between different time zones
cloud based load balancers may not necessarily replace a local on-premise load balancing facility. In that case, what is their primary role?
redis runs as a single process using one core whereas memcached can efficiently use multicore systems with multithreading technology
compare multithreading on memcached and redis
redis provides persistance while memcached does not although it can be achieved with third party tools
compare persistence of memcached and redis
redis automates replication whereas in memcached it is subject to third party tools
compare replication in redis vs memcached
memcached is simple but leaves most of the effort for managing clusters to the developer. Redis automates most of the scalability and data division issues
compare the simplicity of memcached and redis
what are the steps that occur when a user writes a blob to blob store
if the request succesfully passes the rate limiter, the load balancer forwards the client's request to one of the front end servers. Front end server then requests the master node. The master node assigns a unique ID, splits the data into chunks. The chunks are replicated for redundancy. The master node stores metadata in metadata storage. After writing the blob, a fully qualified path if returned to the client.
top-level domain name servers
in a DNS, what type of name server do root level servers forward queries to?
blob storage
in a client side monitoring service, what kind of database will we use to store the time series data?
the URI namespace delegation of all objects cached in the CDN
in a content delivery network, what does the origin server provide to the request routing system
filterer, error aggregator, alert aggregator
in a distributed logger, what are the three services that will work on the pub-sub data
least response time
in a performance sensitive service, which load balancing algorithm might be preferred?
a queue
in a pub-sub system, what is used to maintain each topic
relational
in a pub-sub system, what type of database (relational or non-relational) is used to maintain which consumers have subscribed to which topics
by maintaining a set of rules in a rules and actions database
in a service side distributed monitoring system, how can we ensure that our system will send out an alert when a metric breaches a critical value
execute these tasks in off-peak resource times
in a social app like facebook, some tasks do not need to be executed urgently such as recommending friends. What is a good way to execute these tasks in a resource efficient mannager?
execution caps
in a task scheduler, some task scripts may never halt due to a bug in the script. What is the solution?
monitoring the health of servers directly and handling TCP congestion control protocols
in addition to distributing traffic to servers, what is the role of tier three load balancers which sit at layer 7?
we mark blobs as deleted and they will be garbage collected later
in blob storage, deleting from many nodes takes time and holding a client until that is done is not a viable option. What is the solution?
the data node
in blob storage, where does the garbage collection process run: at the master node, the data node, or the monitoring system?
partitions
in blob store, there are a large number of data nodes on which blobs can be stored. It would take a while to search each one for a particular blob. What method is used to overcome this?
metadata storage
in blob store, where are the partition mappings stored?
embedding two propers in the application which will be an agent and a collector
in client side error monitoring, what is an improvement on the simple prober approach?
these components extract and filter terms from the partitions assigned to it by the cluster manager. These machines output inverted indexes in parallel which serve as input to reducers.
in mapreduce, what is the role of mappers?
the reducer combines mappings for various terms to generate a summarized index
in mapreduce, what is the role of reducers
the manager initiates the process by assigning a set of partitions to mappers. Once the mappers are done, the cluster manager assigns the output of mappers to reducers
in mapreduce, what is the role of the cluster manager?
using a time window approach where we have to sort messages received within a specific time frame and then put them in the relevant queue
in messaging queues, we frequently need to sort messages based on timestamps. How can we minimize latency based on this extra processing time?
we can tag a unique process identifier with the time stamp
in messaging queues, we will use time stamps based on synchronized clocks to order incoming messages. How can we ensure that we order messages when two concurrent sessions ask for a timestamp at the exact same time?
asynchronous writes
in the case of distributed caches, do we prefer synchronous or asynchronous writes to replica servers to copy data?
any random host within the cluster
in the cluster of independent hosts model, which component is responsible for replicating messages in the other nodes?
placing CDN proxy servers in ISP network
in the context of CDN deployment, what does off-premises mean?
a smaller data center could be placed near major IXPs
in the context of CDN deployment, what does on-premises deployment mean
time to live
in the periodic polling method of CDN content consistency, a lot of bandwidth is unnescessarily consumed by polling for content that does not change frequently. What is a strategy that addresses this issue
sign bit, 41 bits for time stamp in milliseconds, 10 bits for a worker number which identifies the server, and 12 bits for the sequence number which is incremented by the server each time
in the twitter snowflake sequencer method, what are the different types of bits used for encoding
nonpersisten
is a cache a persistent or non persistent storage area?
client, rate limiter, load balancer, front-end servers, data nodes, master node, metadata storage, monitoring service, administrator
list the key components of blob store design
edge side includes (ESI) markup language
name a popular dynamic data compression technique for caching data in a CDN
routing system, scrubber servers, proxy servers, distribution system, origin servers, management system
name the components of a content delivery network (CDN)
monitor anomalies in the use of CPU/memory/disk/network bandwidth by a process, monitor overall server health, monitor hardware component faults on a server such as memory failures, monitor server's ability to reach out-of-server critical services such as a network file system
what are some examples of things we want our monitoring system to do for us in distributed monitoring?
failure in DNS name resolution, any failure in routing along the path from the client to the service provider, any failures with third-party infrastructure such as middleboxes and CDNs
what are some factors that can cause failure in clients being unable to reach the server?
temperature differences, equipment age, manufacturing defects, virtualized clocks
what are some of the causes of physical clock drift
proxy servers usually serve content from RAM, CDN proxy servers are usually placed near the users, the request routing system ensures that users are directed to the nearest proxy server, proxy servers have long-tail content sotred in nonvolatile storage systems
what are some of the design decisions in CDNs that minimize latency
preventing resource starvation, managing policies and quotas, controlling data flow, and avoiding excess costs
what are some scenarios where a rate limiter would be useful?
DNS redirection, anycast, client multiplexing, HTTP redirection
what are some techniques that can be used to route users to the nearest proxy server in a CDN?
the advantage is that keys are uniformly distributed across nodes. The disadvantage is that we cannot perform range queries with this technique
what are the advantages and disadvantages of hash based sharding
flexibility because the database can be modfied while queries are happening, reduced redundancy, concurrency, backup and disaster recovery
what are the advantages of relational databases
it can cause a burst of traffic as long as there are enough tokens. it is space efficient as there are limited states
what are the advantages of the token bucket algorithm for rate limiters
an inverted index facilitates full-text searches and reduces the time of counting the occurrence of a word in each document because we have mappings against each term
what are the advantages of using an inverted index in search?
there is flexibility in terms of hardware choices for each functionality and it is possible to scale application servers and cache servers separately
what are the advantages of using dedicated cache servers for sharding?
range-query-based scheme is easy to implement and range queries can be performed using the partition keys
what are the advantages of using key-range based data sharding?
if a node fails or does routine maintenance, the workload is uniformly distributed over other nodes. It is up to each node to decide how many virtual nodes it is responsible for which allows nodes with more computational capacity to take more of the load
what are the benefits of using virtual nodes in consistent hashing?
a cache serves data from ram while a key-value store stores writes data to non volatile storage. Key-value stores should survive failures while cashes must be repopulated from scratch
what are the differences between a cache and a key value store
there is storage overhead and additional processing time up front for adding a new document
what are the disadvantages of using an inverted index in distributed search?
impedance mismatch
what are the downsides of relational databases
the content provider cannot do anything if the public CDN is down, if there are no proxy servers location in the region where some traffic comes from they are out of luck, and it is possible that some domains or IP addresses of CDN providers are blocked or restricted in some places
what are the downsides of using a public CDN
incomplete coverage and lack of user imitation
what are the downsides of using probers to mimick user behavior in client side monitoring?
concurrent management to separate users, we cannot grant different access rights, scaling, content search is difficult
what are the limitations of file storage?
a log accumulator, storage, a log indexer, and visualization
what are the major components of a distributed logging system?
scalable, available, and fault tolerant
what are the non functional requirements of a key-value store
they minimize user perceived latency, pre-generate expensive queries from the database, store user session data temporarily, serve data from temporary storage even if the data store is down temporarily, reduce network costs by serving data from local resources
what are the primary benefits of a distributed cache?
uniqueness, scalability, availability, and a 64 bit ID
what are the requirements for a unique identifier system
requirements, estimation, storage schema, high level design, api design, detailed design, evaluation, distinctive component
what are the steps in the reshaded approach
configurable service where different applications can choose between the tradeoffs of consistency and availability, the ability to alwasy write to the key value store, and hardware heterogeneity
what are the three functional requirements of a key value store
a data collector service, a time series database, a querying service, a rules and actions database, a blob store, and a service discoverer like kubernetes, an alert manager, and a dashboard
what are the three high level components of a client side distributed monitoring service?
account layer, container layer, blob layer
what are the three levels of abstraction in blob store?
a crawler, an indexer, and a searcher
what are the three main components of a search system?
periodic polling, time to live, and leases
what are the three methods for maintaining content consistency in a CDN?
