Hadoop - HDFS and MR
What is hadoop flow
(input) -> map -> -> combine/sorting -> -> reduce -> (output)
What is NameNode
- It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept.
What happens if you don't override the Mapper methods and keep them as it is?
If you do not override any methods (leaving even map as-is), it will act as the identity function, emitting each input record as a separate output.
Which are the methods in the Mapper interface?
run() setup() map() cleanup() The Mapper contains the run() method, which call its own setup() method only once, it also call a map() method for each input and finally calls it cleanup() method. All above methods you can override in your code.
What is DataNode
DataNode stores data
What is HDFS Block size?
Each block is typically 64Mb or 128Mb in size
Each slave node is configured with job tracker node location.
Each slave node is configured with job tracker node location.
What is a Task instance in Hadoop
Task instances are the actual MapReduce jobs which are run on each slave node. The TaskTracker starts a separate JVM processes to do the actual work (called as Task Instance) this is to ensure that process failure does not take down the task tracker
How job tracker schedules task
When the JobTracker tries to find somewhere to schedule a task within the MapReduce operations, it first looks for an empty slot on the same server that hosts the DataNode containing the data, and if not, it looks for an empty slot on a machine in the same rack.
What happens to data if reducers are set to zero
When you set the reducers to zero no reducers will be executed, and the output of each mapper will be stored to a separate file on HDFS. [This is different from the condition when reducers are set to a number greater than zero and the Mappers output (intermediate data) is written to the Local file system(NOT HDFS) of each mappter slave node.]
What is difference between 0.95 and 1.75 for number of reducers
With 0.95 all of the reduces can launch immediately and start transferring map outputs as the maps finish. With 1.75 the faster nodes will finish their first round of reduces and launch a second wave of reduces doing a much better job of load balancing. Increasing the number of reduces increases the framework overhead, but increases load balancing and lowers the cost of failures.
Main content of hdfs-site.xml
replication number Block size Name directory data directory
Primary role of Secondary name node
- Do periodic checkpoints - Secondary name node periodically downloads the latest image file and edit logs, add them to the existing image, and then uploads them into the primary name node - In case, you use the fsImage at the secondary namenode, it would not have the latest edit logs. Also, if you are using the seconary namenode image, you have to restart the entire cluster. - Secondary namenode stores the editlog by copying the the logs from the namenode. This way, the space on the namenode is freed which reduces the start-up time. Secondary namenode doesn't store the metadata and hence, if Namenode is corrupted, entire HDFS breaks down
What is difference of HDFS over NAS
- HDFS is distributed across local drives of all machines - HDFS uses MapReduce. NAS can't because data is stored separately from computational area - HDFS support data replication
Default replication factor
3. Two blocks on on same rack on different rack
What is task tracker in Hadoop
A TaskTracker is a slave node daemon in the cluster that accepts tasks (Map, Reduce and Shuffle operations) from a JobTracker. There is only One Task Tracker process run on any hadoop slave node. Task Tracker runs on its own JVM process. Every TaskTracker is configured with a set of slots, these indicate the number of tasks that it can accept.
What is the InputSplit in map reduce software?
An InputSplit is a logical representation of a unit (A chunk) of input work for a map task; e.g., a filename and a byte range within that file to process or a row set in a text file.
How can you add the arbitrary key-value pairs in your mapper?
Arbitary key, value pairs can be set to Job Configuration. e.g. with Job.getConfiguration().set("myKey", "myVal"), and then retrieve this data in your mapper with Context.getConfiguration().get("myKey"). This kind of functionality is typically done in the Mapper's setup() method.
What are the actions job tracker performs?
Client applications submit jobs to the Job tracker. The JobTracker talks to the NameNode to determine the location of the data The JobTracker locates TaskTracker nodes with available slots at or near the data The JobTracker submits the work to the chosen TaskTracker nodes. The TaskTracker nodes are monitored. If they do not submit heartbeat signals often enough, they are deemed to have failed and the work is scheduled on a different TaskTracker. A TaskTracker will notify the JobTracker when a task fails. The JobTracker decides what to do then: it may resubmit the job elsewhere, it may mark that specific record as something to avoid, and it may may even blacklist the TaskTracker as unreliable. When the work is completed, the JobTracker updates its status. Client applications can poll the JobTracker for information.
What are combiners?
Combiners are used to increase the efficiency of a MapReduce program. They are used to aggregate intermediate map output locally on individual mapper outputs. Combiners can help you reduce the amount of data that needs to be transferred across to the reducers.
What is compute and Storage nodes?
Compute Node: This is the computer or machine where your actual business logic will be executed. Storage Node: This is the computer or machine where your file system reside to store the processing data. In most of the cases compute node and storage node would be the same machine.
Which object can be used to get the progress of a particular job ?
Context
Where does task instance run?
Each task instance runs on separate JVM so that failure of one task instance doesn't bring down the other task instance. Each Task Instance runs on its own JVM process. There can be multiple processes of task instance running on a slave node. This is based on the number of slots configured on task tracker. By default a new task instance JVM process is spawned for a task
Where do you specify the Mapper Implementation?
Generally inside job itself
HDFS is write once and read multiple times
HDFS supports write-once-read-many semantics on files.
HDFS uses rack-aware replica placement policy
HDFS uses rack-aware replica placement policy
Why speculative execution is important?
Hadoop can run multiple instance of a given MR job if it feels that certain job is not happening at the fast pace. The job which is finished faster gives the output which is used. Other operations are then closed. This may result in delays in a full job due to only one machine not performing well. To avoid this, speculative execution in Hadoop can run multiple copies of same map or reduce task on different slave nodes. The results from the first node to finish are used.
HDFS, NAS and SAN
Hadoop distributed file system, Network attached storage and Storage area network
How many Daemon processes run on a Hadoop system?
Hadoop is comprised of five separate daemons. Each of these daemon run in its own JVM. Following 3 Daemons run on Master nodes NameNode - This daemon stores and maintains the metadata for HDFS. Secondary NameNode - Performs housekeeping functions for the NameNode. JobTracker - Manages MapReduce jobs, distributes individual tasks to machines running the Task Tracker. Following 2 Daemons run on each Slave nodes DataNode - Stores actual HDFS data blocks. TaskTracker - Responsible for instantiating and monitoring individual Map and Reduce tasks.
Data format of Hadoop
Hadoop works on data pairs (key, Value)
In Hive, what is the difference between External vs. Internal Tables?
Hive manages Internal tables. When a DROP is performed, the data and metadata is deleted. On an External table, Hive does not manage. When a DROP is performed here, only the metadata is deleted. The data is still intact.
What is Identity Reducer
IdentitityReducer- it writes all the input values ot the output. If reducer class is not set, identityReducer is taken as default value. org.apache.hadoop.mapred.lib.IdentityReducer Performs no reduction, writing all input values directly to the output. If MapReduce programmer do not set the Reducer Class using JobConf.setReducerClass then IdentityReducer.class is used as a default value.
What is Identity Mapper
IdentityMapper- It maps the input with the output. If nothing is specified, then Identitymapper class is used as default value. org.apache.hadoop.mapred.lib.IdentityMapper Implements the identity function, mapping inputs directly to outputs. If MapReduce programmer do not set the Mapper Class using JobConf.setMapperClass then IdentityMapper.class is used as a default value.
MapReduce data types are all immutable
If in a mapping task you change an input (key, value) pair, it does not get reflected back in the input files
When is the reducers are started in a MapReduce job?
In a MapReduce job reducers do not start executing the reduce method until the all Map jobs have completed. Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The programmer defined reduce method is called only after all the mappers have finished.
What happens if number of reducers are 0?
In this case the outputs of the map-tasks go directly to the FileSystem, into the output path set by setOutputPath(Path). The framework does not sort the map-outputs before writing them out to the FileSystem
Explain the Reducer's reduce phase?
In this phase the reduce(MapOutKeyType, Iterable, Context) method is called for each pair in the grouped inputs. The output of the reduce task is typically written to the FileSystem via Context.write(ReduceOutKeyType, ReduceOutValType). Applications can use the Context to report progress, set application-level status messages and update Counters, or just indicate that they are alive. The output of the Reducer is not sorted.
Explain the shuffle?
Input to the Reducer is the sorted output of the mappers. In this phase the framework fetches the relevant partition of the output of all the mappers, via HTTP.
What happens if reduce() method or any other method is not overwritten
It behaves as IdentityReducer. the input keys are stored without any changes
What is the use of combiner
It is an optional component or class, and can be specify via Job.setCombinerClass(ClassName), to perform local aggregation of the intermediate outputs, which helps to cut down the amount of data transferred from the Mapper to the Reducer.
can a job contain zero reducers
It is legal to set the number of reduce-tasks to zero if no reduction is desired.
What is context.write() call do?
It writes the output to File System
Main content of mapred-site.xml
Job tracker address Max map tasks Heap size Local directory System directory
What is job tracker in Hadoop
JobTracker is the daemon service for submitting and tracking MapReduce jobs in Hadoop
What is the Reducer used for?
Reducer reduces a set of intermediate values which share a key to a (usually smaller) set of values. The number of reduces for the job is set by the user via Job.setNumReduceTasks(int).
How are joins performed?
Joins are utilized for optimizing queries. Need to decide which column you want to merge your files on.
How many NameNode process run on Hadoop cluster
Just One Namenode process NameNode run on its own JVM
What is writable interface
Key = org.apache.hadoop.io.WritableComparable interface. Value = org.apache.hadoop.io.Wrtable interface org.apache.hadoop.io.Writable is a Java interface. Any key or value type in the Hadoop Map-Reduce framework implements this interface. Implementations typically implement a static read(DataInput) method which constructs a new instance, calls readFields(DataInput) and returns the instance.
What is WritableComparable interface?
Key = org.apache.hadoop.io.WritableComparable interface. Value = org.apache.hadoop.io.Wrtable interface org.apache.hadoop.io.WritableComparable is a Java interface. Any type which is to be used as a key in the Hadoop Map-Reduce framework should implement this interface. WritableComparable objects can be compared to each other using Comparators.
When Hadoop is not best choice
Low latency operations - where faster response is required. Sequential algorithms - Where the output of one dataset need to feed to other dataset Joins - If you want to join one dataset to another to perform some operation
What Mapper does?
Maps are the individual tasks that transform input records into intermediate records. The transformed intermediate records do not need to be of the same type as the input records. A given input pair may map to zero or many output pairs.
How does an Hadoop application look like or their basic components?
Minimally an Hadoop application would have following components. Input location of data Output location of processed data. A map task. A reduced task. Job configuration The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTracker which then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.
What are main contents of core-site.xml
NameNode address, temp hadoop directory
How NameNode Handles data node failures?
NameNode periodically receives a Heartbeat and a Blockreport from each of the DataNodes in the cluster. Receipt of a Heartbeat implies that the DataNode is functioning properly. A Blockreport contains a list of all blocks on a DataNode. When NameNode notices that it has not recieved a hearbeat message from a data node after a certain amount of time, the data node is marked as dead. Since blocks will be under replicated the system begins replicating the blocks that were stored on the dead datanode. The NameNode Orchestrates the replication of data blocks from one datanode to another. The replication data transfer happens directly between datanodes and the data never passes through the namenode.
Can we disable speculative execution
Speculative execution is enabled by default. You can disable speculative execution for the mappers and reducers by setting the mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution JobConf options to false, respectively.
Is output of Reducer sorted?
No
Does Secondary name node store metadata
No. Secondary namenode stories only edit log
In the event of Identity reducers(zero reducers), does the map output be sorted?
No. The framework doesn't sort map output
Can reducers work togeather
No. reducers cannot they work in a silo type of environment
Does MapReduce programming model provide a way for reducers to communicate with each other? In a MapReduce job can a reducer communicate with another reducer?
Nope, MapReduce programming model does not allow reducers to communicate with each other. Reducers run in isolation.
How many instances of Job Tracker run in Hadoop cluster
One job tracker process per hadoop cluster Job Tracker runs on its own JVM process. In a typical production cluster its run on a separate machine.
How many instances of TaskTracker run on a Hadoop Cluster
One task tracker per slave node. Multiple task instances on a slave node which runs on separate JVM.
How many instances of JobTracker can run on a Hadoop Cluser?
Only One
What are the primary phases of the Reducer?
Shuffle, Sort and Reduce
What is the meaning of speculative execution in Hadoop?
Speculative execution is a way of coping with individual Machine performance. In large clusters where hundreds or thousands of machines are involved there may be machines which are not performing as fast as others.
What is configuration of a typical slave node on Hadoop cluster?
Task tracker, Datanode daemon and *task-instance Single instance of a Task Tracker is run on each Slave node. Task tracker is run as a separate JVM process. Single instance of a DataNode daemon is run on each Slave node. DataNode daemon is run as a separate JVM process. One or Multiple instances of Task Instance is run on each slave node. Each task instance is run as a separate JVM process. The number of Task instances can be controlled by configuration. Typically a high end machine is configured to run more task instances.
If reducers do not start before all mappers finish then why does the progress on MapReduce job shows something like Map(50%) Reduce(10%)? Why reducers progress percentage is displayed when mapper is not finished yet?
That represents data transfer. Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The progress calculation also takes in account the processing of data transfer which is done by reduce process, therefore the reduce progress starts showing up as soon as any intermediate key-value pair for a mapper is available to be transferred to reducer. Though the reducer progress is updated still the programmer defined reduce method is called only after all the mappers have finished.
What is the use of Context object?
The Context object allows the mapper to interact with the rest of the Hadoop system. It Includes configuration data for the job, as well as interfaces which allow it to emit output.
What is the InputFormat ?
The InputFormat is responsible for enumerate (itemise) the InputSplits, and producing a RecordReader which will turn those logical work units into actual physical input records.
Job Tracker execution steps
The JobTracker is single point of failure for the Hadoop MapReduce service. If it goes down, all running jobs are halted. JobTracker in Hadoop performs following actions Client applications submit jobs to the Job tracker. The JobTracker talks to the NameNode to determine the location of the data The JobTracker locates TaskTracker nodes with available slots at or near the data The JobTracker submits the work to the chosen TaskTracker nodes. The TaskTracker nodes are monitored. If they do not submit heartbeat signals often enough, they are deemed to have failed and the work is scheduled on a different TaskTracker. A TaskTracker will notify the JobTracker when a task fails. The JobTracker decides what to do then: it may resubmit the job elsewhere, it may mark that specific record as something to avoid, and it may may even blacklist the TaskTracker as unreliable. When the work is completed, the JobTracker updates its status. Client applications can poll the JobTracker for information.
What is the Hadoop MapReduce API contract for a key and value Class?
The Key must implement the org.apache.hadoop.io.WritableComparable interface. The value must implement the org.apache.hadoop.io.Writable interface.
How does master slave architecture in the Hadoop?
The MapReduce framework consists of a single master JobTracker and multiple slaves, each cluster-node will have one TaskskTracker. The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.
How mapper is instantiated
The Mapper itself is instantiated in the running job, and will be passed a MapContext object which it can use to configure itself.
How does Mapper's run() method works?
The Mapper.run() method then calls map(KeyInType, ValInType, Context) for each key/value pair in the InputSplit for that task
How task tracker works
The TaskTracker starts a separate JVM processes to do the actual work (called as Task Instance) this is to ensure that process failure does not take down the task tracker. The TaskTracker monitors these task instances, capturing the output and exit codes. When the Task instances finish, successfully or not, the task tracker notifies the JobTracker.
Explain the Reducer's Sort phase?
The framework groups Reducer inputs by keys (since different mappers may have output the same key) in this stage. The shuffle and sort phases occur simultaneously; while map-outputs are being fetched they are merged (It is similar to merge-sort).
Explain reduce()
The heart of Reducer is its reduce() method. This is called once per key; the second argument is an Iterable which returns all the values associated with that key.
What are the restriction to the key and value class ?
The key and value classes have to be serialized by the framework. To make them serializable Hadoop provides a Writable interface. The key of the Map should be comparable, hence the key has to implement one more interface WritableComparable.
Where is the Mapper Output (intermediate key-value data) stored ?
The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the hadoop administrator. The intermediate data is cleaned up after the Hadoop Job completes.
How many maps are there in a particular Job?
The number of maps is usually driven by the total size of the inputs, that is, the total number of blocks of the input files. Generally it is around 10-100 maps per-node. Task setup takes awhile, so it is best if the maps take at least a minute to execute.
What happens after mapper
The output of the Mapper are sorted and Partitions will be created for the output. Number of partition depends on the number of reducer.
How many Reducers should be configured?
The right number of reduces seems to be 0.95 or 1.75 multiplied by (<no. of nodes> * mapreduce.tasktracker.reduce.tasks.maximum).
How client talk to HDFS
Through Hadoop HDFS API. Call to Name node which return list of datanode addresses where block present Call to datanode to retrieve the data
What is the use of custom partitioner
To control which key goes to which reducer
How many JVMs run on a slave node?
Total 3 or more instances of JVM runs on each slave node
Can I set the number of reducers to zero?
Yes, Setting the number of reducers to zero is a valid configuration in Hadoop.
Can you perform a MapReduce job on compressed data?
Yes, you can perform mapReduce job on compressed data. Pentaho has its engine which reads the compressed mapreduce data, uncompresses it and performs the operation. You can also even compress the IKV value to minimize the network bandwidth.
is JobTracker a single point of failure
Yes. The JobTracker is single point of failure for the Hadoop MapReduce service. If it goes down, all running jobs are halted
Can block size and replication be configured
Yes. they can be configured per FILE
When to use combiner in MR
You can use your reducer code as a combiner if the operation performed is commutative and associative. The execution of combiner is not guaranteed, Hadoop may or may not execute a combiner. Also, if required it may execute it more then 1 times. Therefore your MapReduce jobs should not depend on the combiners execution.
What are basic configuration files of Hadoop
core-site.xml hdfs-site.xml mapred-site.xml
How many DataNode process run in Hadoop cluster
only One DataNode process run on any hadoop slave node During startup data node connects to name node
Which interface needs to be implemented to create Mapper and Reducer for the Hadoop?
org.apache.hadoop.mapreduce.Mapper org.apache.hadoop.mapreduce.Reducer
Can datanode talk each other
yes. typically when data replication is happening