Data Science - Big Data Assessment
which of these can be the purpose of counters?
all: the number of tasks that were launched and successfully ran the correct number of bytes that were read and written number of mapper and reducer launched
which of the following will give min value of votes in the dataframe "df"? df.max() df.agg(min("votes")).show() df.agg().min("votes").show() none
df.agg(min("votes")).show()
can Spark RDD be shared between the spark context?
false
is sql querying supported in HBase?
false
A Dataframe can be created from an existing RDD. You would create the Dataframe from the existing RDD by inferring schema using case classes in which one of the given classes? - if your dataset has more than 22 fields - if all your users are going to need dataset parsed in same way - if you have two sets of users who will need the text dataset parsed differently - none of these
if you have two sets of users who will need the text dataset parsed differently Spark SQL and DataFrames - Spark 2.3.1 Documentation (spark.apache.org/docs/2.3.1/sql-programming-guide.html) ctrl+f 'parsed'
below are teh steps given to compute the number of lines which contain the word "ADMIN". val inRDD-sc.textFile("/user/user01/data/weatherdat.csv") val tempRD=inRDD.map(line=>line.split("|")) val tempRDD=inRDD.filter(line=>line.contains("Admin")) val tempCount=tempRDD.count() choose the step where the inputRDD will be computed
inputRDD is computed and data loaded when count() is applied **bc "The transformations are only computed when an action requires a result to be returned to the driver program." https://spark.apache.org/docs/latest/rdd-programming-guide.html
which are the options available in spark to save the data to disk only, where dataFrame object is referred as 'out'?
out.persist(StorageLevel.DISK_ONLY) ??
which of the following tool defines a data flow language? - hive - HBase - MapReduce - Pig
pig
what type of database is HBase? schema-rigid schema-flexi schema-less not a db
schema-less
which is the default serde in Hive? - DynamicSerDe - ThriftSerDe - metadataTypedColumnsetSerDe - LazySimpleSerDe
thriftSerDe
RDDs can also be unpersisted to remove RDD from permanent storage like memory and/or disk
true
spark supports loading data from HBase
true
zero reducers are allowed in map reduce
true
Why is Serde used in hive? - used for serialization and deserialization - compress the file - read and write files into hdfs - lazyloading of files into cache
used for serialization and deserialization * to read and write??
An election RDD is provided with tuple (candidate, count) which of the below Scala snippet is used to get the candidate with the lowest number of votes? val low = electionRDD.sortByKey().first val low = electionRDD.sortByKey(false).first (not this one, goes in descending order here) val low = electionRDD.map(x=>(x._2, x._1)).sortByKey().first val low = ocuntryRDD.map(x->x._2,x._1)).sortByKey(false).first (not this one)
val low = electionRDD.sortByKey().first
partitioner is used in a mapreduce job - when there are multiple reducers - when keys have to be grouped and sent to particular mapper - when keys have to be grouped and send to particular reducer - none
when there are multiple reducers
which of the following language is used by apache CouchDB for storing the data? javascript json csv xml
xml
is relational join on two large tables on common key possible in mapreduce job?
yes
choose the correct statement about RDD - none -rdd is a programming paradigm - rdd is a distributed data structure - rdd is a database
-rdd is a programming paradigm
in what year was apache spark made an open-source technology?
2010
you have the following key-value pairs as output from your map test (the,1) (fox, 1) (faster,1) (than,1) (the,1) (cat,1) how many keys will be passed to the reducer? -3 -4 -5 -6
5
which is the default database for storing metadata of hive tables? - MySQL - derby - oracle - none of these
Derby
Apache CouchDB is written in ____ programming language java javascript erlang python
Erlang
By default, Spark uses which algorithm to remove old and unused RDD to release more memory?
Least Recently Used (LRU)
which is the default storage level in spark? MEMORY_ONLY MEMORY_ONLY_SER MEMORY_AND_DISK MEMORY_AND_DISK_-SER
MEMORY_ONLY
The programming paradigm used in Spark is __________. - none of the options - generalized - map only - mapreduce
MapReduce
what is the input format in mapreduce job to read multiple lines at a time - TextInputFormat - MultiLineInputFormat - NLineTextInputFormat - NLineInputFormat
NLineInputFormat?? textinputformat??
which of these are responsible to output key-value pairs from the Reducer phase to output files? reducer Writer RecordReader RecordWriter none of these
RecordWriter
____ protocol is used for API to access the documents SMTP HTTP IP SSH
SMTP
which tells Spark how and where to access a cluster? Spark Streaming Spark Context Spark Core Spark Session
Spark Context
Yarn architecture, which replaces the classic MR1 architecture, removes which one? - single point of failure in the Namenode - resource pressure on the JobTracker (this one??) - HDFS latency (not this) - ability to run frameworks other than MapReduce, such as MPI (this one??)
ability to run frameworks other than mapreduce such as MPI AND resource pressure on the JobTracker
Oozie workflow jobs are Directed _____ graphs of actions? - acyclical -cyclical -elliptical - all the above
acyclical
rdd is ___ - recomputable - fault-tolerant -all - immutable
all
how would you set the number of executors in any spark based application, assume we need to set 12 executors? --num-executors 12 can't change, it is always fixed conf.set(12) none of the above
chegg is saying none? (i don't trust them) internet saying first one.... although that's for the command line..
which of the following will return all the elements of the dataset as an array at the driver program? - first() - head() - collect() - none of the above
collect()
In mapreduce program the interface a combiner class will implement when the input is IntWritable keys, Text values, and emits IntWritable keys, IntWritable values. Which interface should your class implement - combiner - mapper - reducer - reducer <intWritable, Text, IntWritable, IntWritable>
combiner
which of the following application types can Spark run in addition to batch-processing jobs? all graph processing machine learning stream processin
graph processing
how can you set the reduce phase n MapReduce program as zero?
job.setNumreduceTasks(0)
in map-side join we will store the small file in the distributed cache and read it in the mapper class before the records are processed. Where do you implement the logic to read the file stored in distributed cache and store it into an associate array? combine map init setup
map
what will happen if the reducers are set to zero in the job configuration? - map only job takes place - reduce only job takes place - combiners output will be stored as final output in the hdfs - none of these
map only job takes place
which out of these is not optional in a MapReduce program? - mapper - combiner (optional) - reducer (optional) - none of these
mapper
which of these phases comes after the partitioner phase? -mapper phase - reducer phase - output - none
reducer
which of the following features of apache couchDB is an operation to avail extra disc space for the database by removing unused data? views replication ACID properties compaction
replication
In Spark Context, which all context are available by default (choose multiple)? - sparkcontext - sqlcontext - hivecontext - Streaming context
sparkContext, sqlContext, hiveContext
in scala which of the following would be used to specify a User Defined Function (UDF) that can be used in a SQL statement on Apache Spark DataFrames? registerUDF("func name", func def) sqlContext.udf(function definition) udf((arguments)=>{function definition}) sqlContext.udf.register("func name", func def")
sqlContext.udf.register("func name", func def") docs.databricks.com/spark/latest/spark-sql/udf-scala.html
the data types of the mapper for a product analytics job where the input key is month of data type Intwritable and input values represent items of data typeText, is configured at
the InputFormat used by the job determines the mapper's input key and value types
which of the following is true of running a Spark application on Hadoop YARN? - in Hadoop YARN mode, the RDDs and variables are always in the same memory space - there are two deploy modes that can be used to launch spark application on YARN- client mode and cluster mode - irrespective of the mode, the driver is launched in the client process that submitted the job - running in hadoop YARN has the advantage of having multiple users running the spark interactive shell
there are two deploy modes that can be used to launch spark application on YARN- client mode and cluster mode
when building a standalone application, you need to create the SparkContext. To do this in Scala, you would include which of the following within the main method? val conf=new SparkConf().setAppName("AuctionsApp") val sc= new SparkContext(conf) val sc = SparkContext().setAppName("AuctionsApp"_ val conf= SparkConf(sc) val sc=new SparkContext() val conf= new SparkConf().setAppName("AuctionsApp")
val conf=new SparkConf().setAppName("AuctionsApp") val sc= new SparkContext(conf)
which of the file contains the configuration setting for HDFS daemons? yarn-hdfs.xml hdfs-site.xml mapred-hdfs-site.xml all
hdfs-site.xml
what is RDD Lineage - it is a process that reconstructs lost data partitions - it gives information about the integrity of the input source - it gives information about the schema of the data - it is used for data replication
it is a process that reconstructs lost data partitions (edureke user) chegg says it gives information about the integrity of the input source
why is partition used in hive 1. hive distributes execution load horizontally 2. hadoop specific compression 3. join two tables (not this one) 4. bucketing purpose (not this)
- it's where hive divides a table into parts based on a particular column - stores data in slices, so query response time becomes faster - helps in elimination of data - a partition is created when data is inserted into table - increase db manageability - partitions are horizontal slices of data hive distributes execution load horizontally
which of the following is true about Spark application running on a Hadoop YARN? - yarn provide the facility of having multiple users use the spark interactive shell simultaneously yarn provide the facility of having multiple users use the spark interactive shell simultaneously - in hadoop yarn mode, the RDDs and variables are always in the same memory space - yarn support launching of spark app in client mode and cluster mode deployment - yarn launches the driver in client process where the job is submitted The client mode and cluster mode are the two deploy modes that can be used to launch Spark applications on YARN. In cluster mode, the Spark driver runs inside an application master process which is organized by YARN on the cluster, and the client can go away after commencing the application. In client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.
- yarn support launching of spark app in client mode and cluster mode deployment ??
to launch a spark application in any one of the four modes (local, standalone, Mesos or YARN), use ___ all ./bin/spark-submit ./bin/SparkContext ./bin/submit-app
./bin/spark-submit
The HDFS block size configuration is 128MB in your hadoop cluster. The HDFS directory contains 50 small files each of 200 MB in size. How many map tasks will be created when the inputformat for your job is TextInputFormat? 100 128 50 200
100
which of the following is not a type of counter in MapReduce? 1. job counters (is one) 2. user defined counters (is one) 3. filesystem counters (is one) 4. execute counter
4. execute counter
in a spark program, there are two datasets on weather. First one the weather records has the number of temperature recording of every major place in the country. AverageTemperatureRecords dataset has the details corresponding to average temperature of each place. Also note the exception that all places do not have average temperature weather record in the AverageTemperatureRecords dataset. What would we use to get all the current temperatures and average temperatures of all the places? D1.join(D2) D2.join(D1) D1.leftOuterJoin(D2) D2.leftOuterJoin(D1)
D1.join(D2) ?? inner join (join two sets on key cols, and where keys don't match the rows get dropped from both sets): D1.join(D2, D1("jfieie") === D2("jfei"), "inner") full outer (returns all rows from both sets, where expression doesn't match it returns null on record column): D1.join(D2, D1("id") === D2("id), "outer"')
which of the following is a column-oriented database that runs on top of hdfs - hbase - hive - sqoop - flume
HBase
where does the output of a reducer get stored - hdfs - local system - both - none
HDFS
an instance of the spark sql execution engine that integrates with the data stored in Hive is _____ - sparkHiveConnector - HiveLoader - HiveContext - HiveSparkConnector
HiveContext
the data types of the Mapper for a product analytics job where the input key is month of data type IntWritable and input values represent items of data type Text, is configured at
The inputFormat used by the job determines the mapper's input key and value types
spark can integrate with which of the following data storage systems? - hive - google cloud - all - cassandra
all
the type(s) of operation(s) that can be performed on RDDs is/are ___ all map action transformation action and map
all
what are the benefit(s) of using appropriate file formats in spark? - faster accessing during read and write - all - schema oriented - more compression support
all
what file systems does Apache Spark support? HDFS local system amazon S3 all
all
what kind of data can be handled by spark? semi-structured unstructured structured all
all
which of the following file formats is/are supported by spark? - json - sequence file -csv -all -parquet
all
what is replication factor? - replication factor controls how many times the Name node replicates the meta data - replication factor creates multiple copies of the same file to be served to the clients - replication factor controls how many times each individual block can be replicated - all
all explanation: .The number of nodes where data (rows and partitions) is replicated is equal to the Replication Factor (RF). Data is duplicated over many nodes (RF=N).There is only one copy of a row in a cluster with an RF of one, and there is no way to retrieve the data if the node is hacked or fails. A cluster with RF=2 has two copies of the same row. In most systems, an RF of at least three is utilized.Data is duplicated automatically at all times. Data stored on any of the replicated nodes can be read or written to.
Which of the following is true of caching the RDD?
all - rdd.persist(MEMORY_ONLY) is the same as rdd.cache() - use rdd.cache() to cache the RDD - cache behavior depends on available memory. If there is not enough memory, action will reload from the file instead of the cache
spark can store its data in ___ - all - mongoDB - cassandra - hdfs
hdfs
the following are characteristics shared by hadoop and spark, except ____ - both use open source api's to link between different tools - both are data processing platforms -both have their own file system -both are cluster computing environments
both have their own file system
choose the correct statement: - execution starts with the call of transformation - execution starts with the call of action - all the transformations and actions are lazily evaluated - none
execution starts with the call of action
spark is 100x faster than MapReduce due to development in scala
false
we can edit the data of rdd like conversion to uppercase
false
the client reading the data from HDFS fileSystem in Hadoop does which of the following? gets the data from the namenode gets both the data and block location from the namenode gets the block location from the datanode gets only the block locations from the namenode
gets only the block locations from the namenode
on what basis does partitioner groups the output and sent to the next stage? - on the basis of Key - on the basis of Value - both of these - none of these
key some chegg answer is saying both...
how is versioning information added to data in hbase? - VersionNo - Key Value - KeyNo - VersionValue
keyValue
in reducers the input received after the sort and shuffle phase of the mapreduce will be
keys are presented to reducer in sorted order; values for given key are not sorted
in map reduce job in which point the reducer class reduce method will be invoked? - as soon as at least one mapper has finished processing its input split - as soon as a mapper has emitted at least one record - not until all mappers have finished processing all records - it depends on the inputFormat used for the job
not until all mappers have finished processing all records
find the appropriate Scala statement to load the Transaction file into an RDD. Assume that SparkContext is available as "scCntxt" and SQLContext as "sqlCntxt"? val inData=sqlContext.loadText("/path to file/txn.txt") val inData=sc.loadFile("/path to file/txn.txt") val inData=sc.textFile("/path to file/txn.txt") val inData=sc.loadText("/path to file/txn.txt")
val inData=sc.textFile("/path to file/txn.txt")
Which of the following Scala statements would be most appropriate to load the data (sfpd.txt) into an RDD? Assume that SparkContext is available as the variable sc and SQLContext as the variable sqlContext.
val sfpd = sc.textFile("/path to file/sfpd.txt")
