Spark Architeture

Spark Driver : 

It is the application which trigger the main Method in which the Instance of sparkContext is created. In simple , its a process that creates and  own an instance of Sparkcontext.

It is the Engine of train which holds the responsibility of jobs and task execution. It splits a Spark application into tasks and schedules them to run on executors. A driver is where the task scheduler lives and spawns tasks across workers. A driver coordinates workers and overall execution of tasks.

Master : 

A master is a running Spark instance that connects to a cluster manager for resources. The master  runs executors on cluster nodes. It’s a mediator between driver and cluster but not SC.

Spark Worker : 

Spark Workers or slaves are running Spark instances where executors live to execute tasks. They are the compute nodes in Spark. A worker receives serialized tasks that it runs in a thread pool.

Executors

Executors are distributed agents that execute tasks. They typically run for the entire lifetime of a Spark application. Executors send active task metrics to a driver and inform executor backends about task status updates (task results including).

Executors provide in-memory storage for RDDs that are cached in Spark applications. When executors are started they register themselves with the driver and communicate directly to execute tasks.

Executor offers are described by executor id and the host on which an executor runs.

Executors use a thread pool for sending metrics and launching tasks (using Task Runner) .Each executor can run multiple tasks over its lifetime, both in parallel and sequentially.It is recommended to have as many executors as data nodes and as many cores as you can get from the cluster.

Task Runner :

TaskRunner is a thread of execution that manages a single individual task.

It can be run or killed that boils down to running or killing the task the TaskRunner object manages.

A TaskRunner object is created when an executor is requested to launch a task. It is created with an ExecutorBackend (to send the task’s status updates to), task and attempt ids, task name, and serialized version of the task (as ByteBuffer).

Spark's Component

Spark Components

Spark Core

Spark Core contains the basic functionality of Spark, required for running jobs and needed by other Components which includes - components for task scheduling, memory management, fault recovery, interacting with storage systems.

Spark Core is also home to the API that defines resilient distributed datasets (RDDs), which are Spark’s main programming abstraction. RDDs represent a collection of items distributed across many compute nodes that can be manipulated in parallel.

Spark Core contains logic for accessing various file systems, such as HDFS, Gluster FS,Amazon S3 and so on.

Spark SQL

Spark SQL is Spark’s package for working with structured data. It allows querying data via SQL as well as the Apache Hive variant of SQL—called the Hive Query Language (HQL).

Spark SQL can also be used for reading and writing data to and from various structured formats and data sources, such as JSON, Parquet files, relational databases and Hive.

Spark SQL also includes a Thrift server, which can be used by external systems, such as business intelligence tools, to query data through Spark SQL using classic JDBC and ODBC protocols

Spark Streaming

Spark Streaming is a framework for ingesting and processing real-time streaming data from various sources. Examples of data streams include log files generated by production web servers, or queues of messages containing status updates posted by users of a web service.

Spark Streaming is a framework for ingesting real-time streaming data from various sources.

The supported streaming sources include HDFS, Kafka, Flume, Twitter, ZeroMQ, and custom

ones.

MLlib

Spark comes with a library containing common machine learning (ML) functionality, called MLlib. MLlib provides multiple types of machine learning algorithms, including classification, regression, clustering, and collaborative filtering, as well as supporting functionality such as model evaluation and data import.

GraphX

GraphX is a library for manipulating graphs (e.g., a social network’s friend graph) and performing graph-parallel computations.Graphs are data structures comprising vertices and the edges connecting them.

GraphX provides functions for building graphs, represented as "graph RDDs": EdgeRDD and VertexRDD. GraphX contains implementations of the most important algorithms of graph theory, such as page rank, connected components, shortest paths, SVD++, and others.

Cluster Managers

While maximizing flexibility, Spark can run over a variety of cluster managers, including Hadoop YARN, Apache Mesos, and a simple cluster manager included in Spark itself called the Standalone Scheduler.

What is Apache Spark

Spark :

Apache Spark is general-purpose cluster computing System to process big data at a very fast speed in comparison with traditional Map-Reduce. Spark is design uniquely, which allows it keeping large amounts of data in memory. Spark programs can be 100 times faster than their Map Reduce counterparts.

In Compare to Map reduce model , Spark support more types of computations , including interactive queries and streaming processing .The Biggest difference come up between map reduce and Spark is "Use of memory (RAM)".

In Map reduce, memory is primarily used to actual computation and it makes process very slow. While In Spark, memory is used for both purpose to compute and also to store objects.

Spark is like a shopping mall which have all layouts in one place .It means Spark is designed to cover all types of workloads that required separate distributed systems, including batch applications, iterative algorithms, interactive queries and Streaming.  Spark makes it easy and inexpensive to combine different processing types, which is often necessary in production data analysis pipelines.

Spark is written in Scala and it is designed to be highly accessible by Using Python, Java, Scala, and SQL.

There some application which are not suitable for Spark due to its distributed architecture. Spark’s overhead is negligible when handling large amounts of data, but if you have small amount of data which can be fit in single machine than it will more beneficial to go with some other framework .Also Spark was not made with OLTP application – which have fast and numerous transactions. It is better suited for OLAP – which have batch jobs and data mining.