spark-monitoring

A python library to interact with the Spark History server.

Quickstart

Basic

$ pip install spark-monitoring

import sparkmonitoring as sparkmon

monitoring = sparkmon.client('my.history.server')
print(monitoring.list_applications())

Pandas

$ pip install spark-monitoring[pandas]

import sparkmonitoring as sparkmon
import matplotlib.pyplot as plt

monitoring = sparkmon.df('my.history.server')

apps = monitoring.list_applications()
apps['function'] = apps.name.str.split('(').str.get(0)
print(apps.head().stack())

plt.figure()
apps['duration'].hist(by=apps['function'], figsize=(40, 20))
plt.show()

jobs = monitoring.list_jobs(apps.iloc[0].id)

print(jobs.head().stack())

Proxies

If your Spark history server is behind a proxy, you can connect using the http_proxy and https_proxy environment variables.

GCP

To connect to dataproc History Server, create an SSH tunnel using a local port as described in Cluster web interfaces documentation and then use the environment variables as above.

Note: Use the IP for the cluster master node, as the google short names will not resolve through the socks proxy.

Example

gcloud compute ssh my-cluster-m \
  --project=my-project \
  --zone=us-east4-a -- -D 1080 -N

http_proxy="socks5://localhost:1080" \
    https_proxy="socks5://localhost:1080" \
    python my_spark_history_script.py

Reference

sparkmonitoring.client

Method to return a client to make calls to the spark history server with.

Arguments

Response

• sparkmonitoring.api.ClientV1

Examples

Basic Endpoint

import sparkmonitoring as sparkmon
client = sparkmon.client('my.history.server')

Custom Endpoint

import sparkmonitoring as sparkmon
client = sparkmon.client('my.history.server', port=8080, is_https=True)

sparkmonitoring.df

Method to return a client to make calls to the spark history server with. This client will return pandas dataframes, as opposed ot dictionaries in the standard client. Can be used when the spark-monitoring[pandas] extra is installed.

Arguments

Response

• sparkmonitoring.dataframes.PandasClient

Examples

Basic Endpoint

import sparkmonitoring as sparkmon
client = sparkmon.df('my.history.server')

Custom Endpoint

import sparkmonitoring as sparkmon
client = sparkmon.df('my.history.server', port=8080, is_https=True)

sparkmonitoring.api.ClientV1

A client to interact with the Spark History Server. Generally this class is not instantiated directly, and is accessed via sparkmonitoring.client(...).

Arguments

Methods

• list_applications(...)
• get_application(...)
• list_jobs(...)
• get_job(...)
• list_stages(...)
• list_stage_attempts(...)
• get_stage_attempt(...)
• get_stage_attempt_summary(...)
• get_stage_attempt_tasks(...)
• list_active_executors(...)
• list_executor_threads(...)
• list_all_executors(...)

sparkmonitoring.dataframes.PandasClient.list_applications

A list of all applications.

Arguments

sparkmonitoring.dataframes.PandasClient

A client to interact with the Spark History Server, returning pandas DataFrames. Generally this class is not instantiated directly, and is accessed via sparkmonitoring.df(...).

Arguments

Methods

• list_applications(...)
• get_application(...)
• list_jobs(...)
• get_job(...)
• list_stages(...)