aws-emr

Spark 2.0 Python Machine Learning examples

Spark 2.0 Scala Machine Learning examples

⭐ CLI tool to launch Spark jobs on AWS EMR

AWS Auto Terminate Idle AWS EMR Clusters Framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.

Bits of code I use during live demos

EMR Cost Calculator

A collection of airflow sample workflows for data processing on aws

The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on writing pyspark code.

A Spark application, written in Python, to figure out strongly connected components with Bi-directional Label Propagation algorithm. This project implemented an 1.3GB Twitter network dataset on AWS EMR cluster.

Spark 2.0 R/SparkR Machine Learning examples

Create Data Lake on AWS S3 to store dimensional tables after processing data using Spark on AWS EMR cluster

A cookiecutter template for working with PySpark on AWS EMR

A large-scale data framework that will enable us to store and analyze financial market data and drive future predictions for investment.

Cloud-based AI / ML workflow and data application development framework

Lambda to start EMR and run a map reduce job

Generic python library that enables to provision emr clusters with yaml config files (Configuration as Code)

MapReduce Analysis on Amazon Food Review Dataset (Big-Data)

A Hadoop Map-Reduce job to process the DBLP dataset to produce a graph depicting which professors at the CS department of UIC have co-authored publications.

My AWS Playground

Code and documentation for the demonstration example of the real-time bushfire alerting with the Complex Event Processing (CEP) in Apache Flink on Amazon EMR and a simulated IoT sensor network as described on the AWS Big Data Blog: Real-time bushfire alerting with Complex Event Processing in Apache Flink on Amazon EMR and IoT sensor network.

A Spark application to process the DBLP dataset to find out the Page Rank of faculty at the UIC CS department based on their co-authorships on publications.

AWS EMR Docker integration

Projeto de processamento distribuído de dados utilizando Python, MRJob e AWS EMR

Working examples of Spark ML Pipeline and SMOTE algorithm for synthetic data augmentation

This repository will be used to understand data science and data engineering concepts

Build modern workflows with AWS MWAA, AWS Step Functions, AWS Glue, and AWS EMR

Data Lake hosted on the AWS EMR cluster with S3 buckets used as source and output storages. The analysis was done using AWS Athena.

Machine Learning on a Large 12 GB dataset with Pyspark on AWS EMR

AWS 및 AWS를 이용한 Data Lake 구성 이해

Data Analysis Exercise over Walmart Stock