Breaking Down the Beats: A Comprehensive Guide to Using ML Pipelines to Predict Song Release Years

Linear Regression is the most commonly used predictive analysis. It is used to model the relationship between a dependent variable and one or more independent variables. In this project we create an ML pipeline to train a linear regression model to predict the release year of a song given a set of audio features. We […]

Unlocking the Power of MapReduce: Using Python and Apache Spark for Enhanced Data Processing

Hey there! So we decided to create a Word Count application – a classic MapReduce example. But what the heck is a Word Count application, you ask? It’s basically a program that reads data and calculates the most common words. Easy peasy. For example: dataDF = sqlContext.createDataFrame([('Jax',), ('Rammus',), ('Zac',), ('Xin', ), ('Hecarim', ), ('Zac', ), […]

Data Wrangling Made Easy: Leveraging Apache Spark to Transform Raw Data into Valuable Insights

Hey, so I’m a big fan of Apache Spark and I’ve been using it for all of my independent projects. I recently had this idea to create a project that would showcase how to do some data wrangling with Apache Spark. For this project, we used Apache Spark 2.0.2 on Databricks cloud. Instead of using […]

Making Sense of Big Data: A Beginner’s Guide to Logistic Regression Training in SparkR

Hey there! As your friendly language model, I’m here to help proofread and rewrite your text! Here’s the corrected and rewritten version of your post: Let’s do some Machine Learning with SparkR 1.6! The package only gives us the option to do linear or logistic regression, so for this exercise, we’re going to train a […]

Empowering R Programmers: Exploring the Capabilities of SparkR with RStudio

Let’s talk about SparkR! It’s an R package that provides a lightweight frontend to use Apache Spark from R. I used RStudio and Spark 1.6.1 for this exercise. SparkR has a distributed data frame implementation that supports operations like selection, filtering, and more. Cool, right? In RStudio, run the following code to check the system […]