July 26, 2020
I have been talking about distributed computing in Rust for a long time now. It is more than two and a half years since my Rust is for Big Data blog post where I first talked about the prototype I was working on at the time (which eventually became DataFusion and is now part of Apache Arrow).
May 05, 2020
TL;DR: Stop using musl and alpine for smaller docker images!
April 24, 2020
DataFusion is a Rust-native in-memory query engine, which is part of the Apache Arrow project.
April 18, 2020
In July 2019, I created Ballista as a small proof-of-concept of parallel query execution across a number of Rust executors running in Kubernetes. This PoC generated a good discussion on Hacker News, which I felt demonstrated that there is interest in a platform like this. Unfortunately, it was far from usable for anything real and lacked a well-designed architecture.
February 27, 2020
Over the past decade I’ve spent a fair bit of time either building query engines or building integrations with query engines so I decided to write an introductory book on the subject.
January 10, 2020
Many years ago I wrote a commercial product that could import a database schema and then generate source code based on the schema. There were many different use cases for this product and it could be used to generate simple Data Access Object (DAO) code or even to generate fully working (although very crude) web applications for data entry. Believe it or not, some companies have schemas with more than 500 tables, so tools like this can dramatically reduce development costs. This type of product isn’t very sexy or modern but it generated decent revenue for a side project at the time, and there are still valid use cases today for this type of tool.
November 07, 2019
This blog post is a response to the call for blogs from the Rust Core Team.
October 20, 2019
NOTE: benchmarks have now moved here.
September 22, 2019
DataFusion is an extensible query execution framework implemented in Rust that uses the Apache Arrow memory model, and is part of the Apache Arrow project.
August 31, 2019
Over the past couple days at work we started noticing Spark 2.3 and 2.4 jobs failing with a permissions error across multiple EKS clusters. Here is an example stack trace:
July 16, 2019
Eighteen months ago, I started the DataFusion project with the goal of building a distributed compute platform in Rust that could (eventually) rival Apache Spark. Unsurprisingly, this turned out to be an overly ambitious goal at the time and I fell short of achieving that. However, some very good things came out of this effort. We now have a Rust implementation of Apache Arrow with a growing community of committers, and DataFusion was donated to the Apache Arrow project as an in-memory query execution engine and is now starting to see some early adoption. I even saw the first DataFusion job listing recently, which shows that this effort is already having an impact on the industry.
April 28, 2019
Over the past couple weeks I’ve been working on a couple different efforts around parallel query execution with DataFusion:
April 20, 2019
I’m working on a design to create a physical execution plan for DataFusion (part of Apache Arrow) that will support parallel query execution across multiple threads.
April 02, 2019
DataFusion is an in-memory query engine for analytical queries, implemented in Rust, that uses Apache Arrow for the memory model.
February 05, 2019
I’m excited to announce that DataFusion has now been donated to the Apache Software Foundation as a Rust-native in-memory query engine for the Apache Arrow project.
January 21, 2019
DataFusion is an in-memory query engine implemented in Rust that uses Apache Arrow for the memory model.
November 04, 2018
Earlier this year I put a lot of time and energy into DataFusion with the goal of creating a platform somewhat like Apache Spark, but implemented in Rust, without all the inefficiencies of the JVM. This was quite the journey, and I learned a lot of positive things from this effort, specifically:
May 19, 2018
After the recent fiasco where I moved my web sites from Google Cloud to GitHub Pages and then found out that it wasn’t possible to use a custom domain with SSL with both an apex domain and a www subdomain, I decided to move the sites again this weekend, this time to AWS.
May 15, 2018
Recently I’ve been working on improving the support for aggregate queries in DataFusion. I’m currently using NYC Taxi Trip Record Data for testing.
May 15, 2018
I was excited to see the recent tweet from GitHub just a couple of weeks ago, announcing:
May 04, 2018
I am currently working on a refactor of the Rust implementation of Apache Arrow to change the way that arrays are represented. This is a relatively large change even though this is a tiny codebase so far and I thought it would be good to write up this blog post to explain why I think this is needed. I think this information will also be interesting for any Rust developer who is struggling with making the right choice between (or using the right combination of) enums, structs, generics and traits. I was inspired to write this up after reading this blog post that was posted to Reddit just a few days ago.
April 15, 2018
There are a few interesting things going on with DataFusion that I wanted to share.
April 05, 2018
I’m excited to announce that DataFusion is now using Apache Arrow for its internal memory representation of data. It was already using columnar data structures based on Vec<T> and moving to Arrow was not that big a leap.
April 02, 2018
At the start of the year, I set myself some goals for Q1.
March 17, 2018
Over the past week or so I have been refactoring the core of DataFusion to convert it from a row-based execution engine to perform column-based processing. This was a pretty large refactoring effort but I am now back to roughly the same level of functionality as before (which is definitely still POC but capable of running some real queries).
February 18, 2018
I had limited time to work on DataFusion this weekend but have started to refactor the code base based on some feedback that I received on Reddit last week and have also been working on some benchmarks.
February 11, 2018
Following on from my blog post Rust is for Big Data, I announced my open source distributed data processing project, DataFusion, on reddit last week. I was probably a bit premature in announcing the project since it was at such an early stage but I was excited that I had some simple queries working with quite decent performance (roughly 2x the performance of Apache Spark) and wanted to start generating some interest in the project.
January 31, 2018
I’m reading the following two books at the moment and both are helping me towards my 2018 goals.
January 28, 2018
This blog post isn’t so much about what I want from the Rust language in 2018, but more about where I see an opportunity for Rust to gain more widespread use in 2018.
January 25, 2018
Instead of making new year’s resolutions, I’ve set myself some fairly specific goals for 2018 relating to family, health, finances, hobbies, and career. Many of these goals are broken down into monthly and quarterly objectives and some even have specific objectives and measurable results in true OKR style.
January 25, 2018
This is my “Hello, World” blog post.