It's out of the pipeline! Xplenty's inaugural World Data Pipeline Championship has crowned the world's most incredibly creative data engineers of 2021 — and the talent was astonishing! The winner of this year's competition, whom you'll find out shortly, scooped $5,000 in cash, and there was free swag for the four astounding runners-up. (Xplenty handed out $10,000 in total prizes.) 

A huge, huge thanks to everyone who entered!

The premise was super simple: Use your hard-earned ETL skills to create the most innovative data pipeline or dataset using just Xplenty, its API partners, and publicly available databases. The incredible Xplenty community voted on all submissions received from around the world. Then a panel of senior data engineers from multiple companies — including AWSSlackNBC Universal, and, of course, Xplenty! — picked the cream of the crop.

Congrats to everyone who got involved! Up for the next challenge? Enter Xplenty's competition next year!

Check out this year's champions below.

Enjoying This Article?

Receive great content weekly with the Xplenty Newsletter!

Octopus

Table of Contents

The Rules

Xplenty's World Data Pipeline Championship 2021 was made possible by its amazing community of data professionals who use the platform to create and innovate incredible data pipelines and datasets. In the week leading up to the tournament, Xplenty provided engineers and future pipeline-builders with training and gave entrants access to its proprietary API partners on the day of the competition. 

The Xplenty community reviewed submissions and voted for the most interesting data pipeline or dataset before narrowing entries to five finalists. Then Xplenty's panel of judges reviewed the finalists and picked the winner. Before you find out about the winner, why don't you pick your favorite data pipeline concept below!

Read more: How Do Data Pipelines Fit Into Your Data Stack?

Fish News Network (FNN) - WINNER?

Imagine a news network for salmon. Or tuna. That's the idea behind this insane data pipeline that pulls headlines from CNN and rungs a regex on specific terms for an imaginary audience of fish. You're now watching Fish News Network. Or FNN. 

Mix Max, the inventive developer behind this pipeline, replaced words in CNN headlines — "animal" to "amphibian," "labrador retriever" to "rainbow trout," etc. — then ran each sentence through the fake online RESTful API Dummy API, replaced the sentences, and swapped photos with pictures of fish:

"What I ended up with was a simple pull from Google Sheets to transform and then a new table creation in Snowflake."

*Insert screenshot*

Integrate Your Data Today!

Try Xplenty free for 14 days. No credit card required.

Octopus

IP Geolocation/Azure Blob 

One of the most creative entries this year came from Sunny J, who extracted information from the API IPGeolocation, counted the countries that used IPs on the interface and wrote a Parquet file to Azure Blob.

This pipeline might look simple, but it's ingenious, and the judges loved it.

"I fetched the data from IPGeolocation and parsed it through the REST API stage in the framework," says Sunny.

*Insert screenshot*

Image Tagging 

What if you could organize metadata from an image repository on something like Google Photos and find your favorite images quicker? That's what one developer did by blending machine learning with SilverDiamond APIImagga API, and Xplenty. 

Here's the process: Developer CarlosJaime got a list of random images from Picsum API, added metadata to each using the APIs Imagga and SilverDiamond, and added alternative descriptions for those images for additional context. Finally, the developer parsed the data returned by the APIs and inserted it into the "tagged_images" table on BigQuery.

"I've created a process that gives more value to image metadata by adding tags and generating a data asset that helps analysts and technicians decide which images to use based on tag searches." 

Read more: Google BigQuery vs. Snowflake.

*Insert screenshot*

Just Breathe 

This ETL pipeline helps people visualize air pollution around the world. It generates dummy profiles from DummyAPI and extracts valuable location info by feeding queries into multiple APIs like AmbeeOpenTopoData, and TimeZoneDB. Then local environmental factors such as nitrogen dioxide and air quality indexes feed into the web application Pixe. Ultimately, users can receive notifications about the weather via Twilio SendGrid

"Against the backdrop of increasing air pollution around the world, I wanted to create an application that helps people visualize air pollution and raise awareness of a seemingly invisible growing threat," said developer Lloyd H.

Read more: The Importance and Benefits of a Data Pipeline

*Insert screenshot*

Crypto Data Warehouse 

This ETL pipeline aggregates cryptocurrency data to build a crypto data warehouse. Its developer Sandeep M. retrieved datasets from Coinranking and ExchangeRate and ran them through Xplenty and Google Cloud Storage. Seriously imaginative!

"The purpose of the pipeline is to be run once every 24 hours," says Sandeep. "Every day, it retrieves the latest Bitcoin price and other values. I used Google Cloud for everything."

*Insert screenshot*

The Winners

1st place: Lloyd Hamilton, Just Breathe

2nd place: Carlos Jaime, Imaging Tagging

3rd place: Sandeep Mysore, Crypto Data Warehouse

Community favorite: Lloyd Hamilton, Just Breathe

Want to Build a Pipeline of Your Own? How Xplenty Can Help 

Enjoying This Article?

Receive great content weekly with the Xplenty Newsletter!

Octopus

Think you can do better than this year's winner? There's always next year! In the meantime, create compelling data pipelines and datasets with Xplenty — the no-code data management solution that moves data from disparate sources, transforms it into usable formats, and loads it to a final destination for data analysis and business intelligence

Xplenty point-and-click interface and easy-to-use pipeline builder means you can integrate data from one location to another with no data engineering experience. So you can get creative without the complicated coding. Other Xplenty features include a simple pricing structure, top-tier customer service, and pre-built connectors for improved data migration.  

Xplenty has crowned the planet's most creative data engineers. Now it's your turn to create a data pipeline with little or no code. Discover how Xplenty can help data integration in your organization with this personalized 7-day demo today.