Kedro newsletter — 5 min read

In the pipeline: September 2023

"In the Pipeline" is overflowing with the September's Kedro news.

6 Sept 2023 (last updated 6 Sept 2023)
In the pipeline

This month: a roundup of the summer’s Kedro news, some release updates, and our top picks from recent articles.

Kedro team news

Over the last few months, we’ve been happy to welcome some new team members to the Kedro and Kedro-Viz teams, who have also joined our Technical Steering Committee. Welcome Dmitry SorokinJitendra GundaniyaLaura CoutoRavi Kumar Pilla, and Vladimir Nikolic!

We are also pleased to announce a Kedro baby, delivered safely by one of the team, at the end of July!

Contributors news

We reworked the Kedro contributors guide in August, and moved it to the Kedro wiki. There are loads of different ways to contribute to Kedro and if you want to get involved, we encourage you to look at the table that introduces the Kedro contributor guide.

These are some of the ways to contribute to Kedro

If you spot an article, podcast or video that discusses Kedro, you can also contribute by adding it to the “Awesome Kedro” repository, or letting us know on Slack.

There have been some amazing contributions in recent weeks, including the kedro-vineyard plugin for efficient intermediate sharing in Kedro pipelines, kedro-graphql for serving Kedro projects as a GraphQL API, and kedro-pandera to bring data validation to your Kedro projects.

Release news

August 2023 saw a set of releases to introduce Python 3.11 support across Kedro, Kedro-Viz and Kedro datasets.

Created with imgflip

Kedro version 0.18.13 included these major features and improvements:

  • Added support for Python 3.11.

  • Added new OmegaConfigLoader features: registering of custom resolvers through CONFIG_LOADER_ARGS and support for global variables.

  • Added kedro catalog resolve CLI command that resolves dataset factories in the catalog with any explicit entries in the project pipeline.

  • Simplified the conf folder structure for modular pipelines and updated kedro pipeline create and kedro catalog create accordingly.

  • Made various updates to the Kedro project template and Kedro starters: use of OmegaConfigLoader, transition from setup.py to pyproject.toml, and updated for the simplified conf structure.

Kedro Viz version 6.5 added support for Python 3.11, while Kedro Viz version 6.4 added two new features: feature hint cards to highlight key features of Kedro Viz and support for displaying dataset statistics in the metadata panel for further investigation.

Kedro Datasets version 1.7.0 added polars.GenericDataSet, a dataset backed by polars, a lightning fast dataframe package built entirely using Rust. Kedro Datasets version 1.6.0 added support for Python 3.11.

Recently on the Kedro blog

In the last few weeks we’ve published the following on the Kedro blog:

We’re always looking for collaborators to write about their experiences using Kedro, particularly if you’re working with Kedro datasets or converting an existing project to use Kedro. Get in touch with us on our Slack workspace to tell us your story.

We’re always looking for collaborators to write about their experiences using Kedro, particularly if you’re working with Kedro datasets or converting an existing project to use Kedro. Get in touch with us on our Slack workspace to tell us your story.

Link to create your own badge
Get a "Powered by Kedro" badge

What we’ve learned

We really enjoyed reading more on Medium about the Kedro Vineyard plugin, which is a cloud-native data manager, for data sharing using memory in data science pipelines on Kubernetes.

Quix published an interesting article called “Bridging the gap between data scientists and engineers in machine learning workflows” which is something we regularly discuss within the team.

We found a super-interesting project about font recognition that uses Kedro.

And finally, we enjoyed reading more about data streaming with Kedro over on the QuantumBlack Medium channel.

That’s it for this edition!

And that’s a wrap for this month. But if you can’t wait for next month’s In the Pipeline news, we also toot out regular updates onto Mastodon (https://social.lfx.dev/@kedro) and across the popular channels of the Slack community. Keep an eye on the QuantumBlack LinkedIn feed too!

Don’t forget you can bookmark this blog or add our RSS feed to your favorite reader to stay in the loop and join us in October for another update from the Kedro team.


On this page:

Photo of Jo Stichbury
Jo Stichbury
Technical Writer, QuantumBlack
Share post:
Mastodon logoLinkedIn logo

All blog posts

cover image alt

Databricks — 8 min read

How to integrate Kedro and Databricks Connect

In this blog post, Diego Lira explains how to use Databricks Connect with Kedro for a development experience that works completely inside an IDE.

Diego Lira

11 Aug 2023

cover image alt

Kedro newsletter — 5 min read

In the Pipeline: July 2023

"In the Pipeline" is overflowing with the latest Kedro news, upcoming events, and interesting topics.

Jo Stichbury

12 Jul 2023

cover image alt

Datasets — 10 min read

A new Kedro dataset for Spark Structured Streaming

This post illustrates the extensibility of Kedro with a new dataset for realtime data processing using Spark Structured Streaming.

Jo Stichbury

11 Jul 2023

cover image alt

Databricks, Datasets — 10 min read

How to use Databricks managed Delta tables in a Kedro project

This post explains how to use a newly-released dataset for managed Delta tables in Databricks within your Kedro project.

Jannic Holzer

5 Jul 2023

cover image alt

Pipelines — 5 min read

Get up to speed: how to build a custom Kedro runner

We explain how to write an efficient custom Kedro pipeline runner that continues executing a set of nodes after encountering non-catastrophic failure points.

Nok Lam Chan

21 Jun 2023