A new home for the Kedro blog and some recent releases

Welcome to the new home for the Kedro blog! We’ve previously published on QuantumBlack’s Medium channel, but recent updates and improvements here on the Kedro website mean that we’re now able to bring you a blog for, and about, the open-source Kedro community.

We plan to publish a range of articles by contributors from within the team and beyond. If you’re a Kedroid with an idea for a post, please reach out to us using one of the channels on the Slack organisation, or raise an issue on GitHub.

Kedro releases

We last gave an update on Kedro in late 2022, when we described the features in Kedro version 0.18.4. Since then, we’ve released three additional non-breaking versions of Kedro in the 0.18.x series, with the goal of a regular release cadence at the end of most two-week development sprints.

Some of the highlights of our releases are described below along with links to the full release notes. For each of these releases there’s a straightforward upgrade path with pip or conda. For example, to upgrade to Kedro version 0.18.7 from version 0.18.4:

pip install kedro==0.18.7

or

conda install -c conda-forge kedro==0.18.7

We received many contributions to these new versions from our open-source community and want to thank every contributor for taking the time to extend and improve Kedro.

Kedro version 0.18.7

These are the headline changes (You can find all the details about the Kedro 0.18.7 release on GitHub):

• We added new Kedro CLI command kedro jupyter setup to set up a Jupyter Kernel for Kedro that automatically loads the Kedro extension for ease of use.

• The kedro package command now includes the project configuration in a compressed tar.gz file.

• We’ve added functionality to package and read your configuration as a compressed file. You can now use OmegaConfigLoader to load configuration from compressed files of zip or tar format. (This feature requires fsspec>=2023.1.0).

• In documentation news, we moved seamlessly from kedro.readthedocs.io to docs.kedro.org in this release. We also made some significant improvements to on-boarding documentation that covers setup for new Kedro users and major changes to the spaceflights tutorial to make it faster to work through. We think it’s a better read. Tell us if it’s not.

Kedro version 0.18.6

This was a small release to fix a bug introduced in Kedro 0.18.5 that was causing experiment tracking in Kedro-Viz to fail. You can find all the details about the release of Kedro version 0.18.6 on GitHub.

Kedro version 0.18.5

In February 2023, we released Kedro version 0.18.5, to introduce a brand new config loader powered by OmegaConf. You can now use the omegaconf syntax with kedro run --param.

We also added the following:

• Some improvements to the kedro run command used in the CLI. One changes is to make it more consistent. The flags --node, --tag, and --load-version are deprecated in favour of plural equivalents (--nodes, --tags, and --load-versions) and will be removed in Kedro 0.19.0. An additional change means that you can filter and run nodes by node namespace using the --namespace flag with kedro run.

• There is now support for using generator functions as nodes, i.e. using yield instead of return.

• We added a new node argument to all four dataset hooks

You can find all the details about the Kedro version 0.18.5 release on GitHub.

Kedro datasets releases

Kedro provides numerous different built-in datasets for various file types and file systems, to save you from having to write the logic for reading or writing data, including Pandas, Spark, Dask, NetworkX, Pickle, and more.

There have been several datasets contributed by community members over the past months which include the addition of:

• snowflake.SnowparkTableDataSet by Vladimir Filimonov and Heber Urdaneta

• polars.CSVDataSet by Walber Moreira

As we mentioned in “Keeping up with Kedro”, Kedro version 0.19.0 will move Kedro’s datasets from the main framework project into a separate package called Kedro-Datasets.

Kedro-Viz releases

If you've not yet used it, Kedro-Viz is the interactive development tool for building data science pipelines with Kedro. It comes with an experiment tracking feature enabling you to view and compare different runs of your Kedro project. Check out the Kedro-Viz demo at demo.kedro.

We’ve made three releases of Kedro-Viz this year, plus a patch release. You can find further details of the Kedro-Viz releases on GitHub.

To get the latest release of Kedro-Viz, you can use pip:

pip install kedro-viz==6.0.0

or npm

npm install @quantumblack/kedro-viz@latest

Here’s a summary of what we’ve been working on:

Kedro-Viz version 6.0.0

In this release we bumped the major version to 6.0.0 because of a change in the frontend React code (we bumped the minimum version of React from 16.8.6 to 17.0.2). Additional changes include:

• We added a change so you can now see a preview of your data in the metadata panel.

• You can remove metrics plots from metadata panel and add links to the plots on experiment tracking.

• You can also link plot and JSON dataset names from experiment tracking to the flowchart.

• Kedro-Viz no longer depends on pandas or Plotly.

Kedro-Viz versions 5.3.0 and 5.2.0

We introduced a raft of updates to experiment tracking, the largest being the addition of time series & parallel coordinates metrics plots and delta values.

• We’ve enabled the display of json objects with `react-json-viewer` in experiment tracking.

• We added a feature to show/hide modular pipelines on the pipeline flowchart.

• It’s now possible to retrieve and share URL parameters for each element/section in the flowchart.

We've recently published a blog post about experiment tracking to highlight the latest features and discuss what is coming next.

What's next for the Kedro projects?

We have a broad range of milestones for the Kedro framework that cover areas such as integration with Databricks, enhancements for Jupyter Notebook users and ongoing changes such as the transition of datasets into their own package.

On the to-do list for Kedro-Viz, we’ve included enhanced navigation between flowchart and experiment tracking and collaboration features within Kedro-Viz.

Stand by for a pair of virtual Kedro showcases on 5th April 2023 (9am BST and 4pm BST) to demonstrate some of the features added in the recent releases to the global community.

To suggest features to us, report bugs, or just see what we’re working on right now, visit the Kedro projects on GitHub. We welcome every contribution, large or small.