Welcome to the new home for the Kedro blog! We’ve previously published on QuantumBlack’s Medium channel, but recent updates and improvements here on the Kedro website mean that we’re now able to bring you a blog for, and about, the open-source Kedro community.
We plan to publish a range of articles by contributors from within the team and beyond. If you’re a Kedroid with an idea for a post, please reach out to us using one of the channels on the Slack organisation, or raise an issue on GitHub.
Kedro releases
We last gave an update on Kedro in late 2022, when we described the features in Kedro version 0.18.4. Since then, we’ve released three additional non-breaking versions of Kedro in the 0.18.x series, with the goal of a regular release cadence at the end of most two-week development sprints.
Some of the highlights of our releases are described below along with links to the full release notes. For each of these releases there’s a straightforward upgrade path with pip or conda. For example, to upgrade to Kedro version 0.18.7 from version 0.18.4:
pip install kedro==0.18.7
or
conda install -c conda-forge kedro==0.18.7
We received many contributions to these new versions from our open-source community and want to thank every contributor for taking the time to extend and improve Kedro.
Kedro version 0.18.7
These are the headline changes (You can find all the details about the Kedro 0.18.7 release on GitHub):
We added new Kedro CLI command
kedro jupyter setup
to set up a Jupyter Kernel for Kedro that automatically loads the Kedro extension for ease of use.The
kedro package
command now includes the project configuration in a compressedtar.gz
file.We’ve added functionality to package and read your configuration as a compressed file. You can now use
OmegaConfigLoader
to load configuration from compressed files of zip or tar format. (This feature requiresfsspec>=2023.1.0
).In documentation news, we moved seamlessly from
kedro.readthedocs.io
to docs.kedro.org in this release. We also made some significant improvements to on-boarding documentation that covers setup for new Kedro users and major changes to the spaceflights tutorial to make it faster to work through. We think it’s a better read. Tell us if it’s not.
Kedro version 0.18.6
This was a small release to fix a bug introduced in Kedro 0.18.5 that was causing experiment tracking in Kedro-Viz to fail. You can find all the details about the release of Kedro version 0.18.6 on GitHub.
Kedro version 0.18.5
In February 2023, we released Kedro version 0.18.5, to introduce a brand new config loader powered by OmegaConf. You can now use the omegaconf
syntax with kedro run --param
.
We also added the following:
Some improvements to the
kedro run
command used in the CLI. One changes is to make it more consistent. The flags--node
,--tag
, and--load-version
are deprecated in favour of plural equivalents (--nodes
,--tags
, and--load-versions
) and will be removed in Kedro 0.19.0. An additional change means that you can filter and run nodes by node namespace using the--namespace
flag withkedro run
.There is now support for using generator functions as nodes, i.e. using yield instead of return.
We added a new node argument to all four dataset hooks
You can find all the details about the Kedro version 0.18.5 release on GitHub.
Kedro datasets releases
Kedro provides numerous different built-in datasets for various file types and file systems, to save you from having to write the logic for reading or writing data, including Pandas, Spark, Dask, NetworkX, Pickle, and more.
There have been several datasets contributed by community members over the past months which include the addition of:
snowflake.SnowparkTableDataSet
by Vladimir Filimonov and Heber Urdanetapolars.CSVDataSet
by Walber Moreira
As we mentioned in “Keeping up with Kedro”, Kedro version 0.19.0 will move Kedro’s datasets from the main framework project into a separate package called Kedro-Datasets.
Kedro-Viz releases
If you've not yet used it, Kedro-Viz is the interactive development tool for building data science pipelines with Kedro. It comes with an experiment tracking feature enabling you to view and compare different runs of your Kedro project. Check out the Kedro-Viz demo at demo.kedro.
We’ve made three releases of Kedro-Viz this year, plus a patch release. You can find further details of the Kedro-Viz releases on GitHub.
To get the latest release of Kedro-Viz, you can use pip:
pip install kedro-viz==6.0.0
or npm
npm install @quantumblack/kedro-viz@latest
Here’s a summary of what we’ve been working on:
Kedro-Viz version 6.0.0
In this release we bumped the major version to 6.0.0 because of a change in the frontend React code (we bumped the minimum version of React from 16.8.6 to 17.0.2). Additional changes include:
We added a change so you can now see a preview of your data in the metadata panel.
You can remove metrics plots from metadata panel and add links to the plots on experiment tracking.
You can also link plot and JSON dataset names from experiment tracking to the flowchart.
Kedro-Viz no longer depends on pandas or Plotly.
Kedro-Viz versions 5.3.0 and 5.2.0
We introduced a raft of updates to experiment tracking, the largest being the addition of time series & parallel coordinates metrics plots and delta values.
We’ve enabled the display of json objects with `react-json-viewer` in experiment tracking.
We added a feature to show/hide modular pipelines on the pipeline flowchart.
It’s now possible to retrieve and share URL parameters for each element/section in the flowchart.
We've recently published a blog post about experiment tracking to highlight the latest features and discuss what is coming next.
What's next for the Kedro projects?
We have a broad range of milestones for the Kedro framework that cover areas such as integration with Databricks, enhancements for Jupyter Notebook users and ongoing changes such as the transition of datasets into their own package.
On the to-do list for Kedro-Viz, we’ve included enhanced navigation between flowchart and experiment tracking and collaboration features within Kedro-Viz.
Stand by for a pair of virtual Kedro showcases on 5th April 2023 (9am BST and 4pm BST) to demonstrate some of the features added in the recent releases to the global community.
To suggest features to us, report bugs, or just see what we’re working on right now, visit the Kedro projects on GitHub. We welcome every contribution, large or small.