Lambda Function as a Kedro Node

til (81 Part Series)

1 fugitive verbose commit
2 Adding __rich__ methods to python classes
… 77 more parts…
3 Installing system nerd-fonts with Ansible
4 Finding hidden (dotfiles) using Telescope in neovim
5 Opening files in vim from output of command
6 Serve html from your command line
7 Installing Homebrew on Linux
8 List all the files containing a phrase | ag, rg
9 A Minimal Pluggy Example
10 Running Kedro on Ubuntu 21.10 Impish Indri
11 copier template variables
12 Changing copier template strings (_endops)
13 Tmux Pop size
14 Remove Vim Tab Characters
15 Installing Pipx on Ubuntu
16 Converting markdown to pdf with pandoc on linux
17 Implement –pdb in a python cli
18 Python Find Available Port
19 Rename Python Variables with nvim
20 Python Respect the .gitignore
21 fuzzy wallpaper with fzf
22 Gradual Typing in Python
23 Copier Tasks | Python templating post run task
24 Markata Filters as Telescope Pickers in Neovim
25 Tmux hotkey for copier templates
26 nix rescues modded minecraft night
27 Tmux and Vim Clipboard for Ubuntu
28 Add New Dependencies to Your Kedro Project
29 Squoosh cli
30 Lambda Function as a Kedro Node
31 Find all Headings with BeautifulSoup
32 Nix Versions By Commit Count
33 python dict get
34 Neovim Config for Git
35 Using a Python Markdown ast to Find All Paragraphs
36 Installing Rust and Cargo on Ubuntu 21.10 using Ansible
37 Running a Minecraft Server in Docker
38 ag silver searcher look ahead and look behind
39 Simple Samba Share Setup
40 Read stderr from python subprocess.Popen
41 Python string of letters is a string of letters, but not with special
42 A better copy-mode bind for Tmux
43 Vim remaps use cmd in place of :
44 View AWS event bridge rules with visidata
45 Nested requirements.txt in python
46 Run glances without install with pipx
47 Git reflog is an alias for git log -g
48 Glances webui with pipx
49 Revive files from the dead with git
50 Glances can watch docker processes
51 Git rebase to the beginning of time
52 GitHub Markdown now Supports Mermaid Diagrams
53 Install obs flatpak
54 Textual Popup Hack
55 git diff-filter
56 Recover a lost git branch with checkout
57 Bash mktemp
58 Simple Plain Text Diagrams in HTML
59 Dedupe your shell paths
60 Open Files with Nvim Remote
61 Setup SSH from chromebook to home desktop
62 Configure Git to Always Push to the Current Branch
63 Git Worktrees are not so Scary
64 Install yq | A light weight yaml parser cli
65 Configure Git to Autocorrect Your Fat Fingers
66 How I glob for Files in Python
67 Create a Swapfile on Your Linux Machine
68 Automatically Generate a list of Markown Links in Vim
69 How I read Files in Python
70 Python’s Dict Union Operator | Pep 584
71 How I load Markdown in Python
72 Style Lookatme Slides a bit more Personal
73 Ease into htmx with htmx-get
74 Get Webpage with python requests
75 Set User Agent on pandas read_csv
76 How I setup a sqlite cache in python
77 How I make cache-keys from python objects
78 Don’t inherit from python BaseException, Here’s why.
79 Did my site build just go down?
80 Unzip minecraft mods to their directory from the command line
81 Pygame Boilerplate Apr 2022

I keep my nodes short and sweet. They do one thing and do it well. I turn almost every DataFrame transformation into its own node. It makes it must easier to pull catalog entries, than firing up the pipeline, running it, and starting a debugger. For this reason many of my nodes can be built from inline lambdas.

Examples

Here are two examples, the first one lambda x: x is sometimes referred to as an identity function. This is super common to use in the early phases of a project. It lets you follow standard layering conventions, without skipping a layer, overthinking if you should have the layer or not, and leaves a good placholder to fill in later when you need it.

Many times I just want to get the data in as fast as possible, learn
about it, then go back and tidy it up.

from kedro.pipeline import node

my_first_node = node(
   func=lambda x: x,
   inputs='raw_cars',
   output='int_cars',
   tags=['int',]
   )

my_first_node = node(
   func=lambda cars: cars[['mpg', 'cyl', 'disp',]].query('disp>200'),
   inputs='raw_cars',
   output='int_cars',
   tags=['pri',]
   )

Enter fullscreen mode Exit fullscreen mode

Note: try not to take the idea of a one liner too far. If your
one line function wraps several lines down it probably deserves to be
a real function for readability and a good docstring.

til (81 Part Series)

1 fugitive verbose commit
2 Adding __rich__ methods to python classes
… 77 more parts…
3 Installing system nerd-fonts with Ansible
4 Finding hidden (dotfiles) using Telescope in neovim
5 Opening files in vim from output of command
6 Serve html from your command line
7 Installing Homebrew on Linux
8 List all the files containing a phrase | ag, rg
9 A Minimal Pluggy Example
10 Running Kedro on Ubuntu 21.10 Impish Indri
11 copier template variables
12 Changing copier template strings (_endops)
13 Tmux Pop size
14 Remove Vim Tab Characters
15 Installing Pipx on Ubuntu
16 Converting markdown to pdf with pandoc on linux
17 Implement –pdb in a python cli
18 Python Find Available Port
19 Rename Python Variables with nvim
20 Python Respect the .gitignore
21 fuzzy wallpaper with fzf
22 Gradual Typing in Python
23 Copier Tasks | Python templating post run task
24 Markata Filters as Telescope Pickers in Neovim
25 Tmux hotkey for copier templates
26 nix rescues modded minecraft night
27 Tmux and Vim Clipboard for Ubuntu
28 Add New Dependencies to Your Kedro Project
29 Squoosh cli
30 Lambda Function as a Kedro Node
31 Find all Headings with BeautifulSoup
32 Nix Versions By Commit Count
33 python dict get
34 Neovim Config for Git
35 Using a Python Markdown ast to Find All Paragraphs
36 Installing Rust and Cargo on Ubuntu 21.10 using Ansible
37 Running a Minecraft Server in Docker
38 ag silver searcher look ahead and look behind
39 Simple Samba Share Setup
40 Read stderr from python subprocess.Popen
41 Python string of letters is a string of letters, but not with special
42 A better copy-mode bind for Tmux
43 Vim remaps use cmd in place of :
44 View AWS event bridge rules with visidata
45 Nested requirements.txt in python
46 Run glances without install with pipx
47 Git reflog is an alias for git log -g
48 Glances webui with pipx
49 Revive files from the dead with git
50 Glances can watch docker processes
51 Git rebase to the beginning of time
52 GitHub Markdown now Supports Mermaid Diagrams
53 Install obs flatpak
54 Textual Popup Hack
55 git diff-filter
56 Recover a lost git branch with checkout
57 Bash mktemp
58 Simple Plain Text Diagrams in HTML
59 Dedupe your shell paths
60 Open Files with Nvim Remote
61 Setup SSH from chromebook to home desktop
62 Configure Git to Always Push to the Current Branch
63 Git Worktrees are not so Scary
64 Install yq | A light weight yaml parser cli
65 Configure Git to Autocorrect Your Fat Fingers
66 How I glob for Files in Python
67 Create a Swapfile on Your Linux Machine
68 Automatically Generate a list of Markown Links in Vim
69 How I read Files in Python
70 Python’s Dict Union Operator | Pep 584
71 How I load Markdown in Python
72 Style Lookatme Slides a bit more Personal
73 Ease into htmx with htmx-get
74 Get Webpage with python requests
75 Set User Agent on pandas read_csv
76 How I setup a sqlite cache in python
77 How I make cache-keys from python objects
78 Don’t inherit from python BaseException, Here’s why.
79 Did my site build just go down?
80 Unzip minecraft mods to their directory from the command line
81 Pygame Boilerplate Apr 2022

原文链接：Lambda Function as a Kedro Node