PyTorch :: Performing Derivatives on Tensors

Deep Learning with PyTorch (4 Part Series)

1 PyTorch :: Course Introduction
2 PyTorch :: Understanding Tensors (Part 1)
3 PyTorch :: Understanding Tensors (Part 2)
4 PyTorch :: Performing Derivatives on Tensors

Hello everyone,

Especially to those who fear from maths 😛

In this you will learn how to perform derivatives on tensors using pytorch. Let me tell you, once you will learn, it will be fun for you.

Basically the derivatives are used for generating the parameters in the neural network. For now you will learn the basics of derivatives.

Suppose you have a function f ( x ) = x 2 f(x) = x^2 f(x)=x2 , so according to calculus, the derivative of f ( x ) f(x) f(x) would be d f ( x ) x = 2 x 1 \frac{d f(x)}{x} = 2x^1 xdf(x)=2x1

Let’s see how can we do this and find the value of where $x=9$


<span>import</span> <span>torch</span>
<span>x</span> <span>=</span> <span>torch</span><span>.</span><span>tensor</span><span>(</span><span>9.0</span><span>,</span> <span>requires_grad</span><span>=</span><span>True</span><span>)</span>
<span>print</span><span>(</span><span>x</span><span>)</span>
<span>import</span> <span>torch</span>

<span>x</span> <span>=</span> <span>torch</span><span>.</span><span>tensor</span><span>(</span><span>9.0</span><span>,</span> <span>requires_grad</span><span>=</span><span>True</span><span>)</span>
<span>print</span><span>(</span><span>x</span><span>)</span>
import torch

x = torch.tensor(9.0, requires_grad=True)
print(x)

Enter fullscreen mode Exit fullscreen mode


tensor(9., requires_grad=True)
tensor(9., requires_grad=True)
tensor(9., requires_grad=True)

Enter fullscreen mode Exit fullscreen mode

This time I am setting requires_grad=True, this means we are using pytorch autograd engine to compute the derivatives. Derivatives are calculated by tracing the graph (enabled when requires_grad=True) from root to leaf by using chain rule (back propagation).

Note Only Tensors of floating point dtype can require gradients

For more information you can read this wonderful post: https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95

Also you can read the official documentation regarding autograd on pytorch’s website: https://pytorch.org/docs/stable/notes/autograd.html

Moving on, let’s calculate x 2 x^2 x2


<span>y</span> <span>=</span> <span>x</span><span>**</span><span>2</span>
<span>print</span><span>(</span><span>y</span><span>)</span>
<span>y</span> <span>=</span> <span>x</span><span>**</span><span>2</span>
<span>print</span><span>(</span><span>y</span><span>)</span>
y = x**2
print(y)

Enter fullscreen mode Exit fullscreen mode


tensor(81., grad_fn=<PowBackward0>)
tensor(81., grad_fn=<PowBackward0>)
tensor(81., grad_fn=<PowBackward0>)

Enter fullscreen mode Exit fullscreen mode

If you would see, this time the pytorch has attached a gradient function with the tensor.

To calculate the derivative, you need to call Tensor.backward() method.

And use .grad property on the leaf tensor (here it’s $x$)


<span>y</span><span>.</span><span>backward</span><span>()</span>
<span>print</span><span>(</span><span>x</span><span>.</span><span>grad</span><span>)</span>
<span>y</span><span>.</span><span>backward</span><span>()</span>
<span>print</span><span>(</span><span>x</span><span>.</span><span>grad</span><span>)</span>
y.backward()
print(x.grad)

Enter fullscreen mode Exit fullscreen mode


tensor(18.)
tensor(18.)
tensor(18.)

Enter fullscreen mode Exit fullscreen mode

Isn’t it look like a magic to you? Let’s calculate the derivative of another complex equation.

f ( x ) = sin ⁡ ( x 2 2 ) f(x) = \sin ( \frac{x}{2 \sqrt 2} ) f(x)=sin(22 x)

Where x = 100 x = 100 x=100

If you would simplify the denominator 2 2 2 \sqrt 2 22 , it would be 2 3 2 2^{\frac{3}{2}} 223


<span>x</span> <span>=</span> <span>torch</span><span>.</span><span>tensor</span><span>(</span><span>100.</span><span>,</span> <span>requires_grad</span><span>=</span><span>True</span><span>)</span>
<span>y</span> <span>=</span> <span>torch</span><span>.</span><span>sin</span><span>(</span><span>x</span> <span>/</span> <span>(</span><span>2</span> <span>**</span> <span>1.5</span><span>)</span> <span>)</span>
<span>print</span><span>(</span><span>x</span><span>)</span>
<span>print</span><span>(</span><span>y</span><span>)</span>
<span>y</span><span>.</span><span>backward</span><span>()</span>
<span>print</span><span>(</span><span>x</span><span>.</span><span>grad</span><span>)</span>
<span>x</span> <span>=</span> <span>torch</span><span>.</span><span>tensor</span><span>(</span><span>100.</span><span>,</span> <span>requires_grad</span><span>=</span><span>True</span><span>)</span>

<span>y</span> <span>=</span> <span>torch</span><span>.</span><span>sin</span><span>(</span><span>x</span> <span>/</span> <span>(</span><span>2</span> <span>**</span> <span>1.5</span><span>)</span> <span>)</span>

<span>print</span><span>(</span><span>x</span><span>)</span>
<span>print</span><span>(</span><span>y</span><span>)</span>

<span>y</span><span>.</span><span>backward</span><span>()</span>
<span>print</span><span>(</span><span>x</span><span>.</span><span>grad</span><span>)</span>
x = torch.tensor(100., requires_grad=True)

y = torch.sin(x / (2 ** 1.5) )

print(x)
print(y)

y.backward()
print(x.grad)

Enter fullscreen mode Exit fullscreen mode


tensor(100., requires_grad=True)
tensor(-0.7158, grad_fn=<SinBackward>)
tensor(-0.2469)
tensor(100., requires_grad=True)
tensor(-0.7158, grad_fn=<SinBackward>)
tensor(-0.2469)
tensor(100., requires_grad=True)
tensor(-0.7158, grad_fn=<SinBackward>)
tensor(-0.2469)

Enter fullscreen mode Exit fullscreen mode

If you want to test it, you can go to https://www.derivative-calculator.net/ and type in the above expression.

You can also calculate the partial derivatives in pytorch. For example you have f ( u , v ) = u v + u 2 + v 3 f(u,v) = uv + u^2 + v^3 f(u,v)=uv+u2+v3 , the derivative d f ( u , v ) d u = 2 u + v \frac{d f(u,v)}{du} = 2u + v dudf(u,v)=2u+v and d f ( u , v ) d v = 3 v 2 + u \frac{d f(u,v)}{dv} = 3v^2 + u dvdf(u,v)=3v2+u

Let’s calculate the partial derivative using pytorch where u = 2 , v = 4 u = 2, v = 4 u=2,v=4


<span>u</span> <span>=</span> <span>torch</span><span>.</span><span>tensor</span><span>(</span><span>2.</span><span>,</span> <span>requires_grad</span><span>=</span><span>True</span><span>)</span>
<span>v</span> <span>=</span> <span>torch</span><span>.</span><span>tensor</span><span>(</span><span>4.</span><span>,</span> <span>requires_grad</span><span>=</span><span>True</span><span>)</span>
<span>f</span> <span>=</span> <span>u</span><span>**</span><span>2</span> <span>+</span> <span>v</span><span>**</span><span>3</span> <span>+</span> <span>u</span><span>*</span><span>v</span>
<span>print</span><span>(</span><span>u</span><span>)</span>
<span>print</span><span>(</span><span>v</span><span>)</span>
<span>print</span><span>(</span><span>f</span><span>)</span>
<span>f</span><span>.</span><span>backward</span><span>()</span>
<span>print</span><span>(</span><span>u</span><span>.</span><span>grad</span><span>)</span>
<span>print</span><span>(</span><span>v</span><span>.</span><span>grad</span><span>)</span>
<span>u</span> <span>=</span> <span>torch</span><span>.</span><span>tensor</span><span>(</span><span>2.</span><span>,</span> <span>requires_grad</span><span>=</span><span>True</span><span>)</span>
<span>v</span> <span>=</span> <span>torch</span><span>.</span><span>tensor</span><span>(</span><span>4.</span><span>,</span> <span>requires_grad</span><span>=</span><span>True</span><span>)</span>

<span>f</span> <span>=</span> <span>u</span><span>**</span><span>2</span> <span>+</span> <span>v</span><span>**</span><span>3</span> <span>+</span> <span>u</span><span>*</span><span>v</span>

<span>print</span><span>(</span><span>u</span><span>)</span>
<span>print</span><span>(</span><span>v</span><span>)</span>
<span>print</span><span>(</span><span>f</span><span>)</span>

<span>f</span><span>.</span><span>backward</span><span>()</span>
<span>print</span><span>(</span><span>u</span><span>.</span><span>grad</span><span>)</span>
<span>print</span><span>(</span><span>v</span><span>.</span><span>grad</span><span>)</span>
u = torch.tensor(2., requires_grad=True)
v = torch.tensor(4., requires_grad=True)

f = u**2 + v**3 + u*v

print(u)
print(v)
print(f)

f.backward()
print(u.grad)
print(v.grad)

Enter fullscreen mode Exit fullscreen mode


tensor(2., requires_grad=True)
tensor(4., requires_grad=True)
tensor(76., grad_fn=<AddBackward0>)
tensor(8.)
tensor(50.)
tensor(2., requires_grad=True)
tensor(4., requires_grad=True)
tensor(76., grad_fn=<AddBackward0>)
tensor(8.)
tensor(50.)
tensor(2., requires_grad=True)
tensor(4., requires_grad=True)
tensor(76., grad_fn=<AddBackward0>)
tensor(8.)
tensor(50.)

Enter fullscreen mode Exit fullscreen mode

If you have doubts or any idea reach me out via following sources

Email: tbhaxor@gmail.com (Recommended)
Twitter: @tbhaxor
GitHub: @tbhaxor
LinkedIn: @gurkirat–singh
Instagram: @_tbhaxor_

Deep Learning with PyTorch (4 Part Series)

1 PyTorch :: Course Introduction
2 PyTorch :: Understanding Tensors (Part 1)
3 PyTorch :: Understanding Tensors (Part 2)
4 PyTorch :: Performing Derivatives on Tensors

原文链接：PyTorch :: Performing Derivatives on Tensors

展开阅读全文

文章版权声明 1、本网站名称：拾光赋
2、本站永久网址：https://www.blogs.ink
3、本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ：805375623进行删除处理。
4、本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5、本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6、本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。

THE END