Deep Learning with PyTorch-note

Youtube Video

Blog Post
DeepLizard 中文笔记

8. PyTorch Tensors Explained - Neural Network Programming

9. Creating PyTorch Tensors for Deep Learning - Best Options


10. Flatten, Reshape, and Squeeze Explained

11. CNN Flatten Operation Visualized - Tensor Batch Processing for Deep Learning

  • Flattening Specific Axes Of A Tensor

In the post on CNN input tensor shape, we learned how tensor inputs to a convolutional neural network typically have 4 axes, one for batch size, one for color channels, and one each for height and width.

Notice in the call how we specified the start_dim parameter. This tells the flatten() method which axis it should start the flatten operation. The one here is an index, so it’s the second axis which is the color channel axis. We skip over the batch axis so to speak, leaving it intact.

19. CNN Weights - Learnable Parameters in PyTorch Neural Networks

  • input is 4-dimension, output is 3-dimension, the shape of weight is 3*4.


25. CNN Output Size Formula - Bonus Neural Network Debugging Session

26. CNN Training with Code Example - Neural Network Programming Course

  • Training Using a Single Batch
    • Calculate Loss
    • Calculate Gradients
    • Update Weight

27. CNN Training Loop Explained - Neural Network Code Project

well explained what happened with each loop.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

import torchvision
import torchvision.transforms as transforms

torch.set_printoptions(linewidth=120)
torch.set_grad_enabled(True)

def get_num_correct(preds,labels):
    return preds.argmax(dim=1).eq(labels).sum().item()

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()

        # 维度计算公式:output=(n-filter+2padding)/stride+1。因此:
        # 原始图片大小:28*28
        # conv1后,28-5+1=24。当前图片大小:24*24
        # max_pool2d后,(24-2)/2+1=12。当前图片大小:12*12
        # conv2后,12-5+1=8。当前图片大小:8*8
        # max_pool2后,(8-2)/2+1=4。当前图片大小:4*4。当前这张图片有12g个channels
        # 此时,这张图片的维度为1*12*4*4(batch*channel*heigtht*width)。在进fc1时,把这张图片转为(1行,12*4*4列)维度

        self.conv1=nn.Conv2d(in_channels=1,out_channels=6,kernel_size=5)
        self.conv2=nn.Conv2d(in_channels=6,out_channels=12,kernel_size=5)

        self.fc1=nn.Linear(in_features=12*4*4,out_features=120)
        self.fc2=nn.Linear(in_features=120,out_features=60)
        self.out=nn.Linear(in_features=60,out_features=10) # 有10种输出类型

    def forward(self,t):
        t=F.relu(self.conv1(t));
        t=F.max_pool2d(t,kernel_size=2,stride=2)
        t=F.relu(self.conv2(t))
        t=F.max_pool2d(t,kernel_size=2,stride=2)

        t=t.reshape(-1,12*4*4)
        t=F.relu(self.fc1(t))
        t=F.relu(self.fc2(t))
        t=self.out(t)
        return t

train_set=torchvision.datasets.FashionMNIST(
    root='./data',
    train=True,
    download=True,
    transform=transforms.Compose([
        transforms.ToTensor()
    ])
)

network=Network()
train_loader=torch.utils.data.DataLoader(train_set,batch_size=100)
optimizer=optim.Adam(network.parameters(),lr=0.01)

for epoch in range(5): # epoch表示完整的一批数据
    total_loss = 0
    total_correct = 0
    for batch in train_loader: # batch表示某epoch中的一部分数据
        images,labels=batch
        preds=network(images)
        loss=F.cross_entropy(preds,labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        total_loss+=loss.item()
        total_correct+=get_num_correct(preds,labels)
    print("epoch: ",epoch,
          "total_loss: ",total_loss,
          "total_correct: ",total_correct)
print(total_correct/len(train_set))

# epoch:  0 total_loss:  358.2741909921169 total_correct:  46523
# epoch:  1 total_loss:  237.37768299877644 total_correct:  51164
# epoch:  2 total_loss:  216.7216858714819 total_correct:  51954
# epoch:  3 total_loss:  206.57599268853664 total_correct:  52318
# epoch:  4 total_loss:  200.8457220196724 total_correct:  52480
# 0.8746666666666667

29. Stack vs Concat in PyTorch, TensorFlow & NumPy - Deep Learning Tensor Ops

20220212220317
20220212220400

30. TensorBoard with PyTorch - Visualize Deep Learning Metrics

完整代码:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.tensorboard import SummaryWriter
import torchvision
import torchvision.transforms as transforms


torch.set_printoptions(linewidth=120)
torch.set_grad_enabled(True)

def get_num_correct(preds,labels):
    return preds.argmax(dim=1).eq(labels).sum().item()

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()

        # 维度计算公式:output=(n-filter+2padding)/stride+1。因此:
        # 原始图片大小:28*28
        # conv1后,28-5+1=24。当前图片大小:24*24
        # max_pool2d后,(24-2)/2+1=12。当前图片大小:12*12
        # conv2后,12-5+1=8。当前图片大小:8*8
        # max_pool2后,(8-2)/2+1=4。当前图片大小:4*4。当前这张图片有12g个channels
        # 此时,这张图片的维度为1*12*4*4(batch*channel*heigtht*width)。在进fc1时,把这张图片转为(1行,12*4*4列)维度

        self.conv1=nn.Conv2d(in_channels=1,out_channels=6,kernel_size=5)
        self.conv2=nn.Conv2d(in_channels=6,out_channels=12,kernel_size=5)

        self.fc1=nn.Linear(in_features=12*4*4,out_features=120)
        self.fc2=nn.Linear(in_features=120,out_features=60)
        self.out=nn.Linear(in_features=60,out_features=10) # 有10种输出类型

    def forward(self,t):
        t=F.relu(self.conv1(t));
        t=F.max_pool2d(t,kernel_size=2,stride=2)
        t=F.relu(self.conv2(t))
        t=F.max_pool2d(t,kernel_size=2,stride=2)

        t=t.reshape(-1,12*4*4)
        t=F.relu(self.fc1(t))
        t=F.relu(self.fc2(t))
        t=self.out(t)
        return t

train_set=torchvision.datasets.FashionMNIST(
    root='./data',
    train=True,
    download=True,
    transform=transforms.Compose([
        transforms.ToTensor()
    ])
)

network=Network()
train_loader=torch.utils.data.DataLoader(train_set,batch_size=100,shuffle=True)
optimizer=optim.Adam(network.parameters(),lr=0.01)

images,labels=next(iter(train_loader))
grid=torchvision.utils.make_grid(images)

tb = SummaryWriter()
tb.add_image('images',grid)
tb.add_graph(network,images)

for epoch in range(1): # epoch表示完整的一批数据
    total_loss = 0
    total_correct = 0
    for batch in train_loader: # batch表示某epoch中的一部分数据
        images,labels=batch
        preds=network(images)
        loss=F.cross_entropy(preds,labels)

        optimizer.zero_grad()
        loss.backward() # calculate gradients
        optimizer.step() # update weights

        total_loss+=loss.item()
        total_correct+=get_num_correct(preds,labels)

    tb.add_scalar('Loss',total_loss,epoch)
    tb.add_scalar('Number Correct',total_correct,epoch)
    tb.add_scalar('Accuracy',total_correct/len(train_set),epoch)

    tb.add_histogram('conv1.bias',network.conv1.bias,epoch)
    tb.add_histogram('conv1.weight',network.conv1.weight,epoch)
    tb.add_histogram('conv1.weight.grad',network.conv1.weight.grad,epoch)
    # print(network.conv1.weight.size())
    # print(network.conv1.weight.grad.size())
    # print(network.fc1.weight.grad.size())


    print("epoch: ",epoch,
          "total_loss: ",total_loss,
          "total_correct: ",total_correct)

tb.close()

权重说明:
network.conv1.weight.size()为(6*1*5*5),输入为1channel,输出为6channels, 所以为6*1,kernel大小为5,所以为5*5. 相应地,network.conv1.weight.grad.size()也为(6*1*5*5)

20220213152135

31. Hyperparameter Tuning and Experimenting - Training Deep Neural Networks

32. Training Loop Run Builder - Neural Network Experimentation Code

Using The RunBuilder Class

  • build sets of parameters that define our runs.

All we have to do to add additional values is to add them to the original parameter list, and if we want to add an additional type of parameter, all we have to do is add it. The new parameter and its values will automatically become available to be consumed inside the run. The string output for the run also updates as well.

This functionality will allow us to have greater control as we experiment with different values during training.

from collections import OrderedDict
from collections import namedtuple
from itertools import product

class RunBuilder():
    @staticmethod
    def get_runs(params):

        Run = namedtuple('Run', params.keys())

        runs = []
        for v in product(*params.values()):
            runs.append(Run(*v))

        return runs

# Two parameters:
params = OrderedDict(
    lr = [.01, .001]
    ,batch_size = [1000, 10000]
)

# Three parameters:
params = OrderedDict(
    lr = [.01, .001]
    ,batch_size = [1000, 10000]
    ,device = ["cuda", "cpu"]
)
runs = RunBuilder.get_runs(params)
print(runs)

# output of two parameters
#
# [
#     Run(lr=0.01, batch_size=1000),
#     Run(lr=0.01, batch_size=10000),
#     Run(lr=0.001, batch_size=1000),
#     Run(lr=0.001, batch_size=10000)
# ]

# output of three parameters
#
# [
#     Run(lr=0.01, batch_size=1000, device='cuda'),
#     Run(lr=0.01, batch_size=1000, device='cpu'),
#     Run(lr=0.01, batch_size=10000, device='cuda'),
#     Run(lr=0.01, batch_size=10000, device='cpu'),
#     Run(lr=0.001, batch_size=1000, device='cuda'),
#     Run(lr=0.001, batch_size=1000, device='cpu'),
#     Run(lr=0.001, batch_size=10000, device='cuda'),
#     Run(lr=0.001, batch_size=10000, device='cpu')
# ]

20220213161157

33. CNN Training Loop Refactoring - Simultaneous Hyperparameter Testing

20220213193129

34. PyTorch DataLoader num_workers - Deep Learning Speed Limit Increase

num_workers 并非越大越好。随着num_workers增加,时间先减小后增大。可以在parameters中尝试,从而确定num_workers的具体值

35. PyTorch on the GPU - Training Neural Networks with CUDA

PyTorch: What is the difference between tensor.cuda() and tensor.to(torch.device(“cuda:0”))?
.cuda()/.cpu() is the old, pre-0.4 way. As of 0.4, it is recommended to use .to(device) because it is more flexible
20220213234656

20220213232452
20220213232551
20220213232603

38. PyTorch Sequential Models - Neural Networks Made Easy

用nn.Sequential 构建网络的三种方式

20220216232511

20220216232847

build the same network using the Sequential Class

20220216232935
20220216233235
20220216233347


Supplement: RNN

PyTorch Zero To All Lecture:

Youtube Video
Code
Slides

  • PyTorch Lecture 12: RNN1 - Basics





   转载规则


《Deep Learning with PyTorch-note》 M 采用 知识共享署名 4.0 国际许可协议 进行许可。
 上一篇
概率统计及优化方法 概率统计及优化方法
概率统计 先验概率,后验概率,最大似然 举例西瓜书里“瓜熟蒂落”的例子: 首先要理解“瓜熟”是“因”,“蒂落”是“果”其次要理解“瓜熟”并不一定会“蒂落”(就是那么顽强),而“蒂落”了也并不一定“瓜熟”(被熊孩子掰下来了)也就是说一切都有
2020-11-13
下一篇 
CPP新式转型 CPP新式转型
Type Casting dynamic_cast用于父类和子类之间的转换(一般是父类向子类的安全向下转) dynamic_cast can be used only with pointers and references to ob
2020-10-16
  目录