Báo cáo Image classification techniques môn Nhập môn học máy và khai phá dữ liệu | Trường Đại học Bách Khoa Hà Nội

Machine Learning Report Group 10 May 21, 2025
HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY
SCHOOL OF INFORMATION AND COMMUNICATION TECHNOLOGY MACHINE LEARNING
Topic: IMAGE CLASSIFICATION WITH CIFAR-10
Instructor: Dr. Dinh Viet Sang Program:
Data Science and Artificial Intelligence Members: Le Nhat Hoang 20235498
Dinh Quoc Bao 20235478 Nguyen Hai Dang 20235484 Nguyen Xuan Khai 20235508 Le Hoang Tung 20235572 Contents
1 K-NEAREST NEIGHBOURS 3 1.1 Data Pre-processing
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 SUPPORT VECTOR MACHINE AND PRINCIPAL COMPONENT ANAL- YSIS 10
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2 Data Pre-processing
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3 NEURAL NETWORK 13 3.1 Data Pre-processing
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3.1 Loss evaluation during training . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.3.2 External evaluation
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3.3 Evaluation on each class of the dataset . . . . . . . . . . . . . . . . . . . . . 16
3.4 Remark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.4.1 Strengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.4.2 Weaknesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4 CONVOLUTIONAL NEURAL NETWORKS 19 4.1 Data Pre-processing
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.2 MobileNetV1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.3 ResNet34 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.4 Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2
CHAPTER 1. K-NEAREST NEIGHBOURS 1.1 Data Pre-processing
• Loading CIFAR-10: We load 50,000 training and 10,000 test images [3×32×32] along with their
labels into tensors x train, y train, x test, y test.
– x train: shape (B, C, H, W) tensor where B is batch size, C is channel size, H is height, and W is width.
– y train: shape (B) tensor where B is batch size.
– x test: shape (B, C, H, W) tensor where B is batch size, C is channel size, H is height, and W is width.
– y test: shape (B) tensor where B is batch size.
Each image is composed of 3×32×32 pixel values. The 3×32×32 references their
height/width and the color channels, which most of computer vision (to my knowledge)
use RGB. The labels are simply a tensor of integers ranging from [0,9]. Each integer serves
as an index to a corresponding list of classes:
classes = [’plane’, ’car’, ’bird’, ’cat’, ’deer’, ’dog’, ’frog’, ’horse’, ’ship’, ’truck’]
• Subsampling: To speed up experimentation, we will subsample the dataset (e.g., 5,000 training
images and 500 test images) by selecting the first n examples from each tensor.
• Flattening: Each image tensor of shape [N, 3, 32, 32] is reshaped to [N, 3072] by flattening the
color channels and pixels so that each row corresponds to all pixel values for one image. 1.2 Model Architecture
• During building process with kNN model, we also subsample the training set to 500 and
the test to 250 to save time.
• kNN Classifier: A simple nearest-neighbors approach.
• Distance Metric: Euclidean distance between flattened pixel vectors.
• Implementation Variants:
1. Two-Loop Version: Nested loops over each test and training image to compute distances in
O(Ntest × Ntrain × D). 3 lOMoAR cPSD| 59595715 CHAPTER 1. K- NEAREST NEIGHBOURS
2. No-Loop Version: Use broadcasting and matrix algebra:
where sums are precomputed per-row and cross-terms via matrix multiplication.
• Prediction: For each test image, select the k training images with smallest distances, look up their labels, and assign the most frequent label.
Label prediction: [language=Python] dists = computedistancesnoloops(self.xtrain,xtest)
Index over test images for i in range(dists.shape[1]): Find index of k lowest values x = torch.topk(dists[:, i], k, largest=False).indices
Index the labels according to x klowestlabels = self.ytrain[x]
ytestpred[i] = themostfrequentoccuringindexytestpred[i] = torch.argmax(torch.bincount(klowestlabels)) return ytestpred
Accurracy checking: [language=Python] ytestpred = self.predict(xtest,k = k)numsamples = xtest.shape[0]numcorrect
= (ytest == ytestpred).sum().item()accuracy = 100.0∗numcorrect/numsamples 1.3 Results • Speed Comparison:
– Two-loop: ∼1.92 seconds
– No-loop: ∼0.01 seconds (we use this to minimize executed time)
Downloaded by Taj Tjk (tajtjkadv86@gmail.com) lOMoAR cPSD| 59595715 CHAPTER 1. K- NEAREST NEIGHBOURS
• Accuracy with k = 5, 5000 trains/500 tests:
– 27.8% classification accuracy (139/500 correct) on CIFAR-10 dataset
[language=Python] torch.manualseed(0)numtrain = 5000numtest = 500xtrain,ytrain,xtest,ytest =
cifar10(numtrain,numtest)classifier = KnnClassifier(xtrain,ytrain)classifier.checkaccuracy(xtest,ytest, 5)
Got 139/500 correct; accuracy is 27.8% • Cross Validation:
– Cross validation further segregates our training set into ”chunks”. For our subsample of
5,000 test images and labels, we can create 5 tensors of shape [1,000, 3072]. One of those
5 chunks becomes what is called the validation set to evaluate an optimal k. The validation
set does the job of the previously defined test set.
→ Accuracies when running cross validation on our subsample:
k = mygreen1 got accuracies: mygreen[26.3, 25.7, 26.4, 27.8, 26.6] k =
mygreen3 got accuracies: mygreen[23.9, 24.9, 24.0, 26.6, 25.4] k =
mygreen5 got accuracies: mygreen[24.8, 26.6, 28.0, 29.2, 28.0] k =
mygreen8 got accuracies: mygreen[26.2, 28.2, 27.3, 29.0, 27.3] k =
mygreen10 got accuracies: mygreen[26.5, 29.6, 27.6, 28.4, 28.0] k =
mygreen13 got accuracies: mygreen[26.2, 29.1, 27.3, 27.4, 27.3] k =
mygreen20 got accuracies: mygreen[27.0, 27.9, 27.9, 28.2, 28.5] k =
mygreen23 got accuracies: mygreen[26.9, 28.4, 27.7, 27.1, 27.3] k =
mygreen36 got accuracies: mygreen[26.9, 27.7, 28.2, 26.4, 27.2] k =
mygreen50 got accuracies: mygreen[27.1, 28.8, 27.8, 26.9, 26.6] k =
mygreen59 got accuracies: mygreen[26.5, 28.1, 26.8, 26.5, 27.0] k =
mygreen100 got accuracies: mygreen[25.6, 27.0, 26.3, 25.6, 26.3] k =
mygreen109 got accuracies: mygreen[26.1, 26.8, 26.5, 25.2, 26.8]
Downloaded by Taj Tjk (tajtjkadv86@gmail.com) lOMoAR cPSD| 59595715 CHAPTER 1. K- NEAREST NEIGHBOURS 5
Figure 1.1: k-Accuracy visualization with cross validation in subsample
• Running on test subsample: [language=Python] Accuracy of best k (10): torch.manualseed(0)xtrainall,ytrai
cifar10(5000,500)p = knngetbestk(ktoaccuracies)classifier = KnnClassifier(xtrainall,ytrainall)classifier.c p)
Got 141/500 correct; accuracy is 28.2%
• Tests again on entire dataset:
– Instead of training on a subsample with 5000 training images, we train all 50,000 images
in training set throughout cross validation, finding best k is 1 (an image with smallest
distance compares to the test images determines the label of test sample)
Downloaded by Taj Tjk (tajtjkadv86@gmail.com)
Figure 1.2: k-Accuracy over whole dataset
• Test the best k over the whole dataset: [language=Python] Accuracy of best k (1):
torch.manualseed(0)xtrainall,ytrainall,xtestall,ytestall = cifar10()p =
knngetbestk(ktoaccuracies)classifier
KnnClassifier(xtrainall,ytrainall)classifier.checkaccuracy(xtestall,ytestall,k = p)
Got 3539/10000 correct; accuracy is 35.39%
• Checking whether the best k we found is the real best by checking accuracy of all k: [1, 3, 5, 8, 10, 13, 23, 20,
36, 50, 59, 100, 109] k = 1 : Got 3539 / 10000 correct; accuracy is 35.39k = 3 : Got 3303 / 10000 correct; accuracy
is 33.03k = 5 : Got 3398 / 10000 correct; accuracy is 33.98k = 8 : Got 3415 / 10000 correct; accuracy is 34.15k =
10 : Got 3386 / 10000 correct; accuracy is 33.86k = 13 : Got 3417 / 10000 correct; accuracy is 34.17k = 20 : Got
3375 / 10000 correct; accuracy is 33.75k = 23 : Got 3349 / 10000 correct; accuracy is 33.49k = 36 : Got 3313 /
10000 correct; accuracy is 33.13k = 50 : Got 3239 / 10000 correct; accuracy is 32.39k = 59 : Got 3216 / 10000
correct; accuracy is 32.16k = 100 : Got 3118 / 10000 correct; accuracy is 31.18k = 109 : Got 3094 / 10000 correct; accuracy is 30.94
k = 1 is the real best k 1.4 Conclusion • Strengths:
– Conceptually simple; no training time required beyond storing data.
– Serves as a baseline to compare more complex models. • Weaknesses:
– Computationally expensive at prediction time (O(Ntrain)).
– Poor accuracy on image data (≈ 30% vs > 90% for CNNs).
– Sensitive to choice of k and distance metric. • When to Use:
– Educational purposes to understand distance-based classification and broadcasting efficiency.
– As a quick baseline on small datasets or embedded tasks with limited classes. lOMoAR cPSD| 59595715
CHAPTER 2. SUPPORT VECTOR MACHINE AND PRINCIPAL COMPONENT ANALYSIS 2.1 Introduction
We will explore the application of Support Vector Machines (SVM) combined with Principal
Component Analysis (PCA) for image classification on the CIFAR-10 dataset. The goal is to
understand the interplay between dimensionality reduction via PCA and the classification
performance of SVMs, including the impact of kernel choice and hyperparameter tuning. 2.2 Data Pre-processing
• Loading CIFAR-10: We use torchvision to load the CIFAR-10 dataset and apply necessary transforms.
• Tensor Conversion: Images are initially loaded as PyTorch tensors, typically with a shape of
[3, 32, 32] (Channels, Height, Width).
• Flattening: Each image tensor is flattened into a 1D vector. For CIFAR-10 images, this results in
3 × 32 × 32 = 3072 features per image.
• Data Format Conversion: For compatibility with libraries like Scikit-learn, the training and
test sets (features and labels) are converted to NumPy arrays.
• Data Splitting: The original training set (50,000 images) is split into a new training subset (e.g.,
49,000 images) and a validation subset (e.g., 1,000 images). The validation set is used for
hyperparameter tuning before evaluating on the final test set (10,000 images). 2.3 Model Architecture
The classification pipeline involves two main stages: dimensionality reduction using PCA, followed
by classification using an SVM.
• Principal Component Analysis (PCA):
– PCA is employed to reduce the dimensionality of the flattened image data (3072 features).
It works by transforming the high-dimensional data into a lower-dimensional space (the
principal components) while maximizing the variance of the data in this new space. This
helps preserve the most important patterns and relationships.
– We experiment with reducing the dimensionality to 150 and 500 principal components. •
Support Vector Machine (SVM):
– Core Idea: The SVM algorithm seeks to find an optimal hyperplane that best separates
data points belonging to different classes in the (potentially transformed) feature space.
– Margin Maximization: SVM achieves this by maximizing the margin, which is defined as
the distance from the hyperplane to the nearest data points (the support vectors) from
each class. By maximizing this margin, SVM aims for better generalization.
– Kernel Trick: When the data is not linearly separable in its original feature space, SVMs
can utilize a technique called the kernel trick. Kernels are functions that implicitly map
the data into a higher-dimensional space where linear separation might be feasible,
without explicitly computing the coordinates in that higher-dimensional space.
– Chosen Kernel: For this dataset and experimental setup, a polynomial kernel is used, with
the polynomial degree set to 3 (a common default in libraries like Scikit-learn).
– Regularization Parameter (C): The C parameter in SVMs is a regularization parameter
that controls the trade-off between maximizing the margin and minimizing the
classification error on the training data. A larger C allows for a smaller margin but fewer training Downloaded by Taj Tjk (tajtjkadv86@gmail.com) lOMoAR cPSD| 59595715
misclassifications, while a smaller C encourages a larger margin, potentially at the cost of
more training misclassifications. For these experiments, C is set to 5. 10
CHAPTER 2. SUPPORT VECTOR MACHINE AND PRINCIPAL COMPONENT ANALYSIS 2.4 Results
• Variance Explained by PCA (Fig.01):
– When reducing the dimensionality to 150 components, PCA retains 93.1% of the
variance in the original data.
– If the number of components is increased to 500, PCA retains 98.5% of the variance.
Figure 2.1: Cumulative explained variance as a function of the number of PCA components for CIFAR- 10 dataset.
• SVM Classification Accuracy with PCA: We evaluate the SVM’s performance using the
polynomial kernel (degree 3) and C=5 on the PCA-transformed data.
– With 150 principal components, the SVM achieves a test accuracy of 42.8%.
– With 500 principal components, the SVM achieves a test accuracy of 40.4%.
• Discussion of Results:
– Interestingly, increasing the number of PCA components from 150 to 500 (thereby
retaining more variance) resulted in a decrease in test accuracy. This suggests that for this
particular setup, using fewer components (150) not only reduces computational 11
Downloaded by Taj Tjk (tajtjkadv86@gmail.com) lOMoAR cPSD| 59595715
requirements but may also help in filtering out noise or less discriminative features,
leading to better generalization for the SVM.
– The choice of a polynomial kernel was made for these experiments. It’s noted that the
optimal kernel can change when data is transformed by PCA, as the components exist in a
different coordinate system. The polynomial kernel appeared to be a suitable choice for
this transformed space compared to a linear kernel that might have been considered for
the original high-dimensional space.
CHAPTER 2. SUPPORT VECTOR MACHINE AND PRINCIPAL COMPONENT ANALYSIS 2.5 Conclusion
• Strengths of SVM+PCA Approach:
– SVMs can be effective in high-dimensional spaces, and even more so when combined with
PCA to reduce dimensionality and potential noise.
– The kernel trick provides flexibility for SVMs to model non-linear decision boundaries.
– PCA reduces computational complexity for the subsequent SVM training and prediction phases.
– PCA can improve generalization by removing redundant or noisy features, as potentially
observed in our results where fewer components yielded better accuracy.
• Weaknesses of SVM+PCA Approach:
– SVM training can be computationally intensive for very large datasets, although PCA
mitigates this to some extent.
– The choice of PCA components, SVM kernel, and its parameters (e.g., C, gamma, degree) is
crucial and often requires careful tuning (e.g., via cross-validation).
– PCA involves a loss of information, though it aims to preserve variance. The discarded
variance might contain subtle information useful for classification in some cases.
– SVMs are inherently binary classifiers; for multi-class problems like CIFAR-10, strategies
like one-vs-one or one-vs-rest are employed, which can increase complexity. • When to Use:
– When dealing with high-dimensional data where dimensionality reduction is beneficial
for performance and computational efficiency.
– For classification tasks where a clear margin of separation is expected, or can be achieved via kernel transformations.
– As a strong baseline model, especially when feature engineering is complex or when
exploring the impact of dimensionality reduction. Model Configuration PCA Components Test Accuracy (%)
SVM (Poly Kernel, degree=3, C=5) + PCA 150 42.8
SVM (Poly Kernel, degree=3, C=5) + PCA 500 40.4
Table 2.1: Performance of SVM with Polynomial Kernel and PCA on CIFAR-10. 12
Downloaded by Taj Tjk (tajtjkadv86@gmail.com) lOMoAR cPSD| 59595715
CHAPTER 3. NEURAL NETWORK 3.1 Data Pre-processing
• Loading CIFAR-10: We load 45,000 training, 5,000 validation and 10,000 test images
[32×32×3] along with their labels into variables trainloader, valloader, testloader.
– trainloader: DataLoader object with shape (M, input, label) for the training set
– valloader: DataLoader object with shape (N, input, label) for the validation set
– testloader: DataLoader object with shape (O, input, label) for the test set Where: ∗ N: the number of samples
∗ input: image of size H×W×C, where H is height, W is width, C is channel size
∗ label: a number in range [0:9] corresponding to each class in the dataset M = 45,000 samples N = 5,000 samples O = 10,000 samples H = W = 32 pixels C = 3 channels (RGB) label ∈classes
classes = [’plane’, ’car’, ’bird’, ’cat’, ’deer’, ’dog’, ’frog’, ’horse’, ’ship’, ’truck’]
Images from the CIFAR-10 dataset
• Normalization: Before loading into the model, the input images are normalized using the
original mean and std. values given for the CIFAR-10 dataset.
Mean = (0.4914, 0.4822, 0.4465)
Std = (0.2023, 0.1994, 0.2010)
• Flattening: Each image input of shape [32, 32, 3] is reshaped to [3072] by flattening the color
channels and pixels so that each row corresponds to all pixel values for one image.
• Batch training: For the training set specifically, we split it into batches of size 128 before
training. Since the training set contains around 45,000 images, this means about 350 batches
are iterated over for each epoch during the training phase. 13
Downloaded by Taj Tjk (tajtjkadv86@gmail.com) 3.2 Model Architecture Model architecture
• The model consists of four fully connected layers:
1. Input layer: Receives an image of shape [32x32x3] (Flattened to 3072) and returns tensor of shape [1024]
2. Hidden layer 1: Receives a tensor of shape [1024] and returns tensor of shape [512]
3. Hidden layer 2: Receives a tensor of shape [512] and returns tensor of shape [256]
4. Output layer: Receives a tensor of shape [256] and returns tensor of shape [10]
• Activation function: For the first three layers, the ReLU activation function is used to introduce
non-linearity while also avoiding vanishing gradients. For the last layer, the Softmax activation
function is used for outputing prediction values i.e. the probability that the image belongs to each class.
• Loss function: The Categorical Cross-Entropy loss function is used for this multi-class classification problem.
• Dropout: For every layer (except the output layer), Dropout is introduced with a dropout rate of 0.5 to avoid overfitting.
• Optimizer: The Adam optimizer is used for this problem
• Maximum number of epochs for training: 50 CHAPTER 3. NEURAL NETWORK 3.3 Results 3.3.1
Loss evaluation during training
Training and validation loss graph
We can see the training and validation loss values in the graph above. There are a few options for us to choose:
• Stop training where validation loss is lowest (epoch 32)
• Stop training around the intersection area of training and validation loss (epoch 13)
• Other strategies for checkpoint selection
The training phase ends with a loss value of around 1.2 3.3.2 External evaluation
After training the model successfully, we test the performance of the obtained model on the 10,000
test images, which yields an accuracy of around 53%
Model performance on a few samples from external test set 15 3.3.3
Evaluation on each class of the dataset
Accuracy values on each class during testing
From this, we can infer that classes where images contain less noise often yield a higher accuracy.
(e.g: ”frog” and ”ship” images have less noisy backgrounds) 3.4 Remark
Although the performance of fully connected neural network is generally better than some other
previous technologies (SVMs, KNN,...), this method still fails to classify images from the CIFAR-10
dataset effectively compared to more novel architectures (CNN, ViT,...). 3.4.1 Strengths
• Fully connected neural network is very simple and easy to implement with few details in terms of architecture.
• This model is flexible in receiving input of many types, since we only need to flatten the initial
input into a vector of size [N]
• The relationship between each neuron from two adjacent layers is direct, which helps to ensure
that all data interacts with each other and no bias is introduced. 3.4.2 Weaknesses
• Loss of spatial information: Initially, we have to flatten the images of shape [32x32x3] into a
1D tensor; therefore, the original spatial information which represents each pixel’s location is
lost during the process. Therefore, this method is not very effective in preserving spatial information. CHAPTER 3. NEURAL NETWORK
An illustration of image flattening
• High number of parameters: Fully connected neural networks yield a low cost-efficiency rate.
They often require tuning a large number of parameters, especially for image inputs.
Consequently, there is a burden on computational resources. Overview of our model
For example, in the 4-layer model we designed, the minimal number of parameters required is:
1024 x 512 x 256 x 10 = 1,342,177,280 parameters 3.5 Conclusion
In this work, we implemented a fully connected neural network (FCNN) to classify images from
the CIFAR-10 dataset. While the model achieved moderate performance with a test accuracy of
approximately 53%, its limitations became evident when compared to more specialized architectures
such as Convolutional Neural Network (CNN) or Vision Transformer (ViT).
The FCNN’s simplicity and ease of implementation make it a simple, yet fundamental baseline for
understanding deep learning models. It effectively demonstrates fundamental concepts such as
activation functions, dropout regularization, and loss optimization. However, the model’s inability to
capture spatial information and high computational requirements result in suboptimal performance on this classification task. 17
In summary, while FCNNs offer educational value and are suitable for tabular or low-dimensional
data, they are not well-suited for image classification tasks like CIFAR-10. For such applications,
convolutional architectures are significantly more effective due to their ability to exploit the spatial
structure of data types such as images.