In neural networks, a pooling layer is a kind of network layer that downsamples and aggregates information that is dispersed among many vectors into fewer vectors.[1] It has several uses. It removes redundant information, reducing the amount of computation and memory required, makes the model more robust to small variations in the input, and increases the receptive field of neurons in later layers in the network.
Convolutional neural network pooling
editPooling is most commonly used in convolutional neural networks (CNN). We[who?] describe pooling in 2-dimensional CNNs. The generalization to n-dimensions is immediate.
As notation, we consider a tensor , where is height, is width, and is the number of channels. A pooling layer outputs a tensor .
We define two variables called "filter size" and "stride". Sometimes, it is necessary to use a different filter size and stride for horizontal and vertical directions. In such cases, we define 4 variables .
The receptive field of an entry in the output tensor are all the entries in that can affect that entry.
Max pooling
editMax Pooling (MaxPool) is commonly used in CNNs to reduce the spatial dimensions of feature maps. It was introduced in 1990.[2]
Define where means the range . Note that we need to avoid the off-by-one error. The next input is and so on. The receptive field of is , so in general, If the horizontal and vertical filter size and strides differ, then in general, More succinctly, we can write .
If is not expressible as where is an integer, then for computing the entries of the output tensor on the boundaries, max pooling would attempt to take as inputs variables off the tensor. In this case, how those non-existent variables are handled depends on the padding conditions, illustrated on the right.
Global Max Pooling (GMP) is a specific kind of max pooling where the output tensor has shape and the receptive field of is all of . That is, it takes the maximum over each entire channel. It is often used just before the final fully connected layers in a CNN classification head.
Average pooling
editAverage pooling (AvgPool) is similarly defined Global Average Pooling (GAP) is defined similarly to GMP. It was first proposed in Network-in-Network.[3] Similarly to GMP, it is often used just before the final fully connected layers in a CNN classification head.
Interpolations
editThere are some interpolations of max pooling and average pooling.
Mixed Pooling is a linear sum of maxpooling and average pooling.[4] That is,
where is either a hyperparameter, a learnable parameter, or randomly sampled anew every time.
Lp Pooling is like average pooling, but uses Lp norm average instead of average: where is the size of receptive field, and is a hyperparameter. If all activations are non-negative, then average pooling is the case of , and maxpooling is the case of . Square-root pooling is the case of .[5]
Stochastic pooling samples a random activation from the receptive field with probability . It is the same as average pooling in expectation.[6]
Softmax pooling is like maxpooling, but uses softmax, i.e. where . Average pooling is the case of , and maxpooling is the case of [5]
Other poolings
editSpatial pyramidal pooling applies max pooling (or any other form of pooling) in a pyramid structure. That is, it applies global max pooling, then applies max pooling to the image divided into 4 equal parts, then 16, etc. The results are then concatenated. It is a hierarchical form of global pooling, and similar to global pooling, it is often used just before a classification head.[7]
Region of Interest Pooling (also known as RoI pooling) is a variant of max pooling used in R-CNNs for object detection.[8]
Covariance pooling computes the covariance matrix of the vectors which is then flattened to a -dimensional vector . Global covariance pooling is used similarly to global max pooling. As average pooling computes the average, which is a first-degree statistic, and covariance is a second-degree statistic, covariance pooling is also called "second-order pooling". It can be generalized to higher-order poolings.[9][10]
Blur Pooling means applying a blurring method, before downsampling. For example, the Rect-2 blur pooling means taking an average pooling at , then taking every second pixel (identity with ).[11]
Vision Transformer pooling
editIn Vision Transformers (ViT), there are the following common kinds of poolings.
BERT-like pooling uses a dummy [CLS]
token ("classification"). For classification, the output at [CLS]
is the classification token, which is then processed by a LayerNorm-feedforward-softmax module into a probability distribution, which is the network's prediction of class probability distribution. This is the one used by the original ViT[12] and Masked Autoencoder.[13]
Global average pooling (GAP) does not use the dummy token, but simply takes the average of all output tokens as the classification token. It was mentioned in the original ViT as being equally good.[12]
Multihead attention pooling (MAP) applies a multiheaded attention block to pooling. Specifically, it takes as input a list of vectors , which might be thought of as the output vectors of a layer of a ViT. It then applies a feedforward layer on each vector, resulting in a matrix . This is then sent to a multiheaded attention, resulting in , where is a matrix of trainable parameters.[14] This was first proposed in the Set Transformer architecture.[15]
Later papers demonstrated that GAP and MAP both perform better than BERT-like pooling.[14][16]
Graph neural network pooling
editThis section may require copy editing for notations need to be explained and unified with the rest of the article. (September 2024) |
In graph neural networks (GNN), there are also two forms of pooling: global and local. Global pooling can be reduced to a local pooling where the receptive field is the entire output.
- Local pooling: a local pooling layer coarsens the graph via downsampling. Local pooling is used to increase the receptive field of a GNN, in a similar fashion to pooling layers in convolutional neural networks. Examples include k-nearest neighbours pooling, top-k pooling,[17] and self-attention pooling.[18]
- Global pooling: a global pooling layer, also known as readout layer, provides fixed-size representation of the whole graph. The global pooling layer must be permutation invariant, such that permutations in the ordering of graph nodes and edges do not alter the final output.[19] Examples include element-wise sum, mean or maximum.
Local pooling layers coarsen the graph via downsampling. We present here several learnable local pooling strategies that have been proposed.[19] For each cases, the input is the initial graph is represented by a matrix of node features, and the graph adjacency matrix . The output is the new matrix of node features, and the new graph adjacency matrix .
Top-k pooling
editWe first set
where is a learnable projection vector. The projection vector computes a scalar projection value for each graph node.
The top-k pooling layer [17] can then be formalised as follows:
where is the subset of nodes with the top-k highest projection scores, denotes element-wise matrix multiplication, and is the sigmoid function. In other words, the nodes with the top-k highest projection scores are retained in the new adjacency matrix . The operation makes the projection vector trainable by backpropagation, which otherwise would produce discrete outputs.[17]
Self-attention pooling
editWe first set
where is a generic permutation equivariant GNN layer (e.g., GCN, GAT, MPNN).
The Self-attention pooling layer[18] can then be formalised as follows:
where is the subset of nodes with the top-k highest projection scores, denotes element-wise matrix multiplication.
The self-attention pooling layer can be seen as an extension of the top-k pooling layer. Differently from top-k pooling, the self-attention scores computed in self-attention pooling account both for the graph features and the graph topology.
See also
editReferences
edit- ^ Zhang, Aston; Lipton, Zachary; Li, Mu; Smola, Alexander J. (2024). "7.5. Pooling". Dive into deep learning. Cambridge New York Port Melbourne New Delhi Singapore: Cambridge University Press. ISBN 978-1-009-38943-3.
- ^ Yamaguchi, Kouichi; Sakamoto, Kenji; Akabane, Toshio; Fujimoto, Yoshiji (November 1990). A Neural Network for Speaker-Independent Isolated Word Recognition. First International Conference on Spoken Language Processing (ICSLP 90). Kobe, Japan. Archived from the original on 2021-03-07. Retrieved 2019-09-04.
- ^ Lin, Min; Chen, Qiang; Yan, Shuicheng (2013). "Network In Network". arXiv:1312.4400 [cs.NE].
- ^ Yu, Dingjun; Wang, Hanli; Chen, Peiqiu; Wei, Zhihua (2014). "Mixed Pooling for Convolutional Neural Networks". In Miao, Duoqian; Pedrycz, Witold; Ślȩzak, Dominik; Peters, Georg; Hu, Qinghua; Wang, Ruizhi (eds.). Rough Sets and Knowledge Technology. Lecture Notes in Computer Science. Vol. 8818. Cham: Springer International Publishing. pp. 364–375. doi:10.1007/978-3-319-11740-9_34. ISBN 978-3-319-11740-9.
- ^ a b Boureau, Y-Lan; Ponce, Jean; LeCun, Yann (2010-06-21). "A theoretical analysis of feature pooling in visual recognition". Proceedings of the 27th International Conference on International Conference on Machine Learning. ICML'10. Madison, WI, USA: Omnipress: 111–118. ISBN 978-1-60558-907-7.
- ^ Zeiler, Matthew D.; Fergus, Rob (2013-01-15), Stochastic Pooling for Regularization of Deep Convolutional Neural Networks, arXiv:1301.3557
- ^ He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2015-09-01). "Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition". IEEE Transactions on Pattern Analysis and Machine Intelligence. 37 (9): 1904–1916. arXiv:1406.4729. doi:10.1109/TPAMI.2015.2389824. ISSN 0162-8828. PMID 26353135.
- ^ Zhang, Aston; Lipton, Zachary; Li, Mu; Smola, Alexander J. (2024). "14.8. Region-based CNNs (R-CNNs)". Dive into deep learning. Cambridge New York Port Melbourne New Delhi Singapore: Cambridge University Press. ISBN 978-1-009-38943-3.
- ^ Tuzel, Oncel; Porikli, Fatih; Meer, Peter (2006), Leonardis, Aleš; Bischof, Horst; Pinz, Axel (eds.), "Region Covariance: A Fast Descriptor for Detection and Classification", Computer Vision – ECCV 2006, vol. 3952, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 589–600, doi:10.1007/11744047_45, ISBN 978-3-540-33834-5, retrieved 2024-09-09
- ^ Wang, Qilong; Xie, Jiangtao; Zuo, Wangmeng; Zhang, Lei; Li, Peihua (2020). "Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization". IEEE Transactions on Pattern Analysis and Machine Intelligence: 1–1. doi:10.1109/TPAMI.2020.2974833. ISSN 0162-8828.
- ^ Zhang, Richard (2018-09-27). "Making Convolutional Networks Shift-Invariant Again".
{{cite journal}}
: Cite journal requires|journal=
(help) - ^ a b Dosovitskiy, Alexey; Beyer, Lucas; Kolesnikov, Alexander; Weissenborn, Dirk; Zhai, Xiaohua; Unterthiner, Thomas; Dehghani, Mostafa; Minderer, Matthias; Heigold, Georg; Gelly, Sylvain; Uszkoreit, Jakob (2021-06-03). "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale". arXiv:2010.11929 [cs.CV].
- ^ He, Kaiming; Chen, Xinlei; Xie, Saining; Li, Yanghao; Dollar, Piotr; Girshick, Ross (June 2022). "Masked Autoencoders Are Scalable Vision Learners". 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. doi:10.1109/cvpr52688.2022.01553.
- ^ a b Zhai, Xiaohua; Kolesnikov, Alexander; Houlsby, Neil; Beyer, Lucas (June 2022). "Scaling Vision Transformers". 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. doi:10.1109/cvpr52688.2022.01179.
- ^ Lee, Juho; Lee, Yoonho; Kim, Jungtaek; Kosiorek, Adam; Choi, Seungjin; Teh, Yee Whye (2019-05-24). "Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks". Proceedings of the 36th International Conference on Machine Learning. PMLR: 3744–3753.
- ^ Karamcheti, Siddharth; Nair, Suraj; Chen, Annie S.; Kollar, Thomas; Finn, Chelsea; Sadigh, Dorsa; Liang, Percy (2023-02-24), Language-Driven Representation Learning for Robotics, doi:10.48550/arXiv.2302.12766, retrieved 2024-09-09
- ^ a b c Gao, Hongyang; Ji, Shuiwang Ji (2019). "Graph U-Nets". arXiv:1905.05178 [cs.LG].
- ^ a b Lee, Junhyun; Lee, Inyeop; Kang, Jaewoo (2019). "Self-Attention Graph Pooling". arXiv:1904.08082 [cs.LG].
- ^ a b Liu, Chuang; Zhan, Yibing; Li, Chang; Du, Bo; Wu, Jia; Hu, Wenbin; Liu, Tongliang; Tao, Dacheng (2022). "Graph Pooling for Graph Neural Networks: Progress, Challenges, and Opportunities". arXiv:2204.07321 [cs.LG].