I am looking at an implementation of RCNN for text classification using PyTorch. Full Code. There are two points where the dimensions of tensors are permuted using the permute function. The first is after the LSTM layer and before tanh. The second is after a linear layer and before a max pooling layer.
Could you please explain why the permutation is necessary or useful?
Relevant Code
def forward(self, x): # x.shape = (seq_len, batch_size) embedded_sent = self.embeddings(x) # embedded_sent.shape = (seq_len, batch_size, embed_size) lstm_out, (h_n,c_n) = self.lstm(embedded_sent) # lstm_out.shape = (seq_len, batch_size, 2 * hidden_size) input_features = torch.cat([lstm_out,embedded_sent], 2).permute(1,0,2) # final_features.shape = (batch_size, seq_len, embed_size + 2*hidden_size) linear_output = self.tanh( self.W(input_features) ) # linear_output.shape = (batch_size, seq_len, hidden_size_linear) linear_output = linear_output.permute(0,2,1) # Reshaping fot max_pool max_out_features = F.max_pool1d(linear_output, linear_output.shape[2]).squeeze(2) # max_out_features.shape = (batch_size, hidden_size_linear) max_out_features = self.dropout(max_out_features) final_out = self.fc(max_out_features) return self.softmax(final_out) Similar Code in other Repositories
Similar implementations of RCNN use permute or transpose. Here are examples:
2 Answers
What permute function does is rearranges the original tensor according to the desired ordering, note permute is different from reshape function, because when apply permute, the elements in tensor follow the index you provide where in reshape it's not.
Example code:
import torch var = torch.randn(2, 4) pe_var = var.permute(1, 0) re_var = torch.reshape(var, (4, 2)) print("Original size:\n{}\nOriginal var:\n{}\n".format(var.size(), var) + "Permute size:\n{}\nPermute var:\n{}\n".format(pe_var.size(), pe_var) + "Reshape size:\n{}\nReshape var:\n{}\n".format(re_var.size(), re_var)) Outputs:
Original size: torch.Size([2, 4]) Original var: tensor([[ 0.8250, -0.1984, 0.5567, -0.7123], [-1.0503, 0.0470, -1.9473, 0.9925]]) Permute size: torch.Size([4, 2]) Permute var: tensor([[ 0.8250, -1.0503], [-0.1984, 0.0470], [ 0.5567, -1.9473], [-0.7123, 0.9925]]) Reshape size: torch.Size([4, 2]) Reshape var: tensor([[ 0.8250, -0.1984], [ 0.5567, -0.7123], [-1.0503, 0.0470], [-1.9473, 0.9925]]) With the role of permute in mind we could see what first permute does is reordering the concatenate tensor for it to fit the inputs format of self.W, i.e with batch as first dimension; and the second permute does similar thing because we want to max pool the linear_output along the sequence and F.max_pool1d will pool along the last dimension.
I am adding this answer to provide additional PyTorch-specific details.
It is necessary to use permute between nn.LSTM and nn.Linear because the output shape of LSTM does not correspond to the expected input shape of Linear.
nn.LSTM outputs output, (h_n, c_n). Tensor output has shape seq_len, batch, num_directions * hidden_size nn.LSTM. nn.Linear expects an input tensor with shape N,∗,H, where N is batch size and H is number of input features. nn.Linear.
It is necessary to use permute between nn.Linear and nn.MaxPool1d because the output of nn.Linear is N, L, C, where N is batch size, C is the number of features, and and L is sequence length. nn.MaxPool1d expects an input tensor of shape N, C, L. nn.MaxPool1d
I reviewed seven implementations of RCNN for text classification with PyTorch on GitHub and gitee and found that permute and transpose are the normal ways to convert the output of one layer to the input of a subsequent layer.
