Transformer Encoder

class textbox.module.Encoder.transformer_encoder.TransformerEncoder(embedding_size, ffn_size, num_enc_layers, num_heads, attn_dropout_ratio=0.0, attn_weight_dropout_ratio=0.0, ffn_dropout_ratio=0.0)

Bases: Module

The stacked Transformer encoder layers.

forward(x, kv=None, self_padding_mask=None, output_all_encoded_layers=False)

Implement the encoding process step by step.

Parameters

• x (Torch.Tensor) – target sequence embedding, shape: [batch_size, sequence_length, embedding_size].

• kv (Torch.Tensor) – the cached history latent vector, shape: [batch_size, sequence_length, embedding_size], default: None.

• self_padding_mask (Torch.Tensor) – padding mask of target sequence, shape: [batch_size, sequence_length], default: None.

• output_all_encoded_layers (Bool) – whether to output all the encoder layers, default: False.

Returns

output features, shape: [batch_size, sequence_length, ffn_size].

Return type

Torch.Tensor

training: bool