textbox.module.Attention

Attention Layers

class textbox.module.Attention.attention_mechanism.BahdanauAttention(source_size, target_size)

Bases: Module

Bahdanau Attention is proposed in the following paper:

Neural Machine Translation by Jointly Learning to Align and Translate.

Reference:

https://arxiv.org/abs/1409.0473

forward(hidden_states, encoder_outputs, encoder_masks)

Bahdanau attention

Parameters

• hidden_states – shape: [batch_size, tgt_len, target_size]

• encoder_outputs – shape: [batch_size, src_len, source_size]

• encoder_masks – shape: [batch_size, src_len]

Returns

• context: shape: [batch_size, tgt_len, source_size]

• probs: shape: [batch_size, tgt_len, src_len]

Return type

tuple

score(hidden_states, encoder_outputs)

Calculate the attention scores between encoder outputs and decoder states.

training: bool

class textbox.module.Attention.attention_mechanism.LuongAttention(source_size, target_size, alignment_method='concat', is_coverage=False)

Bases: Module

Luong Attention is proposed in the following paper: Effective Approaches to Attention-based Neural Machine Translation.

Reference:

https://arxiv.org/abs/1508.04025

forward(hidden_states, encoder_outputs, encoder_masks, coverages=None)

Luong attention

Parameters

• hidden_states – shape: [batch_size, tgt_len, target_size]

• encoder_outputs – shape: [batch_size, src_len, source_size]

• encoder_masks – shape: [batch_size, src_len]

Returns

• context: shape: [batch_size, tgt_len, source_size]

• probs: shape: [batch_size, tgt_len, src_len]

Return type

tuple

score(hidden_states, encoder_outputs, coverages=None)

Calculate the attention scores between encoder outputs and decoder states.

training: bool

class textbox.module.Attention.attention_mechanism.MonotonicAttention(source_size, target_size, init_r=- 4)

Bases: Module

Monotonic Attention is proposed in the following paper:

Online and Linear-Time Attention by Enforcing Monotonic Alignments.

Reference:

https://arxiv.org/abs/1704.00784

exclusive_cumprod(x)

Exclusive cumulative product [a, b, c] => [1, a, a * b]

gaussian_noise(*size)

Additive gaussian nosie to encourage discreteness

hard(hidden_states, encoder_outputs, encoder_masks, previous_probs=None)

Hard monotonic attention (Test)

Parameters

• hidden_states – shape: [batch_size, tgt_len, target_size]

• encoder_outputs – shape: [batch_size, src_len, source_size]

• encoder_masks – shape: [batch_size, src_len]

• previous_probs – shape: [batch_size, tgt_len, src_len]

Returns

• context: shape: [batch_size, tgt_len, source_size]

• probs: shape: [batch_size, tgt_len, src_len]

Return type

tuple

safe_cumprod(x)

Numerically stable cumulative product by cumulative sum in log-space

score(hidden_states, encoder_outputs)

Calculate the attention scores between encoder outputs and decoder states.

soft(hidden_states, encoder_outputs, encoder_masks, previous_probs=None)

Soft monotonic attention (Train)

Parameters

• hidden_states – shape: [batch_size, tgt_len, target_size]

• encoder_outputs – shape: [batch_size, src_len, source_size]

• encoder_masks – shape: [batch_size, src_len]

• previous_probs – shape: [batch_size, tgt_len, src_len]

Returns

• context: shape: [batch_size, tgt_len, source_size]

• probs: shape: [batch_size, tgt_len, src_len]

Return type

tuple

training: bool

class textbox.module.Attention.attention_mechanism.MultiHeadAttention(embedding_size, num_heads, attn_weight_dropout_ratio=0.0, return_distribute=False)

Bases: Module

Multi-head Attention is proposed in the following paper:

Attention Is All You Need.

Reference:

https://arxiv.org/abs/1706.03762

forward(query, key, value, key_padding_mask=None, attn_mask=None)

Multi-head attention

Parameters

• query – shape: [batch_size, tgt_len, embedding_size]

• value (key and) – shape: [batch_size, src_len, embedding_size]

• key_padding_mask – shape: [batch_size, src_len]

• attn_mask – shape: [batch_size, tgt_len, src_len]

Returns

• attn_repre: shape: [batch_size, tgt_len, embedding_size]

• attn_weights: shape: [batch_size, tgt_len, src_len]

Return type

tuple

reset_parameters()

training: bool

class textbox.module.Attention.attention_mechanism.SelfAttentionMask(init_size=100)

Bases: Module

forward(size)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

static get_mask(size)

training: bool