textbox.data.dataloader.abstract_dataloader

class textbox.data.dataloader.abstract_dataloader.AbstractDataLoader(config, dataset, batch_size=1, shuffle=False, drop_last=True, DDP=False)[source]

Bases: object

AbstractDataLoader is an abstract object which would return a batch of data.

And it is also the ancestor of all other dataloader.

Parameters

  • config (Config) – The config of dataloader.

  • dataset (Corpus) – The corpus for partition of dataset.

  • batch_size (int, optional) – The batch_size of dataloader. Defaults to 1.

  • shuffle (bool) – If True, dataloader will shuffle before every epoch.

dataset

The necessary elements of this dataloader.

Type

dict

pr

Pointer of dataloader.

Type

int

step

The increment of pr for each batch.

Type

int

batch_size

The max interaction number for all batch.

Type

int

get_reference()[source]

Get reference documents for current data loader return is supposed to be reference_corpus as list -> list -> word