nlpretext

All the goto functions you need to handle NLP use-cases, integrated in NLPretext.

class Preprocessor

Bases: object

pipe(operation, args=None)

Add an operation and its arguments to pipe in the preprocessor.

Parameters:

• operation (callable) – text preprocessing function

• args (dict of arguments) –

Return type:

None

static build_pipeline(operation_list)

Build sklearn pipeline from a operation list.

Parameters:

operation_list (iterable) – list of __operations of preprocessing

Return type:

sklearn.pipeline.Pipeline

run(text)

Apply pipeline to text.

Parameters:

text (string) – text to preprocess

Return type:

string

• augmentation
  • text_augmentation
    • CouldNotAugment
    • UnavailableAugmenter
    • augment_text()
    • process_entities_and_text()
    • are_entities_in_augmented_text()
    • get_augmenter()
    • get_augmented_entities()
    • clean_sentence_entities()
    • check_interval_included()
• basic
  • preprocess
    • normalize_whitespace()
    • remove_whitespace()
    • lower_text()
    • filter_groups()
    • ungroup_ignored_stopwords()
    • remove_stopwords()
    • remove_eol_characters()
    • fix_bad_unicode()
    • unpack_english_contractions()
    • replace_urls()
    • replace_emails()
    • replace_phone_numbers()
    • replace_numbers()
    • replace_currency_symbols()
    • remove_punct()
    • remove_accents()
    • remove_multiple_spaces_and_strip_text()
    • filter_non_latin_characters()
• cli
  • preprocess
    • run()
• social
  • preprocess
    • remove_mentions()
    • extract_mentions()
    • remove_html_tags()
    • remove_emoji()
    • convert_emoji_to_text()
    • extract_emojis()
    • extract_hashtags()
    • remove_hashtag()
• token
  • preprocess
    • remove_stopwords()
    • remove_tokens_with_nonletters()
    • remove_special_caracters_from_tokenslist()
    • remove_smallwords()
  • tokenizer
    • LanguageNotHandled
    • LanguageNotInstalledError
    • SpacyModel
      • SpacyModel.SingletonSpacyModel
      • SpacyModel.model
      • SpacyModel.get_lang_model()
    • tokenize()
    • untokenize()
    • convert_tokens_to_string()
    • convert_string_to_tokens()

preprocessor

class Preprocessor

Bases: object

pipe(operation, args=None)

Add an operation and its arguments to pipe in the preprocessor.

Parameters:

• operation (callable) – text preprocessing function

• args (dict of arguments) –

Return type:

None

static build_pipeline(operation_list)

Build sklearn pipeline from a operation list.

Parameters:

operation_list (iterable) – list of __operations of preprocessing

Return type:

sklearn.pipeline.Pipeline

run(text)

Apply pipeline to text.

Parameters:

text (string) – text to preprocess

Return type:

string

textloader

class TextLoader(text_column='text', encoding='utf-8', file_format=None, use_dask=True)

Bases: object

read_text(files_path, file_format=None, encoding=None, compute_to_pandas=True, preprocessor=None)

Read the text files stored in files_path.

Parameters:

• files_path (string | list[string]) – single or multiple files path

• file_format (string) – Format of the files to be loaded, to be selected among csv, json, parquet or txt

• encoding (Optional[str]) – encoding of the text to be loaded, can be utf-8 or latin-1 for example

• compute_to_pandas (bool) – True if user wants Dask Dataframe to be computed as pandas DF, False otherwise

• preprocessor (nlpretext.preprocessor.Preprocessor) – NLPretext preprocessor can be specified to pre-process text after loading

Return type:

dask.dataframe | pandas.DataFrame