augmentation
text_augmentation
Bases:
ValueError
- augment_text(text, method, stopwords=None, entities=None)[source]
Given a text with or without associated entities, generate a new text by modifying some words in the initial one, modifications depend on the chosen method (substitution with synonym, addition, deletion). If entities are given as input, they will remain unchanged. If you want some words other than entities to remain unchanged, specify it within the stopwords argument.
- Parameters:
text (string) –
method ({'wordnet_synonym', 'aug_sub_bert'}) – augmenter to use (‘wordnet_synonym’ or ‘aug_sub_bert’)
stopwords (list, optional) – list of words to freeze throughout the augmentation
entities (list, optional) –
entities associated to text if any, must be in the following format: [
- {
‘entity’: str, ‘word’: str, ‘startCharIndex’: int, ‘endCharIndex’: int
}, {
…
}
]
- Return type:
Augmented text and optional augmented entities
- process_entities_and_text(entities, text, augmented_text)[source]
Given a list of initial entities, verify that they have not been altered by the data augmentation operation and are still in the augmented text.
- Parameters:
entities (list) –
entities associated to text, must be in the following format: [
- {
‘entity’: str, ‘word’: str, ‘startCharIndex’: int, ‘endCharIndex’: int
}, {
…
}
]
text (str) – initial text
augmented_text (str) – new text resulting of data augmentation operation
- Return type:
Augmented text and entities with their updated position in augmented text
- are_entities_in_augmented_text(entities, augmented_text)[source]
Given a list of entities, check if all the words associated to each entity are still present in augmented text.
- Parameters:
entities (list) –
entities associated to initial text, must be in the following format: [
- {
‘entity’: str, ‘word’: str, ‘startCharIndex’: int, ‘endCharIndex’: int
}, {
…
}
]
augmented_text (str) –
- Return type:
True if all entities are present in augmented text, False otherwise
- get_augmenter(method, stopwords=None)[source]
Initialize an augmenter depending on the given method.
- Parameters:
method (str (supported methods: wordnet_synonym and aug_sub_bert)) –
stopwords (list) – list of words to freeze throughout the augmentation
- Return type:
Initialized nlpaug augmenter
- get_augmented_entities(sentence_augmented, entities)[source]
Get entities with updated positions (start and end) in augmented text.
- Parameters:
sentence_augmented (str) – augmented text
entities (list) –
entities associated to initial text, must be in the following format: [
- {
‘entity’: str, ‘word’: str, ‘startCharIndex’: int, ‘endCharIndex’: int
}, {
…
}
]
- Return type:
Entities with updated positions related to augmented text
- clean_sentence_entities(text, entities)[source]
Paired entities check to remove nested entities, the longest entity is kept.
- Parameters:
text (str) – augmented text
entities (list) –
entities associated to augmented text, must be in the following format: [
- {
‘entity’: str, ‘word’: str, ‘startCharIndex’: int, ‘endCharIndex’: int
}, {
…
}
]
- Return type:
Cleaned entities
- check_interval_included(element1, element2)[source]
Comparison of two entities on start and end positions to find if they are nested.
- Parameters:
element1 (dict) –
element2 (dict) –
both of them in the following format {
’entity’: str, ‘word’: str, ‘startCharIndex’: int, ‘endCharIndex’: int
}
- Return type:
Optional[Tuple[Dict[str,Any],Dict[str,Any]]]- Returns:
If there is an entity to remove among the two returns a tuple
(element to remove, element to keep).
If not, returns None