TransWikia.com

NLP Classification labels have many similarirites, converge to and replace to only have one

Data Science Asked on June 12, 2021

I been trying to use the fuzzywuzzy library in Python to find the percentage similarity between strings in the labels. The problem I am having is that there is still many strings that are really similar even when I try to do a find and replace.

I am wondering if there is a method that anyone here has used in order to clean up labels. To give an example. I have these labels that look really identical:

 'Cable replaced',
 'Cable replaced.',
 'Camera is up and recording',
 'Chat closed due to inactivity.',
 'Closing as duplicate',
 'Closing as duplicate.',
 'Closing duplicate ticket.',
 'Closing ticket.',

Ideally I want to be able to find and replace by a common string so we only have say one instance of ‘closing as duplicate’. Any thoughts or suggestions are greatly appreciated.

To provide a more thorough example. Here is what I am trying to do:

import fuzzywuzzy
from fuzzywuzzy import process
import chardet

res = h['resolution'].unique()
res.sort()
res

'All APs are up and stable hence resoling TT  Logs are updated in WL',
'Asset returned to IT hub closing ticket.',
'Auto Resolved - No reply from requester', 'Cable replaced',
'Cable replaced.', 'Camera is up and recording',
'Chat closed due to inactivity.', 'Closing as duplicate',
'Closing as duplicate.', 'Closing duplicate ticket.',
'Closing ticket.', 'Completed', 'Connection to IDF restored',

Oh look at that, lets see if we can find strings like ‘cable replaced’.

# get the top 10 closest matches to "cable replaced"
matches = fuzzywuzzy.process.extract("cable replaced", res, limit=10, scorer=fuzzywuzzy.fuzz.token_sort_ratio)

# take a look at them
matches

[('cable replaced', 100),
 ('cable replaced.', 100),
 ('replaced cable', 100),
 ('replaced scanner cable', 78),
 ('replaced scanner cable.', 78),
 ('scanner cable replaced', 78),
 ('battery replaced', 73),
 ('replaced', 73),
 ('replaced battery', 73),
 ('replaced battery.', 73)]

Hmmm, perhaos I should create a function to replace strings that have a similarity score greater than say 90.

# function to replace rows in the provided column of the provided dataframe
# that match the provided string above the provided ratio with the provided string
def replace_matches_in_column(df, column, string_to_match, min_ratio = 90):
    # get a list of unique strings
    strings = df[column].unique()
    
    # get the top 10 closest matches to our input string
    matches = fuzzywuzzy.process.extract(string_to_match, strings, 
                                         limit=10, scorer=fuzzywuzzy.fuzz.token_sort_ratio)

    # only get matches with a ratio > 90
    close_matches = [matches[0] for matches in matches if matches[1] >= min_ratio]

    # get the rows of all the close matches in our dataframe
    rows_with_matches = df[column].isin(close_matches)

    # replace all rows with close matches with the input matches 
    df.loc[rows_with_matches, column] = string_to_match
    
    # let us know the function's done
    print("All done!")

# use the function we just wrote to replace close matches to "cable replaced" with "cable replaced"
replace_matches_in_column(df=h, column='resolution', string_to_match="cable replaced")

# get all the unique values in the 'City' column
res = h['resolution'].unique()

# sort them alphabetically and then take a closer look
res.sort()
res

'auto resolved - no reply from requester', 'battery replaced',
       'cable replaced', 'camera is up and recording',
       'chat closed due to inactivity.', 'check ok',

Great! Now I only have one instance of ‘cable replaced’. Lets verify that

# get the top 10 closest matches to "cable replaced"
matches = fuzzywuzzy.process.extract("cable replaced", res, limit=10, scorer=fuzzywuzzy.fuzz.token_sort_ratio)

# take a look at them
matches

[('cable replaced', 100),
 ('replaced scanner cable', 78),
 ('replaced scanner cable.', 78),
 ('scanner cable replaced', 78),
 ('battery replaced', 73),
 ('replaced', 73),
 ('replaced battery', 73),
 ('replaced battery.', 73),
 ('replaced.', 73),
 ('hardware replaced', 71)]

Yep! Looking good. Now, this example works great but as you can see it is rather manual. I would ideally like to automate this for all the strings in my resolution column. Any ideas?

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP