LevenshteinDist - script and chart function

Overview

Open the Data load editor and add the load script below to a new tab.

The load script contains:

• A dataset which is loaded into a data table called Example.

• One field in the data table called InputText.

Load script

Example:
Load * inline [
InputText
Sliver
SSiver
SSiveer
];

Results

Load the data and open a sheet. Create a new table and add this field as a dimension:

• InputText

Create the following measure:

• =LevenshteinDist('Silver', InputText), to calculate the minimum number of single character edits required to change the string value for InputText to the word Silver.

The output of the LevenshteinDist function returns the number of changes required to change the InputText to the expected text, Silver. For example, the first row requires two changes to modify the word Sliver to Silver. The second row requires 3 changes: 1) Delete the extra character S. 2) Delete the extra character e. 3) Insert a new character l.

Overview

This example consolidates product names from different systems. The product names do not always use the same spelling due to typos, abbreviations, spacing, or other variations. Using the LevenshteinDist function, you can measure the similarity between two product names and identify which ones likely refer to the same product, even if the names are not identical.

Open the Data load editor and add the load script below to a new tab.

The load script contains:

• A dataset which is loaded into a data table called Example.

• The following fields in the data table:

  • ProductA

  • ProductB

Load script

Example:
Load * inline [
ProductA, ProductB
Coca Cola 330ml, CocaCola 330 ml
Pepsi 500 ml, Pepsi 500ml
Sprite Zero 600 ml, SpriteZero600ml
Red Bull 250ml, Redbull 250ml
];

Results

Load the data and open a sheet. Create a new table and add these fields as dimensions:

• ProductA

• ProductB

Create the following measure:

• =LevenshteinDist(ProductA, ProductB), to calculate the minimum number of single character edits required to change the string value for ProductB to match ProductA.

The Levenshtein distance is a type of fuzzy matching that is widely used as part of spell checkers, optical character recognition, and correction systems in areas such as customer data management, inventory systems, and document processing, where slight variations in text occur frequently.

Load script

Example:
Load *, recno() as ID;
Load 'Silver' as String_1,* inline [
String_2
Sliver
SSiver
SSiveer ];

Example:
Load *, recno()+3 as ID;
Load 'Gold' as String_1,* inline [
String_2
Bold
Bool
Bond ];

Example:
Load *, recno()+6 as ID;
Load 'Ove' as String_1,* inline [
String_2
Ove
Uve
Üve ];

Example:
Load *, recno()+9 as ID;
Load 'ABC' as String_1,* inline [
String_2
DEFG
abc
ビビビ ];

set nullinterpret = '<NULL>';
Example:
Load *, recno()+12 as ID;
Load 'X' as String_1,* inline [
String_2
''
<NULL>
1 ];

R1:
Load
ID,
String_1,
String_2,
LevenshteinDist(String_1, String_2) as LevenshteinDistance
resident Example;

Drop table Example;