Skip to content

[Nori] Add metadata support for Korean analyzer tokens #14940

Description

@twosom

Description

Overview

This issue proposes adding metadata support to the Nori Korean analyzer, allowing users to attach additional information to dictionary words that can be accessed during text analysis.

Background and Motivation

Currently, the Nori analyzer allows users to register words in a custom dictionary, but there's no way to associate additional information with these words. By supporting metadata, we can enable:

  • Attaching semantic category information to words (e.g., "Java" -> "programming language")
  • Preserving information for compound words
  • Custom tagging and classification
  • Domain-specific annotations

Proposed Implementation

  1. Add metadata support to Token class
  2. Create MetadataAttribute and its implementation
  3. Extend user dictionary format with a metadata separator (>>)
  4. Preserve metadata during compound word decomposition

Usage Example

User dictionary:

자바 >> computer language 
java >> computer language 
엘라스틱서치 엘라스틱 서치 >> search engine

and this should be

input : 자바

/* Output:
Term: 자바
Metadata: computer language
POS: NNG
---

input : 엘라스틱서치

/* Output:
Term: 엘라스틱서치
Metadata: search engine
Position Increment: 1
Position Length: 2
---
Term: 엘라스틱
Metadata: search engine
Position Increment: 0
Position Length: 1
---
Term: 서치
Metadata: search engine
Position Increment: 1
Position Length: 1
---

Benefits

  1. Enhanced information modeling: Attach additional information to words to improve search quality
  2. Domain-specific analysis: Define metadata relevant to specific domains
  3. Custom dictionary extension: Add capabilities while maintaining backward compatibility

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions