Description
Overview
This issue proposes adding metadata support to the Nori Korean analyzer, allowing users to attach additional information to dictionary words that can be accessed during text analysis.
Background and Motivation
Currently, the Nori analyzer allows users to register words in a custom dictionary, but there's no way to associate additional information with these words. By supporting metadata, we can enable:
- Attaching semantic category information to words (e.g., "Java" -> "programming language")
- Preserving information for compound words
- Custom tagging and classification
- Domain-specific annotations
Proposed Implementation
- Add metadata support to Token class
- Create
MetadataAttribute and its implementation
- Extend user dictionary format with a metadata separator (
>>)
- Preserve metadata during compound word decomposition
Usage Example
User dictionary:
자바 >> computer language
java >> computer language
엘라스틱서치 엘라스틱 서치 >> search engine
and this should be
input : 자바
/* Output:
Term: 자바
Metadata: computer language
POS: NNG
---
input : 엘라스틱서치
/* Output:
Term: 엘라스틱서치
Metadata: search engine
Position Increment: 1
Position Length: 2
---
Term: 엘라스틱
Metadata: search engine
Position Increment: 0
Position Length: 1
---
Term: 서치
Metadata: search engine
Position Increment: 1
Position Length: 1
---
Benefits
- Enhanced information modeling: Attach additional information to words to improve search quality
- Domain-specific analysis: Define metadata relevant to specific domains
- Custom dictionary extension: Add capabilities while maintaining backward compatibility
Description
Overview
This issue proposes adding metadata support to the Nori Korean analyzer, allowing users to attach additional information to dictionary words that can be accessed during text analysis.
Background and Motivation
Currently, the Nori analyzer allows users to register words in a custom dictionary, but there's no way to associate additional information with these words. By supporting metadata, we can enable:
Proposed Implementation
MetadataAttributeand its implementation>>)Usage Example
User dictionary:
and this should be
input : 자바
input : 엘라스틱서치
Benefits