Skip to content

A curated list of resources dedicated to Python libraries, pre-trained models, dictionaries, and corpora of NLP for Japanese

License

Notifications You must be signed in to change notification settings

forcs-ai/awesome-japanese-nlp-resources

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

awesome-japanese-nlp-resources

Awesome License: CC0-1.0 CC0

A curated list of resources dedicated to Python libraries, pre-trained models, dictionaries, and corpora of NLP for Japanese

This list includes 459 Japanese NLP repositories. A tool for searching these repositories is available on Hugging Face Spaces.

Your contributions are always welcome! Please read the Contribution guidelines before contributing.

Resources that are not available on GitHub are added to the wiki.

English | 日本語 (Japanese) | 繁體中文 (Chinese) | 简体中文 (Chinese)

The latest additions 🎉

ChatGPT

  • chatvrm - ChatVRMはブラウザで簡単に3Dキャラクターと会話ができるデモアプリケーションです。

Updated on May 02, 2023

Contents

Python library

Morphology analysis

  • sudachi.rs - SudachiPy 0.6* and above are developed as Sudachi.rs.
  • Janome - Japanese morphological analysis engine written in pure Python
  • mecab-python3 - mecab-python. mecab-python. you can find original version here:http://taku910.github.io/mecab/
  • mecab - This repository is for building Windows 64-bit MeCab binary and improving MeCab Python binding.
  • fugashi - A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
  • nagisa - A Japanese tokenizer based on recurrent neural networks
  • pyknp - A Python Module for JUMAN++/KNP
  • Mykytea-python - Python wrapper for KyTea
  • konoha - Konoha: Simple wrapper of Japanese Tokenizers
  • natto-py - natto-py combines the Python programming language with MeCab, the part-of-speech and morphological analyzer for the Japanese language.
  • rakutenma-python - Rakuten MA (Python version)
  • python-vaporetto - Vaporetto is a fast and lightweight pointwise prediction based tokenizer. This is a Python wrapper for Vaporetto.
  • dango - An easy to use tokenizer for Japanese text, aimed at language learners and non-linguists
  • rhoknp - Yet another Python binding for Juman++/KNP
  • python-vibrato - Viterbi-based accelerated tokenizer (Python wrapper)
Name downloads/week total downloads stars
SudachiPy Downloads Downloads GitHub Repo stars
Janome Downloads Downloads GitHub Repo stars
mecab-python3 Downloads Downloads GitHub Repo stars
mecab Downloads Downloads GitHub Repo stars
fugashi Downloads Downloads GitHub Repo stars
nagisa Downloads Downloads GitHub Repo stars
pyknp Downloads Downloads GitHub Repo stars
Mykytea-python Downloads Downloads GitHub Repo stars
konoha Downloads Downloads GitHub Repo stars
natto-py Downloads Downloads GitHub Repo stars
rakutenma-python Downloads Downloads GitHub Repo stars
python-vaporetto Downloads Downloads GitHub Repo stars
dango Downloads Downloads GitHub Repo stars
rhoknp Downloads Downloads GitHub Repo stars
python-vibrato Downloads Downloads GitHub Repo stars

Parsing

  • ginza - A Japanese NLP Library using spaCy as framework based on Universal Dependencies
  • cabocha - Yet Another Japanese Dependency Structure Analyzer
  • UniDic2UD - Tokenizer POS-tagger Lemmatizer and Dependency-parser for modern and contemporary Japanese
  • camphr - Camphr - NLP libary for creating pipeline components
  • SuPar-UniDic - Tokenizer POS-tagger Lemmatizer and Dependency-parser for modern and contemporary Japanese with BERT models
  • depccg - A* CCG Parser with a Supertag and Dependency Factored Model
  • bertknp - A Japanese dependency parser based on BERT
  • esupar - Tokenizer POS-Tagger and Dependency-parser with BERT/RoBERTa/DeBERTa models for Japanese and other languages
  • yomikata - Heteronym disambiguation library using a fine-tuned BERT model.
Name downloads/week total downloads stars
ginza Downloads Downloads GitHub Repo stars
cabocha Downloads Downloads GitHub Repo stars
UniDic2UD Downloads Downloads GitHub Repo stars
camphr Downloads Downloads GitHub Repo stars
SuPar-UniDic Downloads Downloads GitHub Repo stars
depccg Downloads Downloads GitHub Repo stars
bertknp - - GitHub Repo stars
esupar Downloads Downloads GitHub Repo stars
yomikata Downloads Downloads GitHub Repo stars

Converter

  • pykakasi - Lightweight converter from Japanese Kana-kanji sentences into Kana-Roman.
  • cutlet - Japanese to romaji converter in Python
  • alphabet2kana - Convert English alphabet to Katakana
  • Convert-Numbers-to-Japanese - Converts Arabic numerals, or 'western' style numbers, to a Japanese context.
  • mozcpy - Mozc for Python: Kana-Kanji converter
  • jamorasep - Japanese text parser to separate Hiragana/Katakana string into morae (syllables).
  • text2phoneme - 日本語文を音素列へ変換するスクリプト
Name downloads/week total downloads stars
pykakasi Downloads Downloads GitHub Repo stars
cutlet Downloads Downloads GitHub Repo stars
alphabet2kana Downloads Downloads GitHub Repo stars
Convert-Numbers-to-Japanese - - GitHub Repo stars
mozcpy Downloads Downloads GitHub Repo stars
jamorasep Downloads Downloads GitHub Repo stars
text2phoneme - - GitHub Repo stars

Preprocessor

  • neologdn - Japanese text normalizer for mecab-neologd
  • jaconv - Pure-Python Japanese character interconverter for Hiragana, Katakana, Hankaku, and Zenkaku
  • mojimoji - A fast converter between Japanese hankaku and zenkaku characters
  • text-cleaning - A powerful text cleaner for Japanese web texts
  • HojiChar - 複数の前処理を構成して管理するテキスト前処理ツール
Name downloads/week total downloads stars
neologdn Downloads Downloads GitHub Repo stars
jaconv Downloads Downloads GitHub Repo stars
mojimoji Downloads Downloads GitHub Repo stars
text-cleaning - - GitHub Repo stars
HojiChar Downloads Downloads GitHub Repo stars

Sentence spliter

  • Bunkai - Sentence boundary disambiguation tool for Japanese texts (日本語文境界判定器)
  • japanese-sentence-breaker - Japanese Sentence Breaker
  • sengiri - Yet another sentence-level tokenizer for the Japanese text
  • budoux - Standalone. Small. Language-neutral. BudouX is the successor to Budou, the machine learning powered line break organizer tool.
  • ja_sentence_segmenter - japanese sentence segmentation library for python
  • hasami - A tool to perform sentence segmentation on Japanese text
  • kuzukiri - Japanese Text Segmenter for Python written in Rust
  • ja-senter-benchmark - Comparison of Japanese Sentence Segmentation Tools
Name downloads/week total downloads stars
bunkai Downloads Downloads GitHub Repo stars
japanese-sentence-breaker Downloads Downloads GitHub Repo stars
sengiri Downloads Downloads GitHub Repo stars
budoux Downloads Downloads GitHub Repo stars
ja_sentence_segmenter Downloads Downloads GitHub Repo stars
hasami Downloads Downloads GitHub Repo stars
kuzukiri Downloads Downloads GitHub Repo stars
ja-senter-benchmark - - GitHub Repo stars

Sentiment analysis

  • oseti - Dictionary based Sentiment Analysis for Japanese
  • negapoji - Japanese negative positive classification.日本語文書のネガポジを判定。
  • pymlask - Emotion analyzer for Japanese text
  • asari - Japanese sentiment analyzer implemented in Python.
Name downloads/week total downloads stars
oseti Downloads Downloads GitHub Repo stars
negapoji - - GitHub Repo stars
pymlask Downloads Downloads GitHub Repo stars
asari Downloads Downloads GitHub Repo stars

Machine translation

  • jparacrawl-finetune - An example usage of JParaCrawl pre-trained Neural Machine Translation (NMT) models.
  • JASS - JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation (LREC2020) & Linguistically Driven Multi-Task Pre-Training for Low-Resource Neural Machine Translation (ACM TALLIP)
  • PheMT - A phenomenon-wise evaluation dataset for Japanese-English machine translation robustness. The dataset is based on the MTNT dataset, with additional annotations of four linguistic phenomena; Proper Noun, Abbreviated Noun, Colloquial Expression, and Variant. COLING 2020.
  • VISA - An ambiguous subtitles dataset for visual scene-aware machine translation
Name downloads/week total downloads stars
jparacrawl-finetune - - GitHub Repo stars
JASS - - GitHub Repo stars
PheMT - - GitHub Repo stars
VISA - - GitHub Repo stars

Named entity recognition

  • namaco - Character Based Named Entity Recognition.
  • entitypedia - Entitypedia is an Extended Named Entity Dictionary from Wikipedia.
  • noyaki - Converts character span label information to tokenized text-based label information.
  • bert-japanese-ner-finetuning - Code to perform finetuning of the BERT model. BERTモデルのファインチューニングで固有表現抽出用タスクのモデルを作成・使用するサンプルです
  • joint-information-extraction-hs - 詳細なアノテーション基準に基づく症例報告コーパスからの固有表現及び関係の抽出精度の推論を行うコード
Name downloads/week total downloads stars
namaco - - GitHub Repo stars
entitypedia - - GitHub Repo stars
noyaki Downloads Downloads GitHub Repo stars
bert-japanese-ner-finetuning - - GitHub Repo stars
joint-information-extraction-hs - - GitHub Repo stars

OCR

  • Manga OCR - About Optical character recognition for Japanese text, with the main focus being Japanese manga
  • mokuro - Read Japanese manga inside browser with selectable text.
  • handwritten-japanese-ocr - Handwritten Japanese OCR demo using touch panel to draw the input text using Intel OpenVINO toolkit
  • OCR_Japanease - 日本語OCR
  • ndlocr_cli - NDLOCRのアプリケーション
  • donut - Official Implementation of OCR-free Document Understanding Transformer (Donut) and Synthetic Document Generator (SynthDoG), ECCV 2022
  • JMTrans - manga translator - get japanese manga from url to translate manga image
  • Kindai-OCR - OCR system for recognizing modern Japanese magazines
  • text_recognition - NDLOCR用テキスト認識モジュール
Name downloads/week total downloads stars
manga-ocr Downloads Downloads GitHub Repo stars
mokuro Downloads Downloads GitHub Repo stars
handwritten-japanese-ocr - - GitHub Repo stars
OCR_Japanease - - GitHub Repo stars
ndlocr_cli - - GitHub Repo stars
donut Downloads Downloads GitHub Repo stars
JMTrans - - GitHub Repo stars
Kindai-OCR - - GitHub Repo stars
text_recognition - - GitHub Repo stars

Tool for pretrained models

Name downloads/week total downloads stars
JGLUE - - GitHub Repo stars
ginza-transformers Downloads Downloads GitHub Repo stars
t5_japanese_dialogue_generation - - GitHub Repo stars
japanese_text_classification - - GitHub Repo stars
Japanese-BERT-Sentiment-Analyzer - - GitHub Repo stars
jmlm_scoring - - GitHub Repo stars
allennlp-shiba-model Downloads Downloads GitHub Repo stars
evaluate_japanese_w2v - - GitHub Repo stars
gector-ja - - GitHub Repo stars
Japanese-BPEEncoder - - GitHub Repo stars
Japanese-BPEEncoder_V2 - - GitHub Repo stars
transformer-copy - - GitHub Repo stars
japanese-stable-diffusion - - GitHub Repo stars
nagisa_bert Downloads Downloads GitHub Repo stars
prefix-tuning-gpt - - GitHub Repo stars
JGLUE-benchmark - - GitHub Repo stars

Others

  • namedivider-python - A tool for dividing the Japanese full name into a family name and a given name.
  • asa-python - A curated list of resources dedicated to Python libraries of NLP for Japanese
  • python_asa - python版日本語意味役割付与システム(ASA)
  • toiro - A comparison tool of Japanese tokenizers
  • ja-timex - 自然言語で書かれた時間情報表現を抽出/規格化するルールベースの解析器
  • JapaneseTokenizers - A set of metrics for feature selection from text data
  • daaja - This repository has implementations of data augmentation for NLP for Japanese.
  • accel-brain-code - The purpose of this repository is to make prototypes as case study in the context of proof of concept(PoC) and research and development(R&D) that I have written in my website. The main research topics are Auto-Encoders in relation to the representation learning, the statistical machine learning for energy-based models, adversarial generation net…
  • kyoto-reader - A processor for KyotoCorpus, KWDLC, and AnnotatedFKCCorpus
  • nlplot - Visualization Module for Natural Language Processing
  • rake-ja - Rapid Automatic Keyword Extraction algorithm for Japanese
  • jel - Japanese Entity Linker.
  • MedNER-J - Latest version of MedEX/J (Japanese disease name extractor)
  • zunda-python - Zunda: Japanese Enhanced Modality Analyzer client for Python.
  • AIO2_DPR_baseline - https://www.nlp.ecei.tohoku.ac.jp/projects/aio/
  • showcase - A PyTorch implementation of the Japanese Predicate-Argument Structure (PAS) analyser presented in the paper of Matsubayashi & Inui (2018) with some improvements.
  • darts-clone-python - Darts-clone python binding
  • jrte-corpus_example - Example codes for Japanese Realistic Textual Entailment Corpus
  • desuwa - Feature annotator to morphemes and phrases based on KNP rule files (pure-Python)
  • HotPepperGourmetDialogue - Restaurant Search System through Dialogue in Japanese.
  • nlp-recipes-ja - Samples codes for natural language processing in Japanese
  • Japanese_nlp_scripts - Small example scripts for working with Japanese texts in Python
  • DNorm-J - Japanese version of DNorm
  • pyknp-eventgraph - EventGraph is a development platform for high-level NLP applications in Japanese.
  • ishi - Ishi: A volition classifier for Japanese
  • python-npylm - ベイズ階層言語モデルによる教師なし形態素解析
  • python-npycrf - 条件付確率場とベイズ階層言語モデルの統合による半教師あり形態素解析
  • unsupervised-pos-tagging - 教師なし品詞タグ推定
  • negima - Negima is a Python package to extract phrases in Japanese text by using the part-of-speeches based rules you defined.
  • YouyakuMan - Extractive summarizer using BertSum as summarization model
  • japanese-numbers-python - A parser for Japanese number (Kanji, arabic) in the natural language.
  • kantan - Lookup japanese words by radical patterns
  • make-meidai-dialogue - Get Japanese dialogue corpus
  • japanese_summarizer - A summarizer for Japanese articles.
  • chirptext - ChirpText is a collection of text processing tools for Python.
  • yubin - Japanese Address Munger
  • jawiki-cleaner - Japanese Wikipedia Cleaner
  • japanese2phoneme - A python library to convert Japanese to phoneme.
  • anlp_nlp2021_d3-1 - This repository contains codes related to the experiments in "An Experimental Evaluation of Japanese Tokenizers for Sentiment-Based Text Classification"
  • aozora_classification - About This project aims to classify Japanese sentence to how well similar to some Japanese classical writers, such as Soseki Natsume, Ogai Mori, Ryunosuke Akutagawa and so on.
  • aozora-corpus-generator - Generates plain or tokenized text files from the Aozora Bunko
  • JLM - A fast LSTM Language Model for large vocabulary language like Japanese and Chinese
  • NTM - Testing of Neural Topic Modeling for Japanese articles
  • EN-JP-ML-Lexicon - This is a English-Japanese lexicon for Machine Learning and Deep Learning terminology.
  • text-generation - Easy-to-use scripts to fine-tune GPT-2-JA with your own texts, to generate sentences, and to tweet them automatically.
  • chainer_nic - Neural Image Caption (NIC) on chainer, its pretrained models on English and Japanese image caption datasets.
  • unihan-lm - The official repository for "UnihanLM: Coarse-to-Fine Chinese-Japanese Language Model Pretraining with the Unihan Database", AACL-IJCNLP 2020
  • mbart-finetuning - Code to perform finetuning of the mBART model.
  • xvector_jtubespeech - xvector model on jtubespeech
  • TinySegmenterMaker - TinySegmenter用の学習モデルを自作するためのツール.
  • Grongish - 日本語とグロンギ語の相互変換スクリプト
  • WordCloud-Japanese - WordCloudでの日本語文章をMecab(形態素解析エンジン)を使用せずに形態素解析チックな表示を実現するスクリプト
  • snark - 日本語ワードネットを利用したDBアクセスライブラリ
  • toEmoji - 日本語文を絵文字だけの文に変換するなにか
  • termextract - - 専門用語抽出アルゴリズムの実装の練習
  • JDT-with-KenLM-scoring - Japanese-Dialog-Transformerの応答候補に対して、KenLMによるN-gram言語モデルでスコアリングし、フィルタリング若しくはリランキングを行う。
  • mixture-of-unigram-model - Mixture of Unigram Model and Infinite Mixture of Unigram Model in Python. (混合ユニグラムモデルと無限混合ユニグラムモデル)
  • hidden-markov-model - Hidden Markov Model (HMM) and Infinite Hidden Markov Model (iHMM) in Python. (隠れマルコフモデルと無限隠れマルコフモデル)
  • Ngram-language-model - Ngram language model in Python. (Nグラム言語モデル)
  • ASRDeepSpeech - Automatic Speech Recognition with deepspeech2 model in pytorch with support from Zakuro AI.
  • neural_ime - Neural IME: Neural Input Method Engine
  • neural_japanese_transliterator - Can neural networks transliterate Romaji into Japanese correctly?
  • tinysegmenter - tokenizer specified for Japanese
  • AugLy-jp - Data Augmentation for Japanese Text on AugLy
  • furigana4epub - A Python script for adding furigana to Japanese epub books using Mecab and Unidic.
  • PyKatsuyou - Japanese verb/adjective inflections tool
  • jageocoder - Pure Python Japanese address geocoder
  • pygeonlp - pygeonlp, A python module for geotagging Japanese texts.
  • nksnd - New kana-kanji conversion engine
  • JaMIE - A Japanese Medical Information Extraction Toolkit
  • fasttext-vs-word2vec-on-twitter-data - fasttextとword2vecの比較と、実行スクリプト、学習スクリプトです
  • minimal-search-engine - 最小のサーチエンジン/PageRank/tf-idf
  • 5ch-analysis - 5chの過去ログをスクレイピングして、過去流行った単語(ex, 香具師, orz)などを追跡調査
  • tweet_extructor - Twitter日本語評判分析データセットのためのツイートダウンローダ
  • japanese-word-aggregation - Aggregating Japanese words based on Juman++ and ConceptNet5.5
  • jinf - A Japanese inflection converter
  • kwja - A unified language analyzer for Japanese
  • mlm-scoring-transformers - Reproduced package based on Masked Language Model Scoring (ACL2020).
  • ClipCap-for-Japanese - [PyTorch] ClipCap for Japanese
  • SAT-for-Japanese - [PyTorch] Show, Attend and Tell for Japanese
  • cihai - Python library for CJK (Chinese, Japanese, and Korean) language dictionary
  • marine - MARINE : Multi-task leaRnIng-based JapaNese accent Estimation
  • whisper-asr-finetune - Finetuning Whisper ASR model
  • japanese_chatbot - A PyTorch Implementation of japanese chatbot using BERT and Transformer's decoder
  • radicalchar - 部首文字正規化ライブラリ
  • akaza - Yet another Japanese IME for IBus/Linux
  • posuto - Japanese postal code data.
  • tacotron2-japanese - Tacotron2 implementation of Japanese
  • ibus-hiragana - ひらがなIME for IBus
  • furiganapad - ふりがなパッド
  • chikkarpy - Japanese synonym library
  • ja-tokenizer-docker-py - Mecab + NEologd + Docker + Python3
  • JapaneseEmbeddingEval - JapaneseEmbeddingEval
  • gptuber-by-langchain - GPTがYouTuberをやります
  • shuwa - Extend GNOME On-Screen Keyboard for Input Methods
  • japanese-nli-model - This repository provides the code for Japanese NLI model, a fine-tuned masked language model.
  • tra-fugu - A tool for Japanese-English translation and English-Japanese translation by using FuguMT
  • fugumt - ぷるーふおぶこんせぷと で公開した機械翻訳エンジンを利用する翻訳環境です。 フォームに入力された文字列の翻訳、PDFの翻訳が可能です。
  • JaSPICE - JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models
  • Retrieval-based-Voice-Conversion-WebUI-JP-localization - jp-localization
  • pyopenjtalk - Python wrapper for OpenJTalk
Name downloads/week total downloads stars
namedivider-python Downloads Downloads GitHub Repo stars
asa-python Downloads Downloads GitHub Repo stars
python_asa - - GitHub Repo stars
toiro Downloads Downloads GitHub Repo stars
ja-timex Downloads Downloads GitHub Repo stars
JapaneseTokenizers - - GitHub Repo stars
daaja Downloads Downloads GitHub Repo stars
accel-brain-code Downloads Downloads GitHub Repo stars
JGLUE - - GitHub Repo stars
kyoto-reader Downloads Downloads GitHub Repo stars
nlplot Downloads Downloads GitHub Repo stars
rake-ja - - GitHub Repo stars
jel Downloads Downloads GitHub Repo stars
MedNER-J - - GitHub Repo stars
zunda-python Downloads Downloads GitHub Repo stars
AIO2_DPR_baseline - - GitHub Repo stars
showcase Downloads Downloads GitHub Repo stars
darts-clone-python Downloads Downloads GitHub Repo stars
jrte-corpus_example - - GitHub Repo stars
desuwa Downloads Downloads GitHub Repo stars
HotPepperGourmetDialogue - - GitHub Repo stars
nlp-recipes-ja - - GitHub Repo stars
Japanese_nlp_scripts - - GitHub Repo stars
DNorm-J - - GitHub Repo stars
pyknp-eventgraph Downloads Downloads GitHub Repo stars
ishi Downloads Downloads GitHub Repo stars
python-npylm - - GitHub Repo stars
python-npycrf - - GitHub Repo stars
unsupervised-pos-tagging - - GitHub Repo stars
negima Downloads Downloads GitHub Repo stars
YouyakuMan - - GitHub Repo stars
japanese-numbers-python Downloads Downloads GitHub Repo stars
kantan - - GitHub Repo stars
make-meidai-dialogue - - GitHub Repo stars
japanese_summarizer - - GitHub Repo stars
chirptext Downloads Downloads GitHub Repo stars
yubin Downloads Downloads GitHub Repo stars
jawiki-cleaner Downloads Downloads GitHub Repo stars
japanese2phoneme Downloads Downloads GitHub Repo stars
anlp_nlp2021_d3-1 - - GitHub Repo stars
aozora_classification - - GitHub Repo stars
aozora-corpus-generator - - GitHub Repo stars
JLM - - GitHub Repo stars
NTM - - GitHub Repo stars
EN-JP-ML-Lexicon - - GitHub Repo stars
text-generation - - GitHub Repo stars
chainer_nic - - GitHub Repo stars
unihan-lm - - GitHub Repo stars
mbart-finetuning - - GitHub Repo stars
xvector_jtubespeech - - GitHub Repo stars
TinySegmenterMaker - - GitHub Repo stars
Grongish - - GitHub Repo stars
WordCloud-Japanese - - GitHub Repo stars
snark - - GitHub Repo stars
toEmoji - - GitHub Repo stars
termextract - - GitHub Repo stars
JDT-with-KenLM-scoring - - GitHub Repo stars
mixture-of-unigram-model - - GitHub Repo stars
hidden-markov-model - - GitHub Repo stars
Ngram-language-model - - GitHub Repo stars
ASRDeepSpeech - - GitHub Repo stars
neural_ime - - GitHub Repo stars
neural_japanese_transliterator - - GitHub Repo stars
tinysegmenter Downloads Downloads GitHub Repo stars
AugLy-jp Downloads Downloads GitHub Repo stars
furigana4epub Downloads Downloads GitHub Repo stars
PyKatsuyou Downloads Downloads GitHub Repo stars
jageocoder Downloads Downloads GitHub Repo stars
pygeonlp Downloads Downloads GitHub Repo stars
nksnd - - GitHub Repo stars
JaMIE - - GitHub Repo stars
fasttext-vs-word2vec-on-twitter-data - - GitHub Repo stars
minimal-search-engine - - GitHub Repo stars
5ch-analysis - - GitHub Repo stars
tweet_extructor - - GitHub Repo stars
japanese-word-aggregation - - GitHub Repo stars
jinf Downloads Downloads GitHub Repo stars
kwja Downloads Downloads GitHub Repo stars
mlm-scoring-transformers - - GitHub Repo stars
ClipCap-for-Japanese - - GitHub Repo stars
SAT-for-Japanese - - GitHub Repo stars
cihai Downloads Downloads GitHub Repo stars
marine Downloads Downloads GitHub Repo stars
whisper-asr-finetune - - GitHub Repo stars
japanese_chatbot - - GitHub Repo stars
radicalchar - - GitHub Repo stars
akaza - - GitHub Repo stars
posuto Downloads Downloads GitHub Repo stars
tacotron2-japanese - - GitHub Repo stars
ibus-hiragana - - GitHub Repo stars
furiganapad - - GitHub Repo stars
chikkarpy Downloads Downloads GitHub Repo stars
ja-tokenizer-docker-py - - GitHub Repo stars
JapaneseEmbeddingEval - - GitHub Repo stars
gptuber-by-langchain - - GitHub Repo stars
shuwa - - GitHub Repo stars
japanese-nli-model - - GitHub Repo stars
tra-fugu - - GitHub Repo stars
fugumt - - GitHub Repo stars
JaSPICE Downloads Downloads GitHub Repo stars
Retrieval-based-Voice-Conversion-WebUI-JP-localization - - GitHub Repo stars
pyopenjtalk Downloads Downloads GitHub Repo stars

C++

Morphology analysis

  • mecab - Yet another Japanese morphological analyzer
  • jumanpp - Juman++ (a Morphological Analyzer Toolkit)
  • kytea - The Kyoto Text Analysis Toolkit for word segmentation and pronunciation estimation, etc.
Name downloads/week total downloads stars
mecab - - GitHub Repo stars
jumanpp - - GitHub Repo stars
kytea - - GitHub Repo stars

Parsing

  • cabocha - Yet Another Japanese Dependency Structure Analyzer
  • knp - A Japanese Parser
Name downloads/week total downloads stars
cabocha - - GitHub Repo stars
knp - - GitHub Repo stars

Others

  • jsc - Joint source channel model for Japanese Kana Kanji conversion, Chinese pinyin input and CJE mixed input.
  • aquaskk - An input method without morphological analysis.
  • mozc - Mozc - a Japanese Input Method Editor designed for multi-platform
  • trimatch - Trimatch: An (Exact|Prefix|Approximate) String Matching Library
  • resembla - Resembla: Word-based Japanese similar sentence search library
Name downloads/week total downloads stars
jsc - - GitHub Repo stars
aquaskk - - GitHub Repo stars
mozc - - GitHub Repo stars
trimatch - - GitHub Repo stars
resembla - - GitHub Repo stars

Rust crate

Morphology analysis

  • lindera - A morphological analysis library.
  • vaporetto - Vaporetto: Very Accelerated POintwise pREdicTion based TOkenizer
  • goya - Japanese Morphological Analysis written in Rust
  • vibrato - vibrato: Viterbi-based accelerated tokenizer
  • yoin - A Japanese Morphological Analyzer written in pure Rust
  • mecab-rs - Safe Rust bindings for mecab a part-of-speech and morphological analyzer library
  • awabi - A morphological analyzer using mecab dictionary
Name downloads/week total downloads stars
lindera - Crates.io GitHub Repo stars
vaporetto - Crates.io GitHub Repo stars
goya - Crates.io GitHub Repo stars
vibrato - Crates.io GitHub Repo stars
yoin - Crates.io GitHub Repo stars
mecab-rs - Crates.io GitHub Repo stars
awabi - Crates.io GitHub Repo stars

Converter

  • wana_kana_rust - Utility library for checking and converting between Japanese characters - Hiragana, Katakana - and Romaji
  • unicode-jp-rs - A Rust library to convert Japanese Half-width-kana[半角カナ] and Wide-alphanumeric[全角英数] into normal ones
  • kana - [Mirror] CLI program for transliterating romaji text to either hiragana or katakana
Name downloads/week total downloads stars
wana_kana_rust - Crates.io GitHub Repo stars
unicode-jp-rs - Crates.io GitHub Repo stars
kana - - GitHub Repo stars

Search engine library

Name downloads/week total downloads stars
lindera-tantivy - Crates.io GitHub Repo stars
tantivy-vibrato - Crates.io GitHub Repo stars

Others

  • daachorse - A fast implementation of the Aho-Corasick algorithm using the compact double-array data structure in Rust.
  • find-simdoc - Finding all pairs of similar documents time- and memory-efficiently
  • crawdad - Rust library of natural language dictionaries using character-wise double-array tries.
  • tokenizer-speed-bench - Comparison code of various tokenizers
  • stringmatch-bench - Here provides benchmark tools to compare the performance of data structures for string matching.
  • vime - Using Vim as an input method for X11 apps
  • voicevox_core - 無料で使える中品質なテキスト読み上げソフトウェア、VOICEVOXのコア
  • akaza - Yet another Japanese IME for IBus/Linux
  • Jotoba - A free online, self-hostable, multilang Japanese dictionary.
  • dvorakjp-romantable - Google 日本語入力用DvorakJPローマ字テーブル / DvorakJP Roman Table for Google Japanese Input
  • niinii - Japanese glossator for assisted reading of text using Ichiran
Name downloads/week total downloads stars
daachorse - Crates.io GitHub Repo stars
find-simdoc - Crates.io GitHub Repo stars
crawdad - Crates.io GitHub Repo stars
tokenizer-speed-bench - - GitHub Repo stars
stringmatch-bench - - GitHub Repo stars
vime - - GitHub Repo stars
voicevox_core - - GitHub Repo stars
akaza - - GitHub Repo stars
Jotoba - - GitHub Repo stars
dvorakjp-romantable - - GitHub Repo stars
niinii - - GitHub Repo stars

JavaScript

Morphology analysis

  • kuromoji.js - JavaScript implementation of Japanese morphological analyzer
  • rakutenma - Rakuten MA - morphological analyzer (word segmentor + PoS Tagger) for Chinese and Japanese written purely in JavaScript. Resources
  • node-mecab-ya - Yet another mecab wrapper for nodejs
  • juman-bin - a User-Extensible Morphological Analyzer for Japanese. 日本語形態素解析システム
  • node-mecab-async - Asynchronous japanese morphological analyser using MeCab.
Name downloads/week total downloads stars
kuromoji.js npm npm GitHub Repo stars
rakutenma npm npm GitHub Repo stars
node-mecab-ya npm npm GitHub Repo stars
juman-bin npm npm GitHub Repo stars
node-mecab-async npm npm GitHub Repo stars

Converter

  • kuroshiro - Japanese language library for converting Japanese sentence to Hiragana, Katakana or Romaji with furigana and okurigana modes supported.
  • kuroshiro-analyzer-kuromoji - Kuromoji morphological analyzer for kuroshiro.
  • hepburn - Node.js module for converting Japanese Hiragana and Katakana script to, and from, Romaji using Hepburn romanisation
  • japanese-numerals-to-number - Converts Japanese Numerals into number
  • jslingua - Javascript libraries to process text: Arabic, Japanese, etc.
  • WanaKana - Javascript library for detecting and transliterating Hiragana <--> Katakana <--> Romaji
  • node-romaji-name - Normalize and fix common issues with Romaji-based Japanese names.
  • kyujitai.js - Utility collections for making Japanese text old-fashioned
Name downloads/week total downloads stars
kuroshiro npm npm GitHub Repo stars
kuroshiro-analyzer-kuromoji npm npm GitHub Repo stars
hepburn npm npm GitHub Repo stars
japanese-numerals-to-number npm npm GitHub Repo stars
jslingua npm npm GitHub Repo stars
WanaKana npm npm GitHub Repo stars
node-romaji-name npm npm GitHub Repo stars
kyujitai.js npm npm GitHub Repo stars

Others

  • bangumi-data - Raw data for Japanese Anime
  • yomichan - Japanese pop-up dictionary extension for Chrome and Firefox.
  • proofreading-tool - GUIで動作する文書校正ツール GUI tool for textlinting.
  • kanjigrid - A web-app displaying the 2200 kanji characters taught in James Heisig's "Remembering the Kanji", 6th edition.
  • japanese-toolkit - Monorepo for Kanji, Furigana, Japanese DB, and others
  • analyze-desumasu-dearu - 文の敬体(ですます調)、常体(である調)を解析するJavaScriptライブラリ
  • hatsuon - Japanese pitch accent utils
  • sentiment_ja_js - Sentiment Analysis in Japanese. sentiment_ja with JavaScript
  • mecab-ipadic-seed - mecab-ipadic seed dictionary reader
  • Japanese-Word-Of-The-Day - Well, a different Japanese word everyday.
  • oskim - Extend GNOME On-Screen Keyboard for Input Methods
  • tweetMapping - 東日本大震災発生から24時間以内につぶやかれたジオタグ付きツイートのデジタルアーカイブです。
Name downloads/week total downloads stars
bangumi-data npm npm GitHub Repo stars
yomichan - - GitHub Repo stars
proofreading-tool - - GitHub Repo stars
kanjigrid - - GitHub Repo stars
japanese-toolkit - - GitHub Repo stars
analyze-desumasu-dearu npm npm GitHub Repo stars
hatsuon npm npm GitHub Repo stars
sentiment_ja_js - - GitHub Repo stars
mecab-ipadic-seed npm npm GitHub Repo stars
Japanese-Word-Of-The-Day npm npm GitHub Repo stars
oskim - - GitHub Repo stars
tweetMapping - - GitHub Repo stars

Go

Morphology analysis

  • kagome - Self-contained Japanese Morphological Analyzer written in pure Go
Name downloads/week total downloads stars
kagome - - GitHub Repo stars

Others

  • ojosama - テキストを壱百満天原サロメお嬢様風の口調に変換します
  • nihongo - Japanese Dictionary
  • yomichan-import - External dictionary importer for Yomichan.
  • imas-ime-dic - THE IDOLM@STER words dictionary for Japanese IME (by imas-db.jp)
  • go-moji - A Go library for Zenkaku/Hankaku conversion
Name downloads/week total downloads stars
ojosama - - GitHub Repo stars
nihongo - - GitHub Repo stars
yomichan-import - - GitHub Repo stars
imas-ime-dic - - GitHub Repo stars
go-moji - - GitHub Repo stars

Java

Morphology analysis

  • kuromoji - Kuromoji is a self-contained and very easy to use Japanese morphological analyzer designed for search
  • Sudachi - A Japanese Tokenizer for Business
  • SudachiDict - A lexicon for Sudachi
Name downloads/week total downloads stars
kuromoji - - GitHub Repo stars
Sudachi - - GitHub Repo stars
SudachiDict - - GitHub Repo stars

Others

  • kanjitomo-ocr - Java library for identifying Japanese characters from images
  • jakaroma - Java library and command-line tool to transliterate Japanese kanji to romaji (Latin alphabet)
  • kakasi-java - Kanji transliteration to hiragana/katakana/romaji, in Java
  • Kamite - A desktop language immersion companion for learners of Japanese
  • react-native-japanese-tokenizer - Async Japanese Tokenizer Native Plugin for React Native for iOS and Android
  • elasticsearch-analysis-japanese - Japanese analyzer uses kuromoji japanese tokenizer for ElasticSearch
  • moji4j - A Java library to converts between Japanese Hiragana, Katakana, and Romaji scripts.
  • neologdn-java - Japanese text normalizer for mecab-neologd
Name downloads/week total downloads stars
kanjitomo-ocr - - GitHub Repo stars
jakaroma - - GitHub Repo stars
kakasi-java - - GitHub Repo stars
Kamite - - GitHub Repo stars
react-native-japanese-tokenizer - - GitHub Repo stars
elasticsearch-analysis-japanese - - GitHub Repo stars
moji4j - - GitHub Repo stars
neologdn-java - - GitHub Repo stars

Pretrained model

Word2Vec

  • japanese-words-to-vectors - Word2vec (word to vectors) approach for Japanese language using Gensim and Mecab.
  • chiVe - Japanese word embedding with Sudachi and NWJC
  • elmo-japanese - elmo-japanese
  • embedrank - Python Implementation of EmbedRank
  • aovec - Easy aozorabunko Word2Vec Builder - 青空文庫全書籍のWord2Vecビルダー+構築済みモデル
  • dependency-based-japanese-word-embeddings - This is a repository for the AI LAB article "係り受けに基づく日本語単語埋込 (Dependency-based Japanese Word Embeddings)" ( Article URL https://ai-lab.lapras.com/nlp/japanese-word-embedding/)
  • jawikivec - Yet Another Japanese-Wikipedia Entity Vectors
  • jawiki_word_vector_updater - 最新の日本語Wikipediaのダンプデータから,MeCabを用いてIPA辞書と最新のNeologd辞書の両方で形態素解析を実施し,その結果に基づいた word2vec,fastText,GloVeの単語分散表現を学習するためのスクリプト
Name downloads/week total downloads stars
japanese-words-to-vectors - - GitHub Repo stars
chiVe - - GitHub Repo stars
elmo-japanese - - GitHub Repo stars
embedrank - - GitHub Repo stars
aovec Downloads Downloads GitHub Repo stars
dependency-based-japanese-word-embeddings - - GitHub Repo stars
jawikivec - - GitHub Repo stars
jawiki_word_vector_updater - - GitHub Repo stars

Transformer based models

  • bert-japanese - BERT models for Japanese text.
  • japanese-pretrained-models - Code for producing Japanese pretrained models provided by rinna Co., Ltd.
  • bert-japanese - BERT with SentencePiece for Japanese text.
  • SudachiTra - Japanese tokenizer for Transformers
  • japanese-dialog-transformers - Code for evaluating Japanese pretrained models provided by NTT Ltd.
  • shiba - Pytorch implementation and pre-trained Japanese model for CANINE, the efficient character-level transformer.
  • Dialog - A PyTorch Implementation of japanese chatbot using BERT and Transformer's decoder
  • language-pretraining - BERT and ELECTRA models of PyTorch implementations for Japanese text.
  • medbertjp - Trials of pre-trained BERT models for the medical domain in Japanese.
  • ILYS-aoba-chatbot - ILYS-aoba-chatbot
  • t5-japanese - Codes to pre-train Japanese T5 models
  • pytorch_bert_japanese - PytorchでBERTの日本語学習済みモデルを利用する
  • Laboro-BERT-Japanese - Laboro BERT Japanese: Japanese BERT Pre-Trained With Web-Corpus
  • RoBERTa-japanese - Japanese BERT Pretrained Model
  • aMLP-japanese - aMLP Transformer Model for Japanese
  • bert-japanese-aozora - Japanese BERT trained on Aozora Bunko and Wikipedia, pre-tokenized by MeCab with UniDic & SudachiPy
  • sbert-ja - Code to train Sentence BERT Japanese model for Hugging Face Model Hub
  • BERT-Japan-vaccination - Official fine-tuning code for "Emotion Analysis of Japanese Tweets and Comparison to Vaccinations in Japan"
  • gpt2-japanese - Japanese GPT2 Generation Model
  • text2text-japanese - gpt-2 based text2text conversion model
  • gpt-ja - GPT-2 Japanese model for HuggingFace's transformers
  • friendly_JA-Model - MT model trained using the friendly_JA Corpus attempting to make Japanese easier/more accessible to occidental people by using the Latin/English derived katakana lexicon instead of the standard Sino-Japanese lexicon
  • albert-japanese - BERT with SentencePiece for Japanese text.
  • ja_text_bert - 日本語WikipediaコーパスでBERTのPre-Trainedモデルを生成するためのリポジトリ
  • DistilBERT-base-jp - A Japanese DistilBERT pretrained model, which was trained on Wikipedia.
  • bert - This repository provides snippets to use RoBERTa pre-trained on Japanese corpus. Our dataset consists of Japanese Wikipedia and web-scrolled articles, 25GB in total. The released model is built based on that from HuggingFace.
  • Laboro-DistilBERT-Japanese - Laboro DistilBERT Japanese
  • luke - LUKE -- Language Understanding with Knowledge-based Embeddings
  • GPTSAN - General-purpose Swich transformer based Japanese language mode
  • japanese-clip - Japanese CLIP by rinna Co., Ltd.
  • AcademicBART - We pretrained a BART-based Japanese masked language model on paper abstracts from the academic database CiNii Articles
  • AcademicRoBERTa - We pretrained a RoBERTa-based Japanese masked language model on paper abstracts from the academic database CiNii Articles.
  • LINE-DistilBERT-Japanese - DistilBERT model pre-trained on 131 GB of Japanese web text. The teacher model is BERT-base that built in-house at LINE.
  • Japanese-Alpaca-LoRA - 日本語に翻訳したStanford Alpacaのデータセットを用いてLLaMAをファインチューニングし作成したLow-Rank AdapterのリンクとGenerateサンプルコード
Name downloads/week total downloads stars
bert-japanese - - GitHub Repo stars
japanese-pretrained-models - - GitHub Repo stars
bert-japanese - - GitHub Repo stars
SudachiTra Downloads Downloads GitHub Repo stars
japanese-dialog-transformers - - GitHub Repo stars
shiba Downloads Downloads GitHub Repo stars
Dialog - - GitHub Repo stars
language-pretraining - - GitHub Repo stars
medbertjp - - GitHub Repo stars
ILYS-aoba-chatbot - - GitHub Repo stars
t5-japanese - - GitHub Repo stars
pytorch_bert_japanese - - GitHub Repo stars
Laboro-BERT-Japanese - - GitHub Repo stars
RoBERTa-japanese - - GitHub Repo stars
aMLP-japanese - - GitHub Repo stars
bert-japanese-aozora - - GitHub Repo stars
sbert-ja - - GitHub Repo stars
BERT-Japan-vaccination - - GitHub Repo stars
gpt2-japanese - - GitHub Repo stars
text2text-japanese - - GitHub Repo stars
gpt-ja - - GitHub Repo stars
friendly_JA-Model - - GitHub Repo stars
albert-japanese - - GitHub Repo stars
ja_text_bert - - GitHub Repo stars
DistilBERT-base-jp - - GitHub Repo stars
bert - - GitHub Repo stars
Laboro-DistilBERT-Japanese - - GitHub Repo stars
luke - - GitHub Repo stars
GPTSAN - - GitHub Repo stars
japanese-clip - - GitHub Repo stars
AcademicBART - - GitHub Repo stars
AcademicRoBERTa - - GitHub Repo stars
LINE-DistilBERT-Japanese - - GitHub Repo stars
Japanese-Alpaca-LoRA - - GitHub Repo stars

ChatGPT

  • VRChatGPT - ChatGPTを使ってVRChat上でお喋り出来るようにするプログラム。
  • AITuberDegikkoMirii - AITuberの基礎となる部分を開発しています
  • wanna - Shell command launcher with natural language
  • ChatdollKit - ChatdollKit enables you to make your 3D model into a chatbot
  • ChuanhuChatGPTJapanese - GUI for ChatGPT API For Japanese
  • AISisterAIChan - ChatGPT3.5を搭載した伺かゴースト「AI妹アイちゃん」です。利用には別途ChatGPTのAPIキーが必要です。
  • vrchatbot - VRChatにAI Botを作るためのリポジトリ
  • gptuber-by-langchain - GPTがYouTuberをやります
  • openai-chatfriend - A chatbox application built using Nuxt 3 powered by Open AI Text completion endpoint. You can select different personality of your AI friend. The default will respond in Japanese. You can use this app to practice your Nihongo skills!
  • chrome-ext-translate-to-hiragana-with-chatgpt - This Chrome extension can translate selected Japanese text to Hiragana by using ChatGPT.
  • azure-search-openai-demo - このサンプルでは、Retrieval Augmented Generation パターンを使用して、独自のデータに対してChatGPT のような体験を作成するためのいくつかのアプローチを示しています。
  • chatvrm - ChatVRMはブラウザで簡単に3Dキャラクターと会話ができるデモアプリケーションです。
Name downloads/week total downloads stars
VRChatGPT - - GitHub Repo stars
AITuberDegikkoMirii - - GitHub Repo stars
wanna Downloads Downloads GitHub Repo stars
ChatdollKit - - GitHub Repo stars
ChuanhuChatGPTJapanese - - GitHub Repo stars
AISisterAIChan - - GitHub Repo stars
vrchatbot - - GitHub Repo stars
gptuber-by-langchain - - GitHub Repo stars
openai-chatfriend - - GitHub Repo stars
chrome-ext-translate-to-hiragana-with-chatgpt - - GitHub Repo stars
azure-search-openai-demo - - GitHub Repo stars
chatvrm - - GitHub Repo stars

Dictionary

  • mecab-ipadic-neologd - Neologism dictionary based on the language resources on the Web for mecab-ipadic
  • tdmelodic - A Japanese accent dictionary generator
  • jamdict - Python 3 library for manipulating Jim Breen's JMdict, KanjiDic2, JMnedict and kanji-radical mappings
  • unidic-py - Unidic packaged for installation via pip.
  • Japanese-Company-Lexicon - Japanese Company Lexicon (JCLdic)
  • manbyo-sudachi - Sudachi向け万病辞書
  • jawiki-kana-kanji-dict - Generate SKK/MeCab dictionary from Wikipedia(Japanese edition)
  • JIWC-Dictionary - dictionary to find emotion related to text
  • JumanDIC - This repository contains source dictionary files to build dictionaries for JUMAN and Juman++.
  • ipadic-py - IPAdic packaged for easy use from Python.
  • unidic-lite - A small version of UniDic for easy pip installs.
  • emoji-ime-dictionary - 日本語で絵文字入力をするための IME 追加辞書 orange_book Google 日本語入力などで日本語から絵文字への変換を可能にする IME 拡張辞書
  • google-ime-dictionary - 日英変換・英語略語展開のための IME 追加辞書 orange_book 日本語から英語への和英変換や英語略語の展開を Google 日本語入力や ATOK などで可能にする IME 拡張辞書
  • dic-nico-intersection-pixiv - ニコニコ大百科とピクシブ百科事典の共通部分のIME辞書
  • google-ime-user-dictionary-ja-en - GoogleIME用カタカナ語辞書プロジェクトのアーカイブです。Project archive of Google IME user dictionary from Katakana word ( Japanese loanword ) to English.
  • emoticon - Google日本語入力の顔文字辞書∩(,,Ò‿Ó,,)∩
  • mecab-mozcdic - open source mozc dictionaryをMeCab辞書のフォーマットに変換したものです。
  • denonbu-ime-dic - 電音IME: Microsoft IMEなどで利用することを想定した「電音部」関連用語の辞書
  • nijisanji-ime-dic - Microsoft IMEなどで利用することを想定した「にじさんじ」関連用語の用語辞書です。
  • pokemon-ime-dic - Microsoft IMEなどで利用することを想定した、現状判明している全てのポケモンの名前を網羅した用語辞書です。
  • EJDict - English-Japanese Dictionary data (Public Domain) EJDict-hand
  • Ayashiy-Nipongo-Dic - 贵樣ばこゐ辞畫を使て正レい日本语を使ラことが出來ゑ。
  • genshin-dict - Windows/macOSで使える原神の単語辞書です
  • jmdict-simplified - JMdict and JMnedict in JSON format
  • mozcdict-ext - Convert external words into Mozc system dictionary
Name downloads/week total downloads stars
mecab-ipadic-neologd - - GitHub Repo stars
tdmelodic - - GitHub Repo stars
jamdict Downloads Downloads GitHub Repo stars
unidic-py Downloads Downloads GitHub Repo stars
Japanese-Company-Lexicon - - GitHub Repo stars
manbyo-sudachi - - GitHub Repo stars
jawiki-kana-kanji-dict - - GitHub Repo stars
JIWC-Dictionary - - GitHub Repo stars
JumanDIC - - GitHub Repo stars
ipadic-py Downloads Downloads GitHub Repo stars
unidic-lite Downloads Downloads GitHub Repo stars
emoji-ime-dictionary - - GitHub Repo stars
google-ime-dictionary - - GitHub Repo stars
dic-nico-intersection-pixiv - - GitHub Repo stars
google-ime-user-dictionary-ja-en - - GitHub Repo stars
emoticon - - GitHub Repo stars
mecab-mozcdic - - GitHub Repo stars
denonbu-ime-dic - - GitHub Repo stars
nijisanji-ime-dic - - GitHub Repo stars
pokemon-ime-dic - - GitHub Repo stars
EJDict - - GitHub Repo stars
Ayashiy-Nipongo-Dic - - GitHub Repo stars
genshin-dict - - GitHub Repo stars
jmdict-simplified - - GitHub Repo stars
mozcdict-ext - - GitHub Repo stars

Corpus

Part-of-speech tagging / Named entity recognition

Name downloads/week total downloads stars
ner-wikipedia-dataset - - GitHub Repo stars
IOB2Corpus - - GitHub Repo stars
TwitterCorpus - - GitHub Repo stars
UD_Japanese-PUD - - GitHub Repo stars
UD_Japanese-GSD - - GitHub Repo stars
KWDLC - - GitHub Repo stars
AnnotatedFKCCorpus - - GitHub Repo stars

Parallel corpus

Name downloads/week total downloads stars
small_parallel_enja - - GitHub Repo stars
Web-Crawled-Corpus-for-Japanese-Chinese-NMT - - GitHub Repo stars
CourseraParallelCorpusMining - - GitHub Repo stars
JESC - - GitHub Repo stars
AMI-Meeting-Parallel-Corpus - - GitHub Repo stars
giant_ja-en_parallel_corpus - - GitHub Repo stars
jesc_small - - GitHub Repo stars
graded-enja-corpus - - GitHub Repo stars
cjk-compsci-terms - - GitHub Repo stars
Laboro-ParaCorpus - - GitHub Repo stars
google-vs-deepl-je - - GitHub Repo stars

Dialog corpus

  • JMRD - Japanese Movie Recommendation Dialogue dataset
  • open2ch-dialogue-corpus - おーぷん2ちゃんねるをクロールして作成した対話コーパス
  • BSD - The Business Scene Dialogue corpus
  • asdc - Accommodation Search Dialog Corpus (宿泊施設探索対話コーパス)
  • japanese-corpus - 日本語の対話データ for seq2seq etc
  • BPersona-chat - This repository contains the Japanese–English bilingual chat corpus BPersona-chat published in the paper Chat Translation Error Detection for Assisting Cross-lingual Communications at AACL-IJCNLP 2022's Workshop Eval4NLP 2022.
  • japanese-daily-dialogue - Japanese Daily Dialogue, or 日本語日常対話コーパス in Japanese, is a high-quality multi-turn dialogue dataset containing daily conversations on five topics: dailylife, school, travel, health, and entertainment.
Name downloads/week total downloads stars
JMRD - - GitHub Repo stars
open2ch-dialogue-corpus - - GitHub Repo stars
BSD - - GitHub Repo stars
asdc - - GitHub Repo stars
japanese-corpus - - GitHub Repo stars
BPersona-chat - - GitHub Repo stars
japanese-daily-dialogue - - GitHub Repo stars

Others

  • jrte-corpus - Japanese Realistic Textual Entailment Corpus (NLP 2020, LREC 2020)
  • kanji-data - A JSON kanji dataset with updated JLPT levels and WaniKani information
  • JapaneseWordSimilarityDataset - Japanese Word Similarity Dataset
  • simple-jppdb - A paraphrase database for Japanese text simplification
  • chABSA-dataset - chakki's Aspect-Based Sentiment Analysis dataset
  • JaQuAD - JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension (2022, Skelter Labs)
  • JaNLI - Japanese Adversarial Natural Language Inference Dataset
  • ebe-dataset - Evidence-based Explanation Dataset (AACL-IJCNLP 2020)
  • emoji-ja - UNICODE絵文字の日本語読み/キーワード/分類辞書
  • nayose-wikipedia-ja - Wikipediaから作成した日本語名寄せデータセット
  • ja.text8 - Japanese text8 corpus for word embedding.
  • ThreeLineSummaryDataset - 3行要約データセット
  • japanese - This repo contains a list of the 44,998 most common Japanese words in order of frequency, as determined by the University of Leeds Corpus.
  • kanji-frequency - Kanji usage frequency data collected from various sources
  • TEDxJP-10K - TEDxJP-10K ASR Evaluation Dataset
  • CoARiJ - Corpus of Annual Reports in Japan
  • technological-book-corpus-ja - 日本語で書かれた技術書を収集した生コーパス/ツール
  • ita-corpus-chuwa - Chunked word annotation for ITA corpus
  • wikipedia-utils - Utility scripts for preprocessing Wikipedia texts for NLP
  • inappropriate-words-ja - 日本語における不適切表現を収集します。自然言語処理の時のデータクリーニング用等に使えると思います。
  • house-of-councillors - 参議院の公式ウェブサイトから会派、議員、議案、質問主意書のデータを整理しました。
  • house-of-representatives - 国会議案データベース:衆議院
  • STAIR-captions - STAIR captions: large-scale Japanese image caption dataset
  • Winograd-Schema-Challenge-Ja - Japanese Translation of Winograd Schema Challenge
  • speechBSD - An extension of the BSD corpus with audio and speaker attribute information
  • ita-corpus - ITAコーパスの文章リスト
  • rohan4600 - モーラバランス型日本語コーパス
  • anlp-jp-history - 言語処理学会年次大会講演の全リスト・機械可読版など
  • keigo_transfer_task - 敬語変換タスクにおける評価用データセット
  • loanwords_gairaigo - English loanwords in Japanese
  • jawikicorpus - Japanese-Wikipedia Wikification Corpus
  • GeneralPolicySpeechOfPrimeMinisterOfJapan - This is the corpus of Japanese Text that general policy speech of prime minister of Japan
  • wrime - WRIME: 主観と客観の感情分析データセット
  • jtubespeech - JTubeSpeech: Corpus of Japanese speech collected from YouTube
  • WikipediaWordFrequencyList - 日本語Wikipediaで使用される頻出単語のリスト
  • kokkosho_data - 車両不具合情報に関するデータセット
  • pdmocrdataset-part1 - デジタル化資料OCRテキスト化事業において作成されたOCR学習用データセット
  • huriganacorpus-ndlbib - 全国書誌データから作成した振り仮名のデータセット
  • jvs_hiho - JVS (Japanese versatile speech) コーパスの自作のラベル
  • hirakanadic - Allows Sudachi to normalize from hiragana to katakana from any compound word list
  • animedb - 約100年に渡るアニメ作品リストデータベース
  • security_words - サイバーセキュリティに関連する公的な組織の日英対応
  • Data-on-Japanese-Diet-Members - 日本の国会議員のデータ
  • honkoku-data - 歴史資料の市民参加型翻刻プラットフォーム「みんなで翻刻」のテキストデータ置き場です。 / Transcription texts created on Minna de Honkoku (https://honkoku.org), a crowdsourced transcription platform for historical Japanese documents.
  • wikihow_japanese - wikiHow dataset (Japanese version)
  • engineer-vocabulary-list - Engineer Vocabulary List in Japanese/English
  • JSICK - Japanese Sentences Involving Compositional Knowledge (JSICK) Dataset/JSICK-stress Test Set
  • phishurl-list - Phishing URL dataset from JPCERT/CC
  • jcms - A Japanese Corpus of Many Specialized Domains (JCMS)
  • aozorabunko_text - text-only archives of www.aozora.gr.jp
  • friendly_JA-Corpus - friendly_JA is a parallel Japanese-to-Japanese corpus aimed at making Japanese easier by using the Latin/English derived katakana lexicon instead of the standard Sino-Japanese lexicon
  • topokanji - Topologically ordered lists of kanji for effective learning
  • isbn4groups - ISBN-13における日本語での出版物 (978-4-XXXXXXXXX) に関するデータ等
  • NMeCab - NMeCab: About Japanese morphological analyzer on .NET
  • ndlngramdata - デジタル化資料から作成したOCRテキストデータのngram頻度統計情報のデータセット
  • ndlngramviewer_v2 - 2023年1月にリニューアルしたNDL Ngram Viewerのソースコード等一式
  • data_set - 法律・判例関係のデータセット
  • huggingface-datasets_wrime - WRIME for huggingface datasets
  • ndl-minhon-ocrdataset - NDL古典籍OCR学習用データセット(みんなで翻刻加工データ)
  • PAX_SAPIENTICA - GIS & Archaeological Simulator. 2023 in development.
  • j-liwc2015 - Japanese version of LIWC2015
  • huggingface-datasets_livedoor-news-corpus - Japanese Livedoor news corpus for huggingface datasets
  • huggingface-datasets_JGLUE - JGLUE: Japanese General Language Understanding Evaluation for huggingface datasets
  • commonsense-moral-ja - JCommonsenseMorality is a dataset created through crowdsourcing that reflects the commonsense morality of Japanese annotators.
  • comet-atomic-ja - COMET-ATOMIC ja
  • dcsg-ja - Dialogue Commonsense Graph in Japanese
  • japanese-toxic-dataset - "Proposal and Evaluation of Japanese Toxicity Schema" provides a schema and dataset for toxicity in the Japanese language.
  • camera - CAMERA (CyberAgent Multimodal Evaluation for Ad Text GeneRAtion) is the Japanese ad text generation dataset.
  • Japanese-Fakenews-Dataset - 日本語フェイクニュースデータセット
  • jpn_explainable_qa_dataset - jpn_explainable_qa_dataset
  • copa-japanese - COPA Dataset in Japanese
  • WLSP-familiarity - Word Familiarity Rate for 'Word List by Semantic Principles (WLSP)'
  • ProSub - A cross-linguistic study of pronoun substitutes and address terms
  • commonsense-moral-ja - JCommonsenseMorality is a dataset created through crowdsourcing that reflects the commonsense morality of Japanese annotators.
  • ramendb - なんとかデータベース( https://supleks.jp/ )からのスクレイピングツールと収集データ
  • huggingface-datasets_CAMERA - CAMERA (CyberAgent Multimodal Evaluation for Ad Text GeneRAtion) for huggingface datasets
  • FactCheckSentenceNLI-FCSNLI- - FactCheckSentenceNLIデータセット
  • databricks-dolly-15k-ja - databricks/dolly-v2-12b の学習データに使用されたdatabricks-dolly-15k.jsonl を日本語に翻訳したデータセットになります。
  • EaST-MELD - EaST-MELD is an English-Japanese dataset for emotion-aware speech translation based on MELD.
  • meconaudio - Mecon Audio(Medical Conference Audio)は厚生労働省主催の先進医療会議の議事録の読み上げデータセットです。
Name downloads/week total downloads stars
jrte-corpus - - GitHub Repo stars
kanji-data - - GitHub Repo stars
JapaneseWordSimilarityDataset - - GitHub Repo stars
simple-jppdb - - GitHub Repo stars
chABSA-dataset - - GitHub Repo stars
JaQuAD - - GitHub Repo stars
JaNLI - - GitHub Repo stars
ebe-dataset - - GitHub Repo stars
emoji-ja - - GitHub Repo stars
nayose-wikipedia-ja - - GitHub Repo stars
ja.text8 - - GitHub Repo stars
ThreeLineSummaryDataset - - GitHub Repo stars
japanese - - GitHub Repo stars
kanji-frequency - - GitHub Repo stars
TEDxJP-10K - - GitHub Repo stars
CoARiJ - - GitHub Repo stars
technological-book-corpus-ja - - GitHub Repo stars
ita-corpus-chuwa - - GitHub Repo stars
wikipedia-utils - - GitHub Repo stars
inappropriate-words-ja - - GitHub Repo stars
house-of-councillors - - GitHub Repo stars
house-of-representatives - - GitHub Repo stars
STAIR-captions - - GitHub Repo stars
Winograd-Schema-Challenge-Ja - - GitHub Repo stars
speechBSD - - GitHub Repo stars
ita-corpus - - GitHub Repo stars
rohan4600 - - GitHub Repo stars
anlp-jp-history - - GitHub Repo stars
keigo_transfer_task - - GitHub Repo stars
loanwords_gairaigo - - GitHub Repo stars
jawikicorpus - - GitHub Repo stars
GeneralPolicySpeechOfPrimeMinisterOfJapan - - GitHub Repo stars
wrime - - GitHub Repo stars
jtubespeech - - GitHub Repo stars
WikipediaWordFrequencyList - - GitHub Repo stars
kokkosho_data - - GitHub Repo stars
pdmocrdataset-part1 - - GitHub Repo stars
huriganacorpus-ndlbib - - GitHub Repo stars
jvs_hiho - - GitHub Repo stars
hirakanadic Downloads Downloads GitHub Repo stars
animedb - - GitHub Repo stars
security_words - - GitHub Repo stars
Data-on-Japanese-Diet-Members - - GitHub Repo stars
honkoku-data - - GitHub Repo stars
wikihow_japanese - - GitHub Repo stars
engineer-vocabulary-list - - GitHub Repo stars
JSICK - - GitHub Repo stars
phishurl-list - - GitHub Repo stars
jcms - - GitHub Repo stars
aozorabunko_text - - GitHub Repo stars
friendly_JA-Corpus - - GitHub Repo stars
topokanji - - GitHub Repo stars
isbn4groups - - GitHub Repo stars
NMeCab - - GitHub Repo stars
ndlngramdata - - GitHub Repo stars
ndlngramviewer_v2 - - GitHub Repo stars
data_set - - GitHub Repo stars
huggingface-datasets_wrime - - GitHub Repo stars
ndl-minhon-ocrdataset - - GitHub Repo stars
PAX_SAPIENTICA - - GitHub Repo stars
j-liwc2015 - - GitHub Repo stars
huggingface-datasets_livedoor-news-corpus - - GitHub Repo stars
huggingface-datasets_JGLUE - - GitHub Repo stars
commonsense-moral-ja - - GitHub Repo stars
comet-atomic-ja - - GitHub Repo stars
dcsg-ja - - GitHub Repo stars
japanese-toxic-dataset - - GitHub Repo stars
camera - - GitHub Repo stars
Japanese-Fakenews-Dataset - - GitHub Repo stars
jpn_explainable_qa_dataset - - GitHub Repo stars
copa-japanese - - GitHub Repo stars
WLSP-familiarity - - GitHub Repo stars
ProSub - - GitHub Repo stars
commonsense-moral-ja - - GitHub Repo stars
ramendb - - GitHub Repo stars
huggingface-datasets_CAMERA - - GitHub Repo stars
FactCheckSentenceNLI-FCSNLI- - - GitHub Repo stars
databricks-dolly-15k-ja - - GitHub Repo stars
EaST-MELD - - GitHub Repo stars
meconaudio - - GitHub Repo stars

Tutorial

Name downloads/week total downloads stars
spacy_tutorial - - GitHub Repo stars
fastTextJapaneseTutorial - - GitHub Repo stars
allennlp-NER-ja - - GitHub Repo stars
chariot-PyTorch-Japanese-text-classification - - GitHub Repo stars
ginza-examples - - GitHub Repo stars
DocumentClassificationUsingBERT-Japanese - - GitHub Repo stars
BERT_Japanese_Google_Colaboratory - - GitHub Repo stars
bert-book - - GitHub Repo stars
janome-tutorial - - GitHub Repo stars
handson-language-models - - GitHub Repo stars
JapaneseNLI - - GitHub Repo stars
deep-learning-with-pytorch-ja - - GitHub Repo stars
bert-classification-tutorial - - GitHub Repo stars

Research summary

  • awesome-bert-japanese - A list of pre-trained BERT models for Japanese with word/subword tokenization + vocabulary construction algorithm information
  • GEC-Info-ja - 文法誤り訂正に関する日本語文献を収集・分類するためのリポジトリ
  • dataset-list - lists of text corpus and more (mainly Japanese)
  • tuning_playbook_ja - ディープラーニングモデルの性能を体系的に最大化するためのプレイブック
  • japanese-pitch-accent-resources - Trying to consolidate japanese phonetic, and in particular pitch accent resources into one list
Name downloads/week total downloads stars
awesome-bert-japanese - - GitHub Repo stars
GEC-Info-ja - - GitHub Repo stars
dataset-list - - GitHub Repo stars
tuning_playbook_ja - - GitHub Repo stars
japanese-pitch-accent-resources - - GitHub Repo stars

Reference

Contributors

About

A curated list of resources dedicated to Python libraries, pre-trained models, dictionaries, and corpora of NLP for Japanese

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published