Skip to main content

😄 HappyNodeTokenizer A basic Twitter aware tokenizer for Javascript environments. A Typescript port of HappyFunTokenizer.py by Christopher Potts and HappierFunTokenizing.py by H. Andrew Schwartz.

This package works with Bun, Node.js, Deno, BrowsersIt is unknown whether this package works with Cloudflare Workers
JSR Score
100%
Published
4 months ago (7.1.0)
type alias TokenizerNormalizationForm

String normalization form

Definition

| "NFC"
| "NFD"
| "NFKC"
| "NFKD"
| null

Add Package

deno add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerNormalizationForm } from "@phughesmcr/happynodetokenizer";

Add Package

npx jsr add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerNormalizationForm } from "@phughesmcr/happynodetokenizer";

Add Package

yarn dlx jsr add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerNormalizationForm } from "@phughesmcr/happynodetokenizer";

Add Package

pnpm dlx jsr add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerNormalizationForm } from "@phughesmcr/happynodetokenizer";

Add Package

bunx jsr add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerNormalizationForm } from "@phughesmcr/happynodetokenizer";