Skip to main content

😄 HappyNodeTokenizer A basic Twitter aware tokenizer for Javascript environments. A Typescript port of HappyFunTokenizer.py by Christopher Potts and HappierFunTokenizing.py by H. Andrew Schwartz.

This package works with Bun, Node.js, Deno, BrowsersIt is unknown whether this package works with Cloudflare Workers
JSR Score
100%
Published
4 months ago (7.1.0)
type alias TokenizerMode

The tokenizer methodology

Definition

"stanford" | "dlatk"

Add Package

deno add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerMode } from "@phughesmcr/happynodetokenizer";

Add Package

npx jsr add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerMode } from "@phughesmcr/happynodetokenizer";

Add Package

yarn dlx jsr add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerMode } from "@phughesmcr/happynodetokenizer";

Add Package

pnpm dlx jsr add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerMode } from "@phughesmcr/happynodetokenizer";

Add Package

bunx jsr add @phughesmcr/happynodetokenizer

Import symbol

import { type TokenizerMode } from "@phughesmcr/happynodetokenizer";