typed-tester

a CLI and Vite plugin which runs "type tests" for your Typescript projects

Install

You can install locally in a project or globally via npm:

# install globally
npm install -g typed-tester

or

# install locally to a project
pnpm install -D typed-tester

Usage

This library's primary utility is to provide a CLI to inspect your repo's type usage and most of all to provide an opinionated way of running type tests.

In addition to the CLI, it also exposes some type utilities as an ES Module export which are intended to be used to build your type tests.

CLI

The CLI has the following commands:

1. Type Testing:

   [npx] typed test [filter]
   
   
   

   This is the primary utility for this repo will identify test files in your repo and run "type tests" on them. Runtime results are not a concern of this command, however, you can optionally "opt in" to get additional type errors or warnings outside of your core tests (although this is reserved to just the "test files").

   By default this library will cache your file dependencies (aka, what symbols does your test file use, and then what is the graph of types which make up those symbols) with the goal of being able to watch and report on changes as you code. To run your testing in "watch mode" add the --watch flag to the end of the command.

   The dependency caching will also cache an AST representation of all source files which were needed to complete your test coverage. This has valuable performance improvements but if you're ever feeling uncertain whether the cache is actually interfering with accurate results you can clear it at startup by adding the --clear flag. Refer to the Cache Details section below for more details.

2. Source Diagnostics:

   [npx] typed diagnostics [filter] [--show-warnings] [--only-top=10]
   
   
   

   Evaluates all of the sources files and provides the following metrics by file:

   • errors: type errors not reduced to warnings
   • warnings: type errors specified to show as warnings (such as type defined but not used)
   • performance metrics:
     • AST build time: how long did it take to create an AST out of the file)
     • Perf per Lines of Code: build time by lines of code
     • Perf per Symbol: build time by symbols in file

3. Source Dependency Graph:

   [npx] typed graph [filter] 
   
   
   

   Produces a list of source type symbols and their dependencies on other symbols.

4. Type Coverage:

   [npx] typed coverage
   
   
   

   Provides the following metrics:

   • total number of symbols found in source code
   • symbols tested directly (and coverage %)
   • symbols tested directly or indirectly (and coverage %)

   Note:

   • These metrics aren't perfect but they're better than the absence of metrics
   • The coverage relating to "directly or indirectly" means ...
     • Where the symbol was directly tested (aka, there was a direct test that symbol result in an expected type) or (less ideally) that some other symbol used the type to calculate a tested outcome.

Type Utilities

The following type utilities can be imported from this library:

• Expect / Test / IT
  • all tests will be wrapped with the Expect/Test/IT utility; this utility will wrap one of the assertions listed below

  Expect, Test, and IT are identical but allow the developer to choose the nomenclature they prefer; in this repo we will use Test

• AssertTrue<T>
  • tests whether the tested type T is the type true
• AssertFalse<T>
  • tests whether the tested type T is the type false
• AssertEqual<T,E>
  • tests that the tested type T equals the expected type E
• AssertExtends<T,E>
  • tests that the tested type T extends the expected type E
• AssertSameValues<T,E>
  • tests that the tested type T is an array type and every element of E and T are the same but the order in which they arrive does not matter
• AssertContains<T,E>
  • when the tested type T is a string:
    • this utility will pass when E is also a string and represents a sub-string of the sting literal T
  • when the tested type T is an array then:
    • this utility

How to Write Type Tests

Type tests should be created for a repos "type utilities" as well as for most if not all of it's exported runtime functions.

• obviously for a "type utility" the test file will only have type tests, but

• anytime you're adding type tests for runtime symbols in your repo it is a good practice to intermingle runtime tests and type tests.

• below is an example of intermingling type and runtime tests:

  import { describe, it, expect } from "vitest";
  import { Test, AssertTrue, AssertFalse } from "typed-tester";
  import { isValidSyntax } from "src";
  
  describe("isValidSyntax(val) -> boolean", () => {
      it("positive testing", () => {
          const t1 = isValidTest
      })
  })
  
  
  
  

  Note: the example uses vitest as test runner for runtime tests but you can use any test runner. Plus most test runners have some conception or corollary of describe and it that provide the structure to tests.

Details

Configuring "test files"

We attempt to find these for you by doing the following two things automatically:

1. If you have a vite.config.ts or vitest.config.ts file we will use the glob patterns defined there to identify candidate files.
2. If neither of the two above files are present then we will use the default include pattern of: {test,tests,src}/**/*.{test,spec}.ts

Finally, if the ENV variable TEST_FILES or VITE_TEST_FILES is set then we will use this value over everything else.

Cache Details

The cached data is stored in the root of the repo which you are testing and will appear in the following files:

• .dependencies.json - is a JSON dictionary whose keys are the test files which were tested, they map to hash values which represent the symbol at a point in time. If that point in time is no longer valid (aka, the cache is stale) then this hash key will no longer be available in the .symbols.json lookup

• .symbols.json is a JSON dictionary whose keys are types in your source code and whose value corresponds to something similar to the following (refer to SymbolDefn in the source for an always current definition):

  export type SymbolDefn = {
      /** symbol name */
      symbol: string;
      /** source file */
      filepath: string;
      /** uses xxhash hasher to determine whether hosting file's contents have changed */
      file_hash: string;
      /** uses xxhash hasher to determine whether the specific text representing the symbol has changed */
      symbol_hash: string;
      /** serialized AST representation */
      ast: string;
  }
  
  
  

To Git or Not to Git the Cache

Rather than state an opinion outright, I will say that users of this library should consider whether the "cache files" should go into the git repo or be included as part of the .gitignore. It really comes down to:

1. your confidence that the caching algorithm's ability to detect stale data and re-cache when necessary
2. the benefits of the increased performance of running off a cache (this will be larger and larger as your types increase in quantity and complexity)

Contributing and Reporting Issues

If you do find any issues with this library please consider making a pull request with a tested fix. If you're only going to enter an issue please be 100% sure that you've provided enough information to easily reproduce your issue (or just don't submit it and live with manual cache breaking).

License

MIT License

Copyright (c) 2024-PRESENT Ken Snyderhttps://github.com/yankeeinlondon

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.