Shannon's Hatchet

something, but almost nothing

Friday, February 24, 2023

TypeScript modules With Emscripten and CMake, part 4

When I set out to create an NPM package for SoundSwallower, I was unable to find much relevant information in the Emscripten documentation or elsewhere on the Web, so I have written this guide, which walks through the process of compiling a library to WebAssembly and packaging it as a CommonJS or ES6 module.

This is part of a series of posts. Start here to read from the beginning.

Building with CMake

Just by screwing around on the command line, we were previously able to produce a more or less useful CommonJS module wrapping the real-valued FFT function from the Kiss FFT library (though not as useful as the existing one on npmjs.org). Now let’s look at how we can build a module with CMake as part of the library’s build system.

As a reminder, we configured CMake to build the library with:

emcmake cmake -S . -B jsbuild -DCMAKE_BUILD_TYPE=Debug \
    -DKISSFFT_TOOLS=OFF -DKISSFFT_STATIC=ON -DKISSFFT_TEST=OFF

When configuring using emcmake, the EMSCRIPTEN variable is defined, so if we want to make all of those flags the defaults, we can add this to CMakeLists.txt after the option definitions (line 54 in the current source):

if(EMSCRIPTEN)
  set(KISSFFT_TOOLS OFF)
  set(KISSFFT_STATIC ON)
  set(KISSFFT_TEST OFF)
endif()

Now let’s add a target to build our module. This is a bit “special” for two reasons:

  • The CMake functions for Emscripten treat any output (even a module) as an “executable”, so we have to make believe we’re linking a program.
  • Even though all of the C code is already in the libkissfft-float.a library, which CMake references with the kissfft target, it still expects to have at least one source file to link into our “executable”.

To satisfy CMake, we will first simply create an empty C file:

touch api.c

We may at some point want to add helper functions for our API, so this isn’t entirely useless - see the corresponding file in SoundSwallower for an example.

Now we will add the necessary CMake configuration to the end of CMakeLists.txt:

if(EMSCRIPTEN)
  add_executable(kissfft.cjs api.c)
  target_link_libraries(kissfft.cjs kissfft)
  target_link_options(kissfft.cjs PRIVATE
    -sMODULARIZE=1
    -sEXPORTED_FUNCTIONS=@${CMAKE_CURRENT_SOURCE_DIR}/exported_functions.txt)
  em_link_post_js(kissfft.cjs api.js)
endif()

A few things to note here:

  • em_link_post_js is not documented, but should be.
  • We have to add ${CMAKE_CURRENT_SOURCE_DIR} to the path to exported_functions.txt so that CMake can find it, since we are building in a separate directory.
  • We can’t use kissfft as the target name since that is already taken by the C library.

Emscripten will automatically append .js and .wasm to the target name, so, after adding this, if you run:

emcmake cmake -S . -B jsbuild -DCMAKE_BUILD_TYPE=Debug
cmake --build jsbuild

You should find the files kissfft.cjs.js and kissfft.cjs.wasm in the jsbuild directory.

Building an ES6 module

Up to this point we have built a CommonJS module, since they are simpler to use in Node.js, but in reality, all the cool kids are now using ES6 modules, and they are particularly preferred when using a bundler for the Web like Webpack or Esbuild. The latest versions of Emscripten do have built-in, if occasionally buggy, support for producing ES6 modules. So, we can add an extra target inside the if(EMSCRIPTEN) block at the end of CMakeLists.txt:

add_executable(kissfft.esm api.c)
target_link_libraries(kissfft.esm kissfft)
target_link_options(kissfft.esm PRIVATE
  -sMODULARIZE=1 -sEXPORT_ES6=1
  -sEXPORTED_FUNCTIONS=@${CMAKE_CURRENT_SOURCE_DIR}/exported_functions.txt)
em_link_post_js(kissfft.esm api.js)

Sadly, there is no way in the Emscripten CMake support to choose a different file extension for a specific target, so we can’t call this kissfft.esm.mjs. In addition, the boilerplate loader code that Emscripten gives us won’t allow us to share the WebAssembly (which is identical) between targets. For the moment we will end up with kissfft.esm.js and kissfft.esm.wasm in the jsbuild directory, and this is a Problem, as we will see soon.

Packaging with NPM

Now that everything is built, it is actually quite simple to package this as an NPM package. No other action is required on your part… well, not quite. First, let’s create a package.json file, which will have one big problem, that we’ll get to later:

{
  "name": "kissfft-example",
  "version": "0.0.1",
  "description": "A very simple example of packaging WebAssembly",
  "types": "./index.d.ts",
  "main": "./jsbuild/kissfft.cjs.js",
  "exports": {
    ".": {
      "types": "./index.d.ts",
      "require": "./jsbuild/kissfft.cjs.js",
      "import": "./jsbuild/kissfft.esm.js",
      "default": "./jsbuild/kissfft.esm.js"
    }
  },
  "author": "David Huggins-Daines <dhd@ecolingui.ca>",
  "homepage": "https://ecolingui.ca/en/blog/emguide",
  "license": "MIT",
  "scripts": {
    "test": "npx tsc test_realfft.ts && node test_realfft.js"
  },
  "files": [
    "index.d.ts",
    "jsbuild/kissfft.*.js",
    "jsbuild/kissfft.*.wasm"
  ],
  "devDependencies": {
    "@types/node": "^18.14.1",
    "typescript": "^4.9.5"
  },
  "dependencies": {
    "@types/emscripten": "^1.39.6"
  }
}

Of note above:

  • We use the exports field to supply different entry points for import and require (but note that this won’t actually work… more below).
  • We just package the stuff we built in place, by including only the files we need with the files field.
  • We point to the type definition file with the types field in two places, for good luck.
  • Although the node we get with emsdk includes @types/emscripten by default, others will not, so it is a package (and not dev) dependency.

Now, assuming you have you have previously created test_realfft.ts (if not, download it here), you should be able to run:

npm install
npm test

And you should see the same output we saw previously. But, did we say there was a problem? Yes. The nifty ES6 model built above won’t actually work in Node, because the Node developers somehow can’t agree to not depend on file extensions to select module systems. Since our package contains both CommonJS (loaded with require) and ES6 (loaded with import) modules, we have to change the file extension on at least one of them to satisfy Node’s simplistic view of the world.

The path of least resistance to fix this and still stay CMakically correct is to add a custom command that copies the built .js file for the ES6 module to a .mjs file:

add_custom_command(
  OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/kissfft.esm.mjs
  DEPENDS kissfft.esm
  COMMAND ${CMAKE_COMMAND} -E copy
  ${CMAKE_CURRENT_BINARY_DIR}/kissfft.esm.js
  ${CMAKE_CURRENT_BINARY_DIR}/kissfft.esm.mjs)
add_custom_target(copy-mjs-bork-bork-bork ALL
  DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/kissfft.esm.mjs)

Now we will modify package.json by changing kissfft.esm.js to kissfft.esm.mjs everywhere, and modifying files to specifically only include the files we need:

  "files": [
    "index.d.ts",
    "jsbuild/kissfft.esm.mjs",
    "jsbuild/kissfft.cjs.js",
    "jsbuild/kissfft.*.wasm"
  ],

You can download the updated version here. And now we can test that both import types work by creating a directory called kissfft-test alongside kissfft, creating the files index.mjs (download here) and index.cjs (download here) in it, then running:

npm link ../kissfft
node index.mjs
node index.cjs

Congratulations! You now have a WebAssembly module that will work as both ES6 and CommonJS, and can also be uploaded to NPM (but please don’t do that). To see what would be packaged, you can run:

npm publish --dry-run

In the next installment, we will see what we can do to make the module as small as possible.