ml5.js Website & Next-Gen Documentation

ml5.js (opens in new tab) is an open-source library developed at NYU ITP/IMA that brings approachable machine learning (FaceMesh, HandPose, BodyPose, ImageClassifier, SoundClassifier, NeuralNetwork, DepthEstimation, and Sentiment) to artists, creative coders, and students on top of TensorFlow.js and p5.js.

Beginning in Daniel Shiffman's ml5.js Research Studio (Spring 2024) and continuing as an open-source Web Developer & Maintainer for the ml5.js project (2024 – Present), I helped design, architect, and maintain the ml5.js 1.0 web presence (ml5js.org (opens in new tab)) and unified reference documentation (docs.ml5js.org (opens in new tab)).

Key Contributions#

1. Redesigning the ml5.js 1.0 Documentation & Website Architecture#

As ml5-next-gen (ml5.js 1.0) overhauled its core APIs around modern TensorFlow.js and MediaPipe tasks (detectStart, detectStop, coordinate normalization, and WebGL/WebGPU backends), the legacy documentation needed a complete structural rebuild:

• Architecture Benchmarking: Prototyped and benchmarked multiple documentation engines—including Next.js + CodeMirror, Gatsby/React (ml5-website-v02-gatsby (opens in new tab)), and Docsify (ml5-website-v02-docsify (opens in new tab))—optimizing for zero-build Markdown contribution workflows so students and faculty can submit documentation PRs without complex frontend build steps.
• Unified Learning Flow: Consolidated API reference specifications, model method signatures, step-by-step quickstarts, and curated community projects across ml5js.org (opens in new tab) and docs.ml5js.org (opens in new tab).

2. Interactive In-Page Code Runner & Webcam Sandboxing#

A central goal of my Spring 2024 Research Proposal (opens in new tab) was eliminating the friction between reading a machine learning API reference and seeing live webcam inference in action:

• Designed an embedded live-code preview workflow allowing learners to inspect, edit, and run p5.js + ml5.js sketches directly alongside each model's parameter documentation.
• Resolved cross-origin iframe HTTPS security policies and Permissions-Policy: camera; microphone delegation so live computer-vision models (ml5.faceMesh(), ml5.handPose(), ml5.bodyPose()) can access webcam streams inline without leaving the documentation page.

// ml5.js 1.0 FaceMesh — Real-time 468-keypoint facial landmark tracking in p5.js
let faceMesh;
let video;
let faces = [];
 
function preload() {
  faceMesh = ml5.faceMesh({ maxFaces: 1, refineLandmarks: false, flipped: true });
}
 
function setup() {
  createCanvas(640, 480);
  video = createCapture(VIDEO, { flipped: true });
  video.size(640, 480);
  video.hide();
  faceMesh.detectStart(video, (results) => {
    faces = results;
  });
}
 
function draw() {
  image(video, 0, 0, width, height);
  for (const face of faces) {
    for (const kp of face.keypoints) {
      fill(0, 255, 128);
      noStroke();
      circle(kp.x, kp.y, 4);
    }
  }
}

3. Open-Source Community & Educator Workshops#

Beyond web engineering, I co-led public workshops introducing the ml5.js 1.0 ecosystem to educators, artists, and developers:

• Exploring Friendly Machine Learning for the Web with ml5.js 1.0 — Technical Workshop at the OUR STUDY GROUP symposium at Google NYC (October 2024).
• Machine Learning Tools for Creative Coding — Open-Source Tools Showcase at COSA × NYU (March 2025), demonstrating ml5.js alongside Jumpstart.