Gaussian Splats

Type on the keyboard

Until you click it, the viewer types “the quick brown fox jumps over the lazy dog” on its own without sound. Click it to take control. The keys you press on your own keyboard then press down on the scan, with a click sound for each press and release, and the text you type appears along the bottom. The sound button in the top corner turns the sound off.

While the viewer has control, you can drag to orbit, scroll to zoom and hold Shift while dragging to pan. Press Esc or click anywhere outside the viewer to give the keyboard back to the page. The viewer also lets go when you scroll it out of view.

How the keys move

The keycaps were found with Segment Anything (SAM). It picked out keycap shapes in three of the photos, and those outlines were projected onto the splat using the camera position of each photo. A second pass asked SAM for the key at each key’s centre in eight photos and assigned each blob to a key by vote. This found 80 keys.

Each blob in the scene is tagged with the key it belongs to. When you press a key, a shader moves that key’s blobs along the direction the key travels and darkens them slightly, and a spring brings the key back up when you let go. The movement is 2.6 times the key’s real travel so the presses are easy to see. The click is generated in the browser with the Web Audio API, and each key has a slightly different pitch.

The scan on its own

This is the same scan without the key animation. Drag to orbit, scroll or pinch to zoom, and right-drag to pan. Scrolling over this viewer zooms it rather than scrolling the page.

How it was captured

The keyboard was photographed from all around with a Fujifilm X100S, using 75 photos. The RAW files were converted to JPEGs, and rembg removed the background from each photo so only the keyboard was reconstructed. COLMAP worked out where the camera was for each photo.

The splat was trained with Brush for 30,000 steps. The result has 83,753 Gaussians and is 19.8 MB as a PLY file.

File sizes and viewers

The viewer without animation loads the scan as a 3.8 MB SOG file, converted from the PLY with PlayCanvas’s splat-transform. It is drawn with Spark on top of three.js. The typing viewer loads the full 19.8 MB PLY, plus a key number and a weight for every blob, and is drawn with the PlayCanvas engine. Neither viewer downloads anything until you scroll to it.