PSMnist

PSMnist is a C#-based PowerShell module for downloading, reading, visualizing, and preparing the MNIST handwritten-digit dataset.

Requirements

• PowerShell 7.4 or later
• A SIXEL-capable terminal for Show-MnistImage

SVG output from ConvertTo-MnistSvg does not require SIXEL support and can be displayed in notebook hosts such as Verso.

Install

The beta release is available from PowerShell Gallery:

Install-Module PSMnist -RequiredVersion 0.1.0-beta1 -AllowPrerelease
Import-Module PSMnist

Quick Start

Download and expand the four standard MNIST files:

$dataPath = Join-Path $PWD data/mnist

Save-MnistDataset -DestinationPath $dataPath
Expand-MnistDataset -Path $dataPath

Read one training example and inspect its label:

$sample = Get-MnistDataset -Path $dataPath -Set Train -First 1
$sample.Label.Value

Render the image directly in a SIXEL-capable terminal:

$sample.Image | Show-MnistImage -Scale 8 -Colors 16

Render it as SVG in a Verso notebook:

$svg = $sample.Image | ConvertTo-MnistSvg -Scale 8
Display $svg 'image/svg+xml'

Prepare examples for a model:

$samples =
    Get-MnistDataset -Path $dataPath -Set Train -First 1000 |
    ConvertTo-MnistTrainingSample -Range ZeroToOne

$samples[0].X.Length # 784 normalized pixel values
$samples[0].Y.Length # 10 one-hot target values

Commands

Command	Purpose
Save-MnistDataset	Download the four gzip-compressed MNIST files
Expand-MnistDataset	Expand downloaded files into IDX files
Read-MnistImageFile	Open an IDX image file with enumeration and indexing
Read-MnistLabelFile	Open an IDX label file with enumeration and indexing
Get-MnistDataset	Pair images and labels into dataset examples
Show-MnistImage	Render an image as SIXEL
ConvertTo-MnistSvg	Convert an image to notebook-friendly SVG
ConvertTo-MnistInputVector	Normalize pixels to 0..1 or -1..1
ConvertTo-OneHot	Convert a label to a one-hot target vector
ConvertTo-MnistTrainingSample	Produce paired model input and target vectors

Read IDX Files

$images = Read-MnistImageFile "$dataPath/train-images-idx3-ubyte"
$labels = Read-MnistLabelFile "$dataPath/train-labels-idx1-ubyte"

$image = $images[0]
$label = $labels[0]

$image.Pixels       # flat byte[784] vector
$image[10, 12]      # pixel at x=10, y=12
$label.Value

Vector Conversion

$zeroToOne = $image | ConvertTo-MnistInputVector -Range ZeroToOne
$minusOneToOne = $image | ConvertTo-MnistInputVector -Range MinusOneToOne

$target = $label | ConvertTo-OneHot
$signedTarget = $label |
    ConvertTo-OneHot -Classes 10 -OnValue 1.0 -OffValue -1.0

Build And Test

The source tree expects the libsixel and PSMnist repositories to be sibling directories.

dotnet build ./PSMnist.csproj
dotnet test ./tests/PSMnist.Tests/PSMnist.Tests.csproj

Run the examples after building:

./examples/quickstart.ps1
./examples/micrograd-prep.ps1 -First 100

License

MIT