From ca4d3ccc080c230782c70b82f532a81b973529ea Mon Sep 17 00:00:00 2001 From: axif Date: Mon, 18 May 2026 21:55:55 +0600 Subject: [PATCH 1/4] docs: add Toolforge deployment guide for Scribe-Server --- Toolforge.md | 159 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 Toolforge.md diff --git a/Toolforge.md b/Toolforge.md new file mode 100644 index 0000000..ad5c8d0 --- /dev/null +++ b/Toolforge.md @@ -0,0 +1,159 @@ +# [Toolforge Deployment](https://toolsadmin.wikimedia.org/) + +[Toolforge](https://wikitech.wikimedia.org/wiki/Help:Toolforge) is Wikimedia's hosting platform for community tools. This guide walks you through deploying Scribe-Server there from scratch — no prior Toolforge experience needed. To deploy Scribe Server here, you first apply at [toolsadmin.wikimedia.org](https://toolsadmin.wikimedia.org/) with the relevant project details. After your application is approved, you gain SSH access and can set up the environment, database, and web service described below. + +## Practical Workflow + +### 1) SSH into Toolforge + +Connect to the login node using your Wikimedia developer account: + +```bash +ssh {user_id}@login.toolforge.org +``` + +Once inside, switch to the Scribe tool account so all subsequent commands run in the correct project context: + +```bash +become testserver-scribe +``` + +Then clone the repository into your project directory: + +```bash +git clone https://github.com/scribe-org/Scribe-Server.git +cd Scribe-Server +``` + +### 2) Set Up Go + +Toolforge's [pre-built web images](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Other_/_generic_web_servers) do not include Go, so you install it manually into your home directory. + +> **Note:** `go-sqlite3` requires CGo and `make`, neither of which is available on Toolforge. Use a pure-Go SQLite driver instead. + +```bash +# Download Go +wget https://go.dev/dl/go1.23.6.linux-amd64.tar.gz + +# Extract the tarball +tar -xzf go1.23.6.linux-amd64.tar.gz -C ~/ + +# Rename the directory for organization +mv ~/go ~/go1.23 + +# Persist environment variables across sessions +echo 'export GOROOT=$HOME/go1.23' >> ~/.bashrc +echo 'export PATH=$PATH:$GOROOT/bin' >> ~/.bashrc + +# Apply changes to the current shell +source ~/.bashrc + +# Verify the installation +go version + +# Clean up the downloaded archive +rm go1.23.6.linux-amd64.tar.gz +``` + +Why this layout: + +- Go is extracted to `~/go1.23` rather than `~/go` to make the version explicit and avoid conflicts if you later upgrade. +- `GOROOT` must point at your custom location because the system has no Go in `PATH`. +- Running `go run .` inside Toolforge binds to `0.0.0.0:8000` — this is expected behavior within the Toolforge network. + +### 3) Configure the Database + +Toolforge provides a shared MariaDB cluster. Your credentials are pre-written to `~/replica.my.cnf` during tool creation. + +First, read your credentials: + +```bash +cat ~/replica.my.cnf +``` + +Then copy the example config and fill in the values: + +```bash +mv config-example.yaml config.yaml +nano config.yaml +``` + +```yaml +# Server configuration +hostPort: 8000 +fileSystem: "./packs" + +# Database configuration +database: + user: {user} + password: {password} + host: tools-db.tools.eqiad1.wikimedia.cloud + port: "3306" + name: {user}__scribe_server_p +``` + +Replace `{user}` and `{password}` with the values from `replica.my.cnf`. The database name follows the Toolforge convention: `{user}__`. + +To inspect the database directly at any time: + +```bash +mysql --defaults-file=~/replica.my.cnf \ + -h tools-db.tools.eqiad1.wikimedia.cloud \ + {user}__scribe_server_p +``` + +Example: + +```bash +mysql --defaults-file=~/replica.my.cnf \ + -h tools-db.tools.eqiad1.wikimedia.cloud \ + s123456__scribe_server_p +``` + +### 4) Build and Run the Server + +Each time you deploy an update, stop the running service, pull the latest code, rebuild the binary, and restart: + +```bash +chmod +x update_data.sh +toolforge webservice stop +git pull origin main +go build -o Scribe-Server . +toolforge webservice --mem 4Gi --cpu 2 jdk17 start \ + /data/project/scribe-server/Scribe-Server/update_data.sh +``` + +Why this sequence: + +- `toolforge webservice stop` ensures the old binary is not locked when you overwrite it. +- `go build -o Scribe-Server .` produces a statically-linked binary that Toolforge can execute directly. +- The `--mem 4Gi --cpu 2` flags allocate enough headroom for data loading on startup. + +### 5) Install pip (Python Tooling) + +If you need Python-based tooling in the project, open a Python 3.13 shell and bootstrap pip: + +```bash +toolforge webservice python3.13 shell + +source venv/bin/activate + +curl -sS https://bootstrap.pypa.io/get-pip.py | python +``` + +### 6) Install PyICU (if ICU Detection Fails) + +The standard PyICU build uses `pkg-config` or `icu-config` to locate ICU headers and libraries. Neither tool is installed on Toolforge, so you must set the paths manually before running `pip install`: + +```bash +export ICU_VERSION=76 +export PYICU_LFLAGS="-L/usr/lib/x86_64-linux-gnu -licui18n -licuuc -licudata" +export PYICU_CFLAGS="-I/usr/include" +pip install PyICU +``` + +Why these variables: + +- `ICU_VERSION` tells the build script which header subdirectory to target. +- `PYICU_LFLAGS` points the linker at the system ICU shared objects already present on Toolforge nodes. +- `PYICU_CFLAGS` points the compiler at the system ICU headers. From faa2f3413ff740bf517ad9124d66ca6934b2e2f9 Mon Sep 17 00:00:00 2001 From: Andrew Tavis McAllister Date: Tue, 19 May 2026 00:23:06 +0200 Subject: [PATCH 2/4] Add toc to documentation and minor updates --- Toolforge.md | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/Toolforge.md b/Toolforge.md index ad5c8d0..381c1b5 100644 --- a/Toolforge.md +++ b/Toolforge.md @@ -1,10 +1,20 @@ # [Toolforge Deployment](https://toolsadmin.wikimedia.org/) -[Toolforge](https://wikitech.wikimedia.org/wiki/Help:Toolforge) is Wikimedia's hosting platform for community tools. This guide walks you through deploying Scribe-Server there from scratch — no prior Toolforge experience needed. To deploy Scribe Server here, you first apply at [toolsadmin.wikimedia.org](https://toolsadmin.wikimedia.org/) with the relevant project details. After your application is approved, you gain SSH access and can set up the environment, database, and web service described below. +[Toolforge](https://wikitech.wikimedia.org/wiki/Help:Toolforge) is Wikimedia's hosting platform for community tools. This guide walks you through deploying Scribe-Server on Toolforge from scratch — no prior Toolforge experience needed. To deploy Scribe Server here, you first apply at [toolsadmin.wikimedia.org](https://toolsadmin.wikimedia.org/) with the relevant project details. After your application is approved, you gain SSH access and can set up the environment, database, and web service described below. + +## Contents + +- [Practical Workflow](#first-steps-as-a-contributor) + - [SSH into Toolforge](#ssh-into-toolforge) + - [Set Up Go](#set-up-go) + - [Configure the Database](#configure-the-database) + - [Build and Run the Server](#build-and-run-the-server) + - [Install pip](#install-pip) + - [Install PyICU](#install-pyicu) ## Practical Workflow -### 1) SSH into Toolforge +### SSH into Toolforge Connect to the login node using your Wikimedia developer account: @@ -25,7 +35,7 @@ git clone https://github.com/scribe-org/Scribe-Server.git cd Scribe-Server ``` -### 2) Set Up Go +### Set Up Go Toolforge's [pre-built web images](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Other_/_generic_web_servers) do not include Go, so you install it manually into your home directory. @@ -61,7 +71,7 @@ Why this layout: - `GOROOT` must point at your custom location because the system has no Go in `PATH`. - Running `go run .` inside Toolforge binds to `0.0.0.0:8000` — this is expected behavior within the Toolforge network. -### 3) Configure the Database +### Configure the Database Toolforge provides a shared MariaDB cluster. Your credentials are pre-written to `~/replica.my.cnf` during tool creation. @@ -110,7 +120,7 @@ mysql --defaults-file=~/replica.my.cnf \ s123456__scribe_server_p ``` -### 4) Build and Run the Server +### Build and Run the Server Each time you deploy an update, stop the running service, pull the latest code, rebuild the binary, and restart: @@ -129,7 +139,7 @@ Why this sequence: - `go build -o Scribe-Server .` produces a statically-linked binary that Toolforge can execute directly. - The `--mem 4Gi --cpu 2` flags allocate enough headroom for data loading on startup. -### 5) Install pip (Python Tooling) +### Install pip If you need Python-based tooling in the project, open a Python 3.13 shell and bootstrap pip: @@ -141,7 +151,10 @@ source venv/bin/activate curl -sS https://bootstrap.pypa.io/get-pip.py | python ``` -### 6) Install PyICU (if ICU Detection Fails) +### Install PyICU + +> [!NOTE] +> The following should be done if ICU Detection Fails. The standard PyICU build uses `pkg-config` or `icu-config` to locate ICU headers and libraries. Neither tool is installed on Toolforge, so you must set the paths manually before running `pip install`: From 2f3b24c881de755ce72af95c344e1fd81bd45f1a Mon Sep 17 00:00:00 2001 From: DeleMike Date: Tue, 19 May 2026 11:12:49 +0100 Subject: [PATCH 3/4] add missing `start-script.sh` and python environment setup --- Toolforge.md | 47 ++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/Toolforge.md b/Toolforge.md index 381c1b5..cfd3005 100644 --- a/Toolforge.md +++ b/Toolforge.md @@ -120,17 +120,37 @@ mysql --defaults-file=~/replica.my.cnf \ s123456__scribe_server_p ``` +### Create the Start Script (Only Once) + +```bash +~/Scribe-Server/start-script.sh << 'EOF' +#!/bin/bash +cd /data/project/testserver-scribe/Scribe-Server +export PORT=8000 +./Scribe-Server +EOF +``` + +**Make it executable:** + +```.bash +chmod +x ~/Scribe-Server/start-script.sh +``` + ### Build and Run the Server Each time you deploy an update, stop the running service, pull the latest code, rebuild the binary, and restart: ```bash -chmod +x update_data.sh +chmod +x start-script.sh toolforge webservice stop git pull origin main go build -o Scribe-Server . toolforge webservice --mem 4Gi --cpu 2 jdk17 start \ - /data/project/scribe-server/Scribe-Server/update_data.sh + /data/project/testserver-scribe/Scribe-Server/start-script.sh + +kubectl --namespace=tool-scribe-server get ingress # check for URL +kubectl logs -l toolforge=tool --tail=50 # see logs; last 50 ``` Why this sequence: @@ -139,21 +159,38 @@ Why this sequence: - `go build -o Scribe-Server .` produces a statically-linked binary that Toolforge can execute directly. - The `--mem 4Gi --cpu 2` flags allocate enough headroom for data loading on startup. -### Install pip +### Python-based Tooling Setup(To Run [Scribe-Data Fetch Script](./update_data.sh) Successfully) If you need Python-based tooling in the project, open a Python 3.13 shell and bootstrap pip: ```bash -toolforge webservice python3.13 shell +toolforge webservice --mem 4Gi python3.13 shell + +python3 -m venv .venv source venv/bin/activate curl -sS https://bootstrap.pypa.io/get-pip.py | python ``` +then in the root of your project, run: + +```bash +./update_data.sh true # we pass `true` to skip DB migration +``` + +once done, exit the python shell by running: `exit` + +The run migration: + +```bash +go build -o ./bin/migrate-scribe-data ./cmd/migrate # to build migration file +./bin/migrate # to run the migration +``` + ### Install PyICU -> [!NOTE] +> [**!NOTE**] > The following should be done if ICU Detection Fails. The standard PyICU build uses `pkg-config` or `icu-config` to locate ICU headers and libraries. Neither tool is installed on Toolforge, so you must set the paths manually before running `pip install`: From 1fc3d349984d2f4d6d46925d3e690c4a6116eeae Mon Sep 17 00:00:00 2001 From: Andrew Tavis McAllister Date: Thu, 21 May 2026 01:14:36 +0200 Subject: [PATCH 4/4] Minor docs fixes --- Toolforge.md | 45 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/Toolforge.md b/Toolforge.md index cfd3005..5f446f1 100644 --- a/Toolforge.md +++ b/Toolforge.md @@ -1,3 +1,5 @@ + + # [Toolforge Deployment](https://toolsadmin.wikimedia.org/) [Toolforge](https://wikitech.wikimedia.org/wiki/Help:Toolforge) is Wikimedia's hosting platform for community tools. This guide walks you through deploying Scribe-Server on Toolforge from scratch — no prior Toolforge experience needed. To deploy Scribe Server here, you first apply at [toolsadmin.wikimedia.org](https://toolsadmin.wikimedia.org/) with the relevant project details. After your application is approved, you gain SSH access and can set up the environment, database, and web service described below. @@ -7,9 +9,10 @@ - [Practical Workflow](#first-steps-as-a-contributor) - [SSH into Toolforge](#ssh-into-toolforge) - [Set Up Go](#set-up-go) - - [Configure the Database](#configure-the-database) + - [Configure Database](#configure-database) + - [Create Start Script](#create-start-script) - [Build and Run the Server](#build-and-run-the-server) - - [Install pip](#install-pip) + - [Python Tooling Setup](#python-tooling-setup) - [Install PyICU](#install-pyicu) ## Practical Workflow @@ -35,6 +38,8 @@ git clone https://github.com/scribe-org/Scribe-Server.git cd Scribe-Server ``` +Back to top. + ### Set Up Go Toolforge's [pre-built web images](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Other_/_generic_web_servers) do not include Go, so you install it manually into your home directory. @@ -71,7 +76,9 @@ Why this layout: - `GOROOT` must point at your custom location because the system has no Go in `PATH`. - Running `go run .` inside Toolforge binds to `0.0.0.0:8000` — this is expected behavior within the Toolforge network. -### Configure the Database +Back to top. + +### Configure Database Toolforge provides a shared MariaDB cluster. Your credentials are pre-written to `~/replica.my.cnf` during tool creation. @@ -120,7 +127,12 @@ mysql --defaults-file=~/replica.my.cnf \ s123456__scribe_server_p ``` -### Create the Start Script (Only Once) +Back to top. + +### Create Start Script + +> [!NOTE] +> The following only needs to be ran once. ```bash ~/Scribe-Server/start-script.sh << 'EOF' @@ -131,12 +143,14 @@ export PORT=8000 EOF ``` -**Make it executable:** +Make the script executable: ```.bash chmod +x ~/Scribe-Server/start-script.sh ``` +Back to top. + ### Build and Run the Server Each time you deploy an update, stop the running service, pull the latest code, rebuild the binary, and restart: @@ -159,9 +173,14 @@ Why this sequence: - `go build -o Scribe-Server .` produces a statically-linked binary that Toolforge can execute directly. - The `--mem 4Gi --cpu 2` flags allocate enough headroom for data loading on startup. -### Python-based Tooling Setup(To Run [Scribe-Data Fetch Script](./update_data.sh) Successfully) +Back to top. + +### Python Tooling Setup + +> [!NOTE] +> The following is needed to Run [Scribe-Data Fetch Script](./update_data.sh) Successfully. -If you need Python-based tooling in the project, open a Python 3.13 shell and bootstrap pip: +If you need Python tooling in the project, open a Python 3.13 shell and bootstrap pip: ```bash toolforge webservice --mem 4Gi python3.13 shell @@ -173,24 +192,26 @@ source venv/bin/activate curl -sS https://bootstrap.pypa.io/get-pip.py | python ``` -then in the root of your project, run: +Then run the following in the root of the project: ```bash ./update_data.sh true # we pass `true` to skip DB migration ``` -once done, exit the python shell by running: `exit` +Once done, exit the python shell by running: `exit` -The run migration: +Then run the migration: ```bash go build -o ./bin/migrate-scribe-data ./cmd/migrate # to build migration file ./bin/migrate # to run the migration ``` +Back to top. + ### Install PyICU -> [**!NOTE**] +> [!NOTE] > The following should be done if ICU Detection Fails. The standard PyICU build uses `pkg-config` or `icu-config` to locate ICU headers and libraries. Neither tool is installed on Toolforge, so you must set the paths manually before running `pip install`: @@ -207,3 +228,5 @@ Why these variables: - `ICU_VERSION` tells the build script which header subdirectory to target. - `PYICU_LFLAGS` points the linker at the system ICU shared objects already present on Toolforge nodes. - `PYICU_CFLAGS` points the compiler at the system ICU headers. + +Back to top.