diff --git a/site/dolt/src/content/architecture/storage-engine.md b/site/dolt/src/content/architecture/storage-engine.md index 111feb71..05f743ec 100644 --- a/site/dolt/src/content/architecture/storage-engine.md +++ b/site/dolt/src/content/architecture/storage-engine.md @@ -54,7 +54,7 @@ Databases are built on top of [Search Trees](https://en.wikipedia.org/wiki/Searc Dolt is built on a novel Search Tree, closely related to a B-tree, called a Probabilistic B-Tree, or Prolly Tree for short. As far as we can tell, Prolly Trees were [invented by the Noms team specifically for database version control](https://github.com/attic-labs/noms/blob/master/doc/intro) and they also coined the term. -## B-Trees +### B-Trees Most SQL databases you are familiar with, like [Postgres](https://www.postgresql.org/) or [MySQL](https://www.mysql.com/), are built on [B-tree](https://www.dolthub.com/blog/2020-04-01-how-dolt-stores-table-data/#b-tree-review) storage. Tables are represented as a map of primary keys to values and the keys of that map are stored in a B-tree. Values are stored in the leaf nodes. @@ -66,7 +66,7 @@ However, finding the differences between two B-trees requires scanning both tree Also, writes to B-trees are not history independent, the order of the writes internally changes the structure of the tree. Thus, storage cannot be easily shared between two versions of the same tree. -## Prolly Trees +### Prolly Trees A Prolly Tree, or Probabilistic B-tree, is a content-addressed B-tree. @@ -82,7 +82,7 @@ Moreover, sections of the tree that share the same root hash can share storage b Prolly trees are described in more detail [here](/architecture/storage-engine/prolly-tree). -## Comparison +### Comparison [The Noms documentation](https://github.com/attic-labs/noms/blob/master/doc/intro#some-properties-of-prolly-trees) provides the following useful algorithmic, big O() comparison of B-trees and Prolly Trees: diff --git a/site/dolt/src/content/introduction/use-cases/audit.md b/site/dolt/src/content/introduction/use-cases/audit.md index 3fdd0415..de320132 100644 --- a/site/dolt/src/content/introduction/use-cases/audit.md +++ b/site/dolt/src/content/introduction/use-cases/audit.md @@ -23,11 +23,11 @@ If you're not ready to switch your primary database to Dolt to get its audit cap ## Dolt replaces... -## Soft Deletes +### Soft Deletes A technique to add audit capability to an existing database is to add [soft deletes](https://www.dolthub.com/blog/2022-11-03-soft-deletes/). Soft delete is the use various techniques to mark data as inactive instead of deleting it. This is strictly worse than a version controlled database for audit purposes. With soft deletes, an operator can still modify data or the application can make mistakes. In Dolt, every write is part of the audit log. It is far more difficult for an operator to change Dolt history. -## Change Data Capture +### Change Data Capture [Change Data Capture](https://www.dolthub.com/blog/2023-03-01-change-data-capture/) is another way to add audit capability to an existing database. Some change data capture techniques are similar to [soft delete](https://www.dolthub.com/blog/2022-11-03-soft-deletes/) strategies. Modern change data capture tools consume replication logs to audit database changes. Dolt can consume the same logs in the [versioned MySQL replica use case](/introduction/use-cases/versioned-replica) producing a simpler and thus, more audit-friendly, change data capture solution. diff --git a/site/dolt/src/content/introduction/use-cases/configuration-management.md b/site/dolt/src/content/introduction/use-cases/configuration-management.md index 51a5dbe9..94f00cb4 100644 --- a/site/dolt/src/content/introduction/use-cases/configuration-management.md +++ b/site/dolt/src/content/introduction/use-cases/configuration-management.md @@ -24,7 +24,7 @@ This use case is particularly popular in video games where much of the game func ## Dolt replaces... -## Files in Git +### Files in Git Most large configuration files are stored and versioned in Git. If the files get too large they are store in cloud storage and linked to Git using [git-lfs](https://git-lfs.com/). If the files are stored in git-lfs, you lose the ability to diff the contents of the files. Dolt improves the experience by adding query capabilities and large fine-grained diffs to the data stored in configuration files. The diff and merge experience will be greatly improved in Dolt for this type of data. diff --git a/site/dolt/src/content/introduction/use-cases/data-and-model-quality.md b/site/dolt/src/content/introduction/use-cases/data-and-model-quality.md index 5d66065e..de86369c 100644 --- a/site/dolt/src/content/introduction/use-cases/data-and-model-quality.md +++ b/site/dolt/src/content/introduction/use-cases/data-and-model-quality.md @@ -31,11 +31,11 @@ Lastly, [commits](/concepts/dolt/git/commits), [logs](/concepts/dolt/git/log), a ## Dolt replaces... -## Unstructured files in cloud storage +### Unstructured files in cloud storage It is common practice to store copies of training data or database backups in cloud storage for model reproducibility. A full copy of the data is stored for every training run. This can become quite expensive and limit the amount of models you can reproduce. Dolt stores only the differences between stored versions decreasing the cost of data storage. Additionally, Dolt can produce diffs between versions of training data producing novel model insights. -## MySQL, Postgres, or other databases +### MySQL, Postgres, or other databases Dolt can replace any database used to store and query data. Many of our customers switch from other OLTP databases like MySQL or Postgres to improve data and model quality through versioning. Customers have also switched to Dolt from document databases like MongoDB. Dolt's additional unique features like branches, diffs, and merges allow for human review of data changes and multiple parallel data projects. diff --git a/site/dolt/src/content/introduction/use-cases/data-sharing.md b/site/dolt/src/content/introduction/use-cases/data-sharing.md index 892cea59..bf5352dc 100644 --- a/site/dolt/src/content/introduction/use-cases/data-sharing.md +++ b/site/dolt/src/content/introduction/use-cases/data-sharing.md @@ -27,11 +27,11 @@ Dolt and DoltHub are also great if vendors share data with you. When you receive ## Dolt replaces... -## Exchanging Files +### Exchanging Files Dolt replaces exchanging flat data files like CSVs via email, FTP servers, or other file transfer techniques. Dolt allows data to maintain schema on exchange including constraints, triggers, and views. This more rich format of exchange reduces transfer errors. Dolt also allows you to change the data to fit your needs and still get updates from your source. Dolt will notify you if your changes [conflict](/concepts/dolt/git/conflicts) with the source. -## External APIs +### External APIs Dolt is ideal for sharing data that does not have an API. But even for data with an API, Dolt is often more convenient. With Dolt, you get all the data and its history. With APIs you often have to assemble the data with multiple API calls. With APIs, the data can change out from under you, whereas with Dolt you can read a version of the data until you are ready to upgrade. DoltHub ships with a [SQL API](/products/dolthub/api/v1alpha1/sql) so you can choose the data sharing solution that is right for your use case. diff --git a/site/dolt/src/content/introduction/use-cases/manual-data-curation.md b/site/dolt/src/content/introduction/use-cases/manual-data-curation.md index f580c2e1..05ffffae 100644 --- a/site/dolt/src/content/introduction/use-cases/manual-data-curation.md +++ b/site/dolt/src/content/introduction/use-cases/manual-data-curation.md @@ -20,7 +20,7 @@ Dolt is a MySQL compatible database so exporting the manually created data to pr ## Dolt replaces... -## Spreadsheets +### Spreadsheets Dolt replaces Excel or Google Sheets for manual data curation. Versioning features allow for more efficient asynchronous collaboration and human review of data changes. The DoltHub interface is still easy enough for non-technical users to contribute and review data changes. diff --git a/site/dolt/src/content/introduction/use-cases/offline-first.md b/site/dolt/src/content/introduction/use-cases/offline-first.md index 5f6980c3..fadc87d0 100644 --- a/site/dolt/src/content/introduction/use-cases/offline-first.md +++ b/site/dolt/src/content/introduction/use-cases/offline-first.md @@ -23,7 +23,7 @@ Conflicting writes are surfaced quickly and an operator or software can take add ## Dolt replaces -## Custom syncing processes +### Custom syncing processes Dolt replaces custom code to synchronize your client and server. This code is complicated and hard to get right. The Git [remote](/concepts/dolt/git/remotes) model of clone, fetch, push, and pull is a proven synchronization model. Dolt brings this model to the database allowing you to remove most of your synchronization code. diff --git a/site/dolt/src/content/introduction/use-cases/vc-your-app.md b/site/dolt/src/content/introduction/use-cases/vc-your-app.md index 53c45eaf..1e7d0d11 100644 --- a/site/dolt/src/content/introduction/use-cases/vc-your-app.md +++ b/site/dolt/src/content/introduction/use-cases/vc-your-app.md @@ -25,11 +25,11 @@ In the past applications that needed these features required [slowly changing di ## Dolt replaces -## Soft Deletes +### Soft Deletes A common technique to version your database is to use [soft deletes](https://www.dolthub.com/blog/2022-11-03-soft-deletes/). When your application would make an update or a delete, you application instead makes an insert and marks the old row invalid. Dolt obviates the need for this technique. You can keep your existing database schema and Dolt ensures every write is non-destructive. Queries against soft deleted rows become Dolt history queries against [system tables](/sql-reference/version-control/dolt-system-tables). -## Slowly Changing Dimension +### Slowly Changing Dimension A more advanced technique for versioning databases is [slowly changing dimension](https://www.dolthub.com/blog/2023-06-22-slowly-changing-dimension/). Slowly Changing Dimension is similar to soft deletes. Additional database columns are added to tables to manage versioning. Dolt is slowly changing dimension on every table by default. Queries involving the slowly changing dimension become Dolt history queries against [system tables](/sql-reference/version-control/dolt-system-tables). Moreover, complicated [merge](/concepts/dolt/git/merge) processes can happen at the database layer. Merges must handled by custom code at the application layer with slowly changing dimension. diff --git a/site/dolt/src/content/introduction/use-cases/versioned-replica.md b/site/dolt/src/content/introduction/use-cases/versioned-replica.md index 86d6c941..ead0cecf 100644 --- a/site/dolt/src/content/introduction/use-cases/versioned-replica.md +++ b/site/dolt/src/content/introduction/use-cases/versioned-replica.md @@ -25,11 +25,11 @@ Additionally, a Dolt replica can be easily cloned (ie. copied) to a developer's ## Dolt replaces... -## Backups and Transaction Logs +### Backups and Transaction Logs Dolt as a versioned replica becomes your first line of defense against a bad operator query, script, or deployment. Dolt is online and contains the full history of your database. In a disaster you can use diffs to find a bad query and roll it back. Then you can produce a database patch and apply it to production. You do not need to reinstall from a backup and play the transaction log back to the point of the failure, an extremely time consuming process. -## Change Data Capture +### Change Data Capture [Change Data Capture](https://www.dolthub.com/blog/2023-03-01-change-data-capture/) is a way to add a history of data changes to an existing database. Modern change data capture tools consume replication logs to produce database changes in a consumable stream. Dolt can consume the same logs producing a simpler change data capture solution. diff --git a/site/dolt/src/content/products/dolthub/continuous-integration/README.md b/site/dolt/src/content/products/dolthub/continuous-integration/README.md index 9f231cce..7a67379e 100644 --- a/site/dolt/src/content/products/dolthub/continuous-integration/README.md +++ b/site/dolt/src/content/products/dolthub/continuous-integration/README.md @@ -18,7 +18,7 @@ The following sections will introduce you to how CI works with Dolt, DoltHub and CI configuration for a DoltHub or DoltLab database is stored in the database itself. At the time of this writing, in order to add CI configuration to a DoltHub or DoltLab database, you will need to have a local Dolt client version >= [v1.45.3](https://github.com/dolthub/dolt/releases/tag/v1.45.3) and will have to clone a copy of the the database. In order to configure CI on the database, you will use Dolt's CI CLI commands. -## Dolt CI Commands +### Dolt CI Commands The primary interface for creating and editing CI configuration in a Dolt database is via the `dolt ci` CLI command. These commands aim to simplify CI configuration in Dolt, so that users do not need to manually interact with the underlying CI tables directly. @@ -57,7 +57,7 @@ A workflow Job is made up of a series of Steps. A step, in its current form, is For more information on Steps, please see the [workflow reference](/products/dolthub/continuous-integration/reference). -## Saved Query +### Saved Query A [Saved Query](/sql-reference/version-control/saved-queries) is a SQL query that is stored and versioned in a Dolt database. For the purpose of DoltHub and DoltLab CI, this allows users to write a SQL query that will be executed on command at a later time, during a CI run. diff --git a/site/dolt/src/content/products/hosted/cloning.md b/site/dolt/src/content/products/hosted/cloning.md index 606e752a..4ad63ed5 100644 --- a/site/dolt/src/content/products/hosted/cloning.md +++ b/site/dolt/src/content/products/hosted/cloning.md @@ -33,7 +33,7 @@ analytics query. [Clone](/cli-reference/cli#dolt-clone) makes it easy to get a l copy of your Hosted database with one command. Do whatever potentially destructive or performance-degrading operations on your laptop while respecting your production database. -## 1. Expose remotesapi endpoint +### 1. Expose remotesapi endpoint In order to enable cloning from your Hosted database, first expose the [remotesapi endpoint](/cli-reference/cli#dolt-sql-server). This will set the port for a server @@ -48,7 +48,7 @@ deployment will also work. ![](../../.gitbook/assets/hosted-create-deployment-remotesapi.png) -## 2. Set remote password and run clone command with user flag +### 2. Set remote password and run clone command with user flag To authenticate against it, you have to set a `DOLT_REMOTE_PASSWORD` environment variable and pass along a `--user` flag to the `dolt clone` command. You can find these @@ -92,7 +92,7 @@ cloning https://dolthub-us-housing.dbs.hosted.doltdb.com/us-housing-prices +----------+ ``` -## 3. Sync your local copy with upstream changes +### 3. Sync your local copy with upstream changes Now we can run whatever queries or schema migrations we want without affecting production. If there are updates to the database, easily sync your local copy using [`dolt @@ -107,7 +107,7 @@ commands to your remotesapi endpoint in the same way by passing the `--user` fla % dolt fetch --user "[username]" ``` -## 4. Sync your upstream with changes from local copy +### 4. Sync your upstream with changes from local copy If you make any changes on your local copy, you can push them to your upstream using [`dolt push`](/cli-reference/cli#dolt-push). Push to a feature branch rather than @@ -127,7 +127,7 @@ You can also push straight to `main` if you'd rather skip review, subject to any % dolt push origin --user "[username]" HEAD:main ``` -## 5. Open a pull request +### 5. Open a pull request Hosted Dolt doesn't have forks — that's a DoltHub and DoltLab feature. To contribute a change for review, push a feature branch to the deployment (as in step 4 above) and diff --git a/site/dolt/src/content/products/hosted/dolthub-as-remote.md b/site/dolt/src/content/products/hosted/dolthub-as-remote.md index 5437d3eb..179435c6 100644 --- a/site/dolt/src/content/products/hosted/dolthub-as-remote.md +++ b/site/dolt/src/content/products/hosted/dolthub-as-remote.md @@ -40,7 +40,7 @@ pull, and fetch from public or private databases. ## Example -## 1. Find a database on DoltHub to clone +### 1. Find a database on DoltHub to clone We have an example user metrics database on DoltHub named `dolthub/user_metrics`. Since we don't want our metrics to be publicly accessible, the database is private. @@ -49,7 +49,7 @@ We want to host this data on Hosted Dolt so that we can use [Google Looker Studio](https://lookerstudio.google.com/) to visualize our data. Learn more about that process [here](https://www.dolthub.com/blog/2023-02-13-dolt-looker/). -## 2. Create a new deployment on Hosted +### 2. Create a new deployment on Hosted Next, I create a [new deployment](https://hosted.doltdb.com/create-deployment) on Hosted and check the `Expose Dolt credentials` check box from the form. @@ -65,14 +65,14 @@ will have access to this key. If I accidentally expose my key or decide I want to remove it, I also have those options there. -## 3. Add public key to DoltHub +### 3. Add public key to DoltHub I click on "Add to DoltHub" to add my public key to DoltHub. This will open my DoltHub [credentials settings page](https://www.dolthub.com/settings/credentials). ![](../../.gitbook/assets/dolthub-credentials-for-hosted.png) -## 4. Connect to Hosted instance and clone DoltHub database +### 4. Connect to Hosted instance and clone DoltHub database Now that I have my credentials set up, I can connect to the Hosted instance using the information in the Connectivity tab and clone my `dolthub/user_metrics` database. @@ -128,7 +128,7 @@ Now I can do whatever I want with my metrics data, including following [these steps](https://www.dolthub.com/blog/2023-02-13-dolt-looker/#create-a-data-source) to connect my Hosted instance to Looker. -## 5. Make changes to database on DoltHub +### 5. Make changes to database on DoltHub Finance reviews our metrics charts and finds a hole in the data suggesting an outage. @@ -139,7 +139,7 @@ editor and create a pull request. They don't even need to know SQL! ![](../../.gitbook/assets/dolthub-spreadsheet-editor-user-metrics.png) -## 6. Pull new DoltHub branch to Hosted instance for testing +### 6. Pull new DoltHub branch to Hosted instance for testing We want to review the new chart from the change before we merge it to `main`. We can pull that branch to our Hosted instance and use it to [create a new branch in @@ -167,7 +167,7 @@ mysql> select * from dolt_diff('main...outage-estimates', 'user_counts'); ``` -## 7. Make a change from Hosted and push it back to DoltHub +### 7. Make a change from Hosted and push it back to DoltHub We are mostly satisfied with the new chart, but I want to make a small update to the metrics for one of the days. I can make the change from Hosted on a branch and push it @@ -207,7 +207,7 @@ I will see the new branch in my database on DoltHub. ![](../../.gitbook/assets/dolthub-outage-updates-branch.png) -## 8. Merge branches into `main` on DoltHub and pull again to Hosted +### 8. Merge branches into `main` on DoltHub and pull again to Hosted From there, our changes are approved and merged into the `main` branch on DoltHub. One more `dolt_pull` will update our `main` branch on Hosted. diff --git a/site/dolt/src/content/products/hosted/getting-started.md b/site/dolt/src/content/products/hosted/getting-started.md index e28df6b0..7850e98b 100644 --- a/site/dolt/src/content/products/hosted/getting-started.md +++ b/site/dolt/src/content/products/hosted/getting-started.md @@ -55,7 +55,7 @@ There are three ways to read or write from Hosted Dolt. You can: In this blog, we will show off (1) and (2) but we're going to start with the Workbench because it is the easiest to use. -## Start the Workbench +### Start the Workbench Click on the Workbench tab of your deployment. @@ -67,7 +67,7 @@ The workbench has writes off by default. We will turn those on and create a data This should feel like a standard SQL workbench like Tableplus or Datagrip but it's web-based and has some extra Dolt-specific features like a Commit Log and Pull Requests. -## Create Some Tables +### Create Some Tables Now let's create some tables using SQL. We're going to enter the following SQL queries into the query box. You have to run them one at a time. @@ -99,7 +99,7 @@ I finish up by subsequently running the last two create table queries. ![](../../.gitbook/assets/hosted-getting-started/multi-table-diff.png) -## Create a Dolt Commit +### Create a Dolt Commit Now time to use my first Dolt feature! I'm going to create a [Dolt Commit](/concepts/dolt/git/commits). Make a Dolt commit when you want to preserve the state of the Dolt database permanently for future reference. @@ -145,7 +145,7 @@ Taylor can now access the deployment page to get connectivity information. Her m Dolt is a MySQL-compatible database. You can connect any client that can connect to MySQL to it. We're going to use the MySQl client that comes with MySQL in this section to connect to Dolt. -## Install +### Install Head over to the [MySQL Getting Started documentation](https://dev.mysql.com/doc/mysql-getting-started/en/) and install MySQL on your machine. I used [Homebrew](https://brew.sh/) to install MySQL on my Mac. @@ -156,7 +156,7 @@ MySQL comes with a MySQL server called `mysqld` and a MySQL client called `mysql mysql Ver 8.0.29 for macos12.2 on x86_64 (Homebrew) ``` -## Connect +### Connect Now, to connect the mysql client to Dolt, you need the host, port, username, and password from the Connectivity tab. @@ -186,7 +186,7 @@ mysql> This MySQL client is connected to your Hosted Dolt instance. Any changes you make here will be visible to users of the workbench or any other client connected to the Hosted Dolt database. -## Create a branch +### Create a branch So let's be safe and make our changes on a [branch](/concepts/dolt/git/branch). A branch in Dolt is a lightweight way of isolating your changes from the "main" copy of the database. Since our goal is to make a Pull Request in the SQL Workbench, making a branch is necessary as pull requests are done between two branches, in this case "main" and our new branch. @@ -216,7 +216,7 @@ mysql> select active_branch(); Great, Taylor is now on a new branch and can safely make her changes. -## Insert some rows +### Insert some rows This is easy if you know SQL. In the MySQL client on the `inserts` branch, Taylor ran the following SQL to add a few of the early employees here at DoltHub and assign them to teams. @@ -273,7 +273,7 @@ mysql> call dolt_commit('-m', 'inserted early employees'); Let's head back over to the workbench to make a Pull Request. -## Make and Review a Pull Request +### Make and Review a Pull Request The Hosted Dolt workbench supports pull requests and human review of your Hosted Dolt database. Pull requests on Hosted are always opened and merged from the workbench — there are no forks (forks are a [DoltHub](/concepts/dolthub/prs) / [DoltLab](/products/doltlab) feature) and no CLI/SQL surface for opening a PR. The branch you're proposing for review can come from anywhere (the workbench, a MySQL client, a [local clone](/products/hosted/cloning), automation), but the PR itself is created in the web UI. @@ -299,7 +299,7 @@ Finally, she clicks "Create pull request". She now sends me this Pull Request li ![](../../.gitbook/assets/hosted-getting-started/workbench-pr.png) -## Review and Merge +### Review and Merge On the Pull Request Page, I have access to a human readable diff of the changes under review. diff --git a/site/dolt/src/content/products/hosted/notable-features.md b/site/dolt/src/content/products/hosted/notable-features.md index 48a38c4c..93b03648 100644 --- a/site/dolt/src/content/products/hosted/notable-features.md +++ b/site/dolt/src/content/products/hosted/notable-features.md @@ -18,21 +18,21 @@ example: ## Operations -## Logs +### Logs While your database is running we collect logging so you can monitor your database. View and download logs from any time period from the deployment console. ![](../../.gitbook/assets/hosted-logs.png) -## Monitoring +### Monitoring We also collect metrics on your running database and display graphs so you can see performance and usage in real time. ![](../../.gitbook/assets/hosted-monitoring.png) -## Custom Configuration +### Custom Configuration Dolt has a myriad of [configuration options](/sql-reference/server/configuration). Most of these can be changed on the running server by setting the appropriate [system @@ -41,14 +41,14 @@ provides a simple UI for viewing and changing your deployment's custom configura ![](../../.gitbook/assets/hosted-custom-config.png) -## Replication +### Replication Scale your deployment read capacity or create hot failover instances by enabling read replication for your deployment. Configure how many replicas you want. You'll get logs and metrics for each server. ![](../../.gitbook/assets/hosted-replica-graphs.png) -## Backups +### Backups Full copies of all your databases are backed up nightly. These backups are kept for 14 days. @@ -66,14 +66,14 @@ requires you to create a support ticket. Learn more about backups [here](https://www.dolthub.com/blog/2022-08-31-hosted-backups/). -## Private Networking +### Private Networking Want to use Hosted Dolt, but don't want your database to be reachable through the public internet. We support both AWS and GCP private networking. [Private networking setup instructions](/products/hosted/private-networking) -## Dolt Upgrades +### Dolt Upgrades We're releasing [new versions of Dolt](https://github.com/dolthub/dolt/releases) at least once per week. Set your service window and your deployed Dolt version will automatically @@ -84,7 +84,7 @@ get upgraded weekly to the latest available version with minimal downtime. Or if you want to upgrade Dolt immediately, you can use the "Update Dolt version now" button in the Actions dropdown. -## Enterprise Support +### Enterprise Support Hosted Dolt comes with enterprise support. [Our team](https://www.dolthub.com/team) of veteran cloud service engineers will keep your database operating smoothly. @@ -97,7 +97,7 @@ will get back to you within the hour. Learn more about our support ticket system [here](https://www.dolthub.com/blog/2022-07-13-hosted-dolt-incident-manager/). -## Access Management +### Access Management Easily manage access to your deployment using the same permissions model as [DoltHub](/concepts/dolthub/permissions) and @@ -121,7 +121,7 @@ a Branch Permissions UI. See ## Quick Start -## Trial Instance +### Trial Instance To lower the cost barrier, Hosted Dolt offers [trial instance](https://www.dolthub.com/blog/2022-10-24-hosted-trial-instances/) for $50 a @@ -129,7 +129,7 @@ month. For $50 you get a t2.medium running in EC2 with 50GB of storage. There is limit for these instances. This isn't a 30-day trial. It's an instance perfect for trialing Dolt. -## Connect Using Any MySQL Client +### Connect Using Any MySQL Client Dolt is a drop in replacement for MySQL. You can connect to your instance from anywhere using your favorite MySQL client using the connectivity information in the Connectivity @@ -139,7 +139,7 @@ tab of your deployment console. ## SQL Workbench -## User Friendly Built-In Web GUI +### User Friendly Built-In Web GUI Each deployment comes with a built-in SQL workbench, inspired by [DoltHub's](https://www.dolthub.com) database page. Browse your data in read-only mode or @@ -151,31 +151,31 @@ favorite DoltHub features, including [pull requests](#pull-requests), [diffs](#d Learn more about the SQL workbench [here](/products/hosted/sql-workbench). -## Pull Requests +### Pull Requests Create pull requests for human review of changes to your database. ![](../../.gitbook/assets/hosted-workbench-pr.png) -## Diffs +### Diffs Analyze your data and schema changes in a diff and debug issues in specific commits. ![](../../.gitbook/assets/hosted-workbench-pr-diff.png) -## Commit Log +### Commit Log View an audit log of all changes made to your data. ![](../../.gitbook/assets/hosted-workbench-commit-log.png) -## ER Diagrams +### ER Diagrams Visualize the entities in your database as well as the relationship between tables. ![](../../.gitbook/assets/hosted-workbench-er-diagram.png) -## CSV Upload and Download +### CSV Upload and Download Update database tables using an uploaded CSV or the built-in spreadsheet editor. @@ -189,7 +189,7 @@ Learn more in [our blog](https://www.dolthub.com/blog/2023-06-30-hosted-workbenc ## Dolt Ecosystem -## Clone a Hosted Instance +### Clone a Hosted Instance In some cases you might want to clone your database from Hosted Dolt so that you can access Dolt's [command line interface](/cli-reference/cli). Hosted provides an option @@ -200,7 +200,7 @@ Hosted. Learn more in our [cloning guide](/products/hosted/cloning). -## Use DoltHub as a Remote +### Use DoltHub as a Remote To interact with DoltHub from your Hosted instance, you can use DoltHub as a [remote](/concepts/dolt/git/remotes). We have [SQL diff --git a/site/dolt/src/content/reference/sql/server/garbage-collection.md b/site/dolt/src/content/reference/sql/server/garbage-collection.md index e3166ccf..67e5efa2 100644 --- a/site/dolt/src/content/reference/sql/server/garbage-collection.md +++ b/site/dolt/src/content/reference/sql/server/garbage-collection.md @@ -37,11 +37,11 @@ $ dolt sql --disable-auto-gc Garbage collection can be run offline using [`dolt gc`](/cli-reference/cli#dolt-gc) or online using [`call dolt_gc()`](/sql-reference/version-control/dolt-sql-procedures#dolt_gc). -## Offline +### Offline If you have access to the server where your Dolt database is located and a Dolt sql-server is not running, navigate to the directory your database is stored in and run `dolt gc`. This will cycle through all the needed chunks in your database and delete those that are unnecessary. This process is CPU and memory intensive. -## Online, with Automatic GC disabled +### Online, with Automatic GC disabled If you have disabled Automatic GC, you can run garbage collection on your running SQL server using [`call dolt_gc`](/sql-reference/version-control/dolt-sql-procedures#dolt_gc) through any connected client. To prevent concurrent writes potentially referencing garbage collected chunks, running diff --git a/site/dolt/src/content/reference/sql/server/troubleshooting.md b/site/dolt/src/content/reference/sql/server/troubleshooting.md index 4f01afc5..ad24ef40 100644 --- a/site/dolt/src/content/reference/sql/server/troubleshooting.md +++ b/site/dolt/src/content/reference/sql/server/troubleshooting.md @@ -7,27 +7,27 @@ Debugging a running Dolt server can be challenging. This document covers the deb ## Basics -## Make sure you are running the latest Dolt version +### Make sure you are running the latest Dolt version Dolt is constantly evolving. We release a new Dolt approximately once a week. Connect to the SQL server and run `select dolt_version()`. Make sure the version matches the latest as seen on the [GitHub releases page](https://github.com/dolthub/dolt/releases). To upgrade the server, download the latest Dolt binary for your platform and replace the Dolt binary on your `PATH` with the downloaded one. Running the install process on most platforms again will do this for you. Restart the Dolt server using `dolt sql-server` to have your running server start using the latest binary. -## Examine your CPU, Memory, and Disk usage +### Examine your CPU, Memory, and Disk usage Dolt consumes CPU, Memory, and Disk. Consuming more of any of these resources than the host has available can lead to degraded performance. Use your system's built in resource monitoring systems to inspect Dolt's usage of these resources. You may need a larger host or additional [read replicas](/sql-reference/server/replication) to support your load. -## Set your log level to DEBUG or TRACE +### Set your log level to DEBUG or TRACE To see queries being run against the server, query results, and query latency set your Dolt log level to `DEBUG` or `TRACE`. This can be done by starting the server like so `dolt sql-server --loglevel=debug` or by setting `log_level: debug` in your `config.yaml`. Your logs should be visible in the shell you started `dolt sql-server` in. -## EXPLAIN PLAN for complex queries +### EXPLAIN PLAN for complex queries Dolt supports the SQL `EXPLAIN PLAN` operation in order for you to see the plan for complex queries. Rearranging your query to perform fewer `JOIN`s or make better use of indexes can help speed up complex queries. Note: `EXPLAIN` currently returns MySQL-consistent but otherwise no-op output. Use `EXPLAIN PLAN` for Dolt formatted plans. -## Compare to MySQL +### Compare to MySQL Dolt strives to be 100% MySQL compatible. If you run a query that works in MySQL but does not work in Dolt, it is a Dolt bug and you should [submit an issue](#submitting-issues). You can dump your Dolt database using [`dolt dump`](/cli-reference/cli#dolt-dump) and import the resulting file into MySQL using `mysql < dump.sql`. The test the query you think should work using any MySQL client. @@ -39,7 +39,7 @@ If you run into any issues requiring engineering attention, please submit a [Git Dolt operational issues usually manifest as slow SQL queries. In rare occasions, Dolt may consume more of your system's resources than you expect. In these cases, this document has some recommendations. -## Server Consuming Disk +### Server Consuming Disk Dolt creates disk garbage on write. As of Dolt 1.75, automatic garbage collection is on by default. Thus, you should not experience disk garbage accumulation under normal operation for newer versions of Dolt. @@ -50,7 +50,7 @@ Another potential cause is a commit-heavy workflow that uses a database design t * Using primary keys with random values. Inserts into indexes with random values guarantees that edits will occur all throughout the index instead of being clustered around the same key space. This results in a rewrite of the Prolly Tree thereby increasing storage disproportionately to the delta of the changes. * Adding a column to a table. A new column forks the storage of the table resulting in a loss of structural sharing. Dolt is row major and builds chunks for each primary key, row values pair. The row values encodes the schema length so every row now requires a new chunk. -## Server Consuming Memory +### Server Consuming Memory Serving Dolt databases requires a fair amount of memory. As a general rule, we recommend a minimum of 2GB available RAM for any production use case. Larger databases or heavier workloads should start @@ -70,7 +70,7 @@ then frees the memory upon restart, you have discoverd a memory leak. Again, ple issue](https://github.com/dolthub/dolt/issues). Memory leaks should be rare and we treat memory leak fixes as high priority. -## Server Consuming CPU +### Server Consuming CPU Under too much concurrent load, Dolt may consume all the CPU on a host. This is likely caused by too much read concurrency. In this case, create more [read replicas](/sql-reference/server/replication) and load balance @@ -79,7 +79,7 @@ your reads among your replicas. If you discover a query consuming all of your CPU, please submit a [GitHub Issue](https://github.com/dolthub/dolt/issues). On rare occasions, this could be a Dolt bug. -## Too much write concurrency +### Too much write concurrency Currently, Dolt is not a high-throughput database for writes. The current transaction model serializes all writes, which means that after a certain threshold of writer concurrency, you'll diff --git a/site/dolt/src/content/reference/sql/sql-support/expressions-functions-operators.md b/site/dolt/src/content/reference/sql/sql-support/expressions-functions-operators.md index e391f5ab..d40d970b 100644 --- a/site/dolt/src/content/reference/sql/sql-support/expressions-functions-operators.md +++ b/site/dolt/src/content/reference/sql/sql-support/expressions-functions-operators.md @@ -3,8 +3,6 @@ title: "Expressions, Functions, and Operators" description: Supported scalar functions, operators, and expressions. --- -## Expressions, Functions, Operators - ## Statements | Component | Supported | Notes and limitations | diff --git a/site/doltgres/src/content/reference/sql/server/troubleshooting.md b/site/doltgres/src/content/reference/sql/server/troubleshooting.md index 94bfc92d..f75253ae 100644 --- a/site/doltgres/src/content/reference/sql/server/troubleshooting.md +++ b/site/doltgres/src/content/reference/sql/server/troubleshooting.md @@ -6,7 +6,7 @@ Debugging a running Doltgres server can be challenging. This document covers the ## Basics -## Make sure you are running the latest Doltgres version +### Make sure you are running the latest Doltgres version Doltgres is constantly evolving. We release a new Doltgres approximately once a week. Connect to the SQL server and run `select dolt_version()`. Make sure the version matches the latest as seen on the @@ -14,15 +14,15 @@ SQL server and run `select dolt_version()`. Make sure the version matches the la To upgrade the server, download the latest Doltgres binary for your platform and replace the Doltgres binary on your `PATH` with the downloaded one. Running the install process on most platforms again will do this for you. Restart the Doltgres server using `doltgres` to have your running server start using the latest binary. -## Examine your CPU, Memory, and Disk usage +### Examine your CPU, Memory, and Disk usage Doltgres consumes CPU, Memory, and Disk. Consuming more of any of these resources than the host has available can lead to degraded performance. Use your system's built in resource monitoring systems to inspect Doltgres's usage of these resources. You may need a larger host or additional [read replicas](/reference/server/replication) to support your load. -## Set your log level to DEBUG or TRACE +### Set your log level to DEBUG or TRACE To see queries being run against the server, query results, and query latency set your Doltgres log level to `DEBUG` or `TRACE`. This can be done by setting `log_level: debug` in your `config.yaml`. Your logs should be visible in the shell you started `doltgres` in. -## Compare to Postgres +### Compare to Postgres Doltgres strives to be 100% Postgres compatible. If you run a query that works in Postgres but does not work in Doltgres, it is a Doltgres bug and you should [submit an issue](#submitting-issues). You can dump your Doltgres database using the `pg_dump` tool and import the resulting file into PostgreSQL. Then test the query you think should work against both servers using any Postgres client and compare the results. @@ -34,7 +34,7 @@ If you run into any issues requiring engineering attention, please submit a [Git Doltgres operational issues usually manifest as slow SQL queries. In rare occasions, Doltgres may consume more of your system's resources than you expect. In these cases, this document has some recommendations. -## Server Consuming Disk +### Server Consuming Disk Doltgres creates disk garbage on write. This can sometimes become a substantial portion of the disk Doltgres is consuming. Doltgres ships with a garbage collection function. Running the garbage collection function can free disk. @@ -47,7 +47,7 @@ Another potential cause is a commit-heavy workflow that uses a database design t - Using primary keys with random values. Inserts into indexes with random values guarantees that edits will occur all throughout the index instead of being clustered around the same key space. This results in a rewrite of the prolly tree thereby increasing storage disproportionately to the delta of the changes. - Adding a column to a table. A new column forks the storage of the table resulting in a loss of structural sharing. Doltgres is row major and builds chunks for each primary key, row values pair. The row values encodes the schema length so every row now requires a new chunk. -## Server Consuming Memory +### Server Consuming Memory Serving Doltgres databases requires a fair amount of memory. As a general rule, we recommend a minimum of 2GB available RAM for any production use case. Larger databases or heavier workloads should start @@ -67,7 +67,7 @@ then frees the memory upon restart, you have discoverd a memory leak. Again, ple issue](https://github.com/dolthub/doltgresql/issues). Memory leaks should be rare and we treat memory leak fixes as high priority. -## Server Consuming CPU +### Server Consuming CPU Under too much concurrent load, Doltgres may consume all the CPU on a host. This is likely caused by too much read concurrency. In this case, create more [read replicas](/reference/server/replication) and load balance @@ -76,7 +76,7 @@ your reads among your replicas. If you discover a query consuming all of your CPU, please submit a [GitHub Issue](https://github.com/dolthub/doltgresql/issues). On rare occasions, this could be a Doltgres bug. -## Too much write concurrency +### Too much write concurrency Currently, Doltgres is not a high-throughput database for writes. The current transaction model serializes all writes, which means that after a certain threshold of writer concurrency, you'll diff --git a/site/doltlab/src/content/introduction/getting-started/getting-started.md b/site/doltlab/src/content/introduction/getting-started/getting-started.md index 399ce21c..dc61ef5a 100644 --- a/site/doltlab/src/content/introduction/getting-started/getting-started.md +++ b/site/doltlab/src/content/introduction/getting-started/getting-started.md @@ -80,7 +80,7 @@ ubuntu@ip-10-2-0-24:~/doltlab$ ./ubuntu_install.sh There are a few dependencies. This will take a couple minutes. -## Make sure Docker works +### Make sure Docker works DoltLab uses Docker to run all of its services. After the bootstrap script is done, make sure Docker works with sudo by running: @@ -197,7 +197,7 @@ The DoltLab instance ships with a default user configured called `admin`. It has ## Basic Configuration -## Single user +### Single user A configured email server is required to create new users. If you try to create a user, you'll get an error that looks like this: @@ -207,7 +207,7 @@ So, you only have a single admin user to play with for now. > UPDATE: as of DoltLab v2.2.0, an email server is no longer required to create new users on a DoltLab instance. New users will be able to create accounts on an a DoltLab instance openly, unless [account whitelisting](/guides/basic#prevent-unauthorized-user-account-creation) has been enabled by the administrator. Additionally, email server configuration is now an exclusive [DoltLab Enterprise feature](/guides/enterprise#connect-doltlab-to-an-smtp-server). -## Create/Modify databases +### Create/Modify databases You can build whatever database you can imagine using the web user interface. The built in SQL workbench can be used to create and edit tables. You can import CSVs. You can edit a table using the spreadsheet editor. You can make branches and Pull Requests. @@ -215,7 +215,7 @@ Here's a simple test database I created using SQL. ![](../../.gitbook/assets/getting-started/doltlab-admin-db.png) -## Clone Databases +### Clone Databases DoltLab is a remote so you can clone databases from it. The remote API is exposed on port 50051. I made my test database public so I don't need any permissions to clone it. @@ -236,7 +236,7 @@ $ dolt sql -q "select * from t" I now have the test database I created on my DoltLab locally. -## Push Databases +### Push Databases To push databases you need your Dolt client authenticated against your DoltLab instance. You can run a fresh `dolt login` against DoltLab using something like this: @@ -279,26 +279,26 @@ You're now ready to try out all the Dolt and DoltLab experiences like Pull Reque ## Adding Additional Functionality -## Create new users +### Create new users As discussed in [the single user section](#single-user), creating users in DoltLab <= `v2.1.6` requires a working SMTP server to send emails. However, DoltLab >= `v2.2.0` does not require an SMTP server connection in order for new users to create accounts on your DoltLab instance. In this newer version, an SMTP server connection can only be made with DoltLab Enterprise. For more information, consult [this guide to set up an email server for DoltLab](/guides/enterprise#set-up-a-smtp-server-using-any-gmail-address). -## Receive Email Notifications +### Receive Email Notifications DoltLab Enterprise sends emails for password resets, pull request and issue status, and a few other use cases. Obviously, these also won't work without a running email server. Consult [this guide to set up an email server for DoltLab](/guides/enterprise#set-up-a-smtp-server-using-any-gmail-address). -## HTTPS +### HTTPS Your DoltLab is currently set up to only use HTTP which is fairly insecure. If it's running on your internal network and you have other threat mitigations, this may be OK. But having it sit on the public internet on AWS without HTTPS is probably not what you want. Native HTTPS support is available in DoltLab Enterprise. [Learn how to set up HTTPS on your DoltLab here](/guides/enterprise#serve-doltlab-over-https-natively). -## Custom URL +### Custom URL Right now, you're stuck hitting the IP address of your host. In order to use a custom URL on the internet, you need a static IP for your EC2 host and a domain name. If your host is running on an internal network, you may need to follow a different process. -## Custom Logo/Colors +### Custom Logo/Colors Yay! Our first DoltLab Enterprise feature. If you would like a custom logo and color scheme for your DoltLab instance, you are going to need [DoltLab Enterprise](/guides/enterprise). DoltLab Enterprise is $5,000/month for unlimited users. It comes with Enterprise Support for Dolt as well. Among [other features](/guides/enterprise), DoltLab Enterprise allows you to customize the look and feel of your DoltLab instance. Learn how to [configure your DoltLab as Enterprise](/guides/enterprise) and [set up your custom look and feel here](/guides/enterprise#use-custom-logo). -## Scalability +### Scalability Lastly, your DoltLab is running on a single host. All the components will scale to the size of that host including storing all the databases that are created. Storage and compute requirements can get big quickly. In order to break your DoltLab up into multiple instances and use cloud storage to store your databases, you need [DoltLab Enterprise](/guides/enterprise). Learn [how to set up Enterprise here](/guides/enterprise). diff --git a/site/doltlab/src/content/older/pre-installer-administrator-guide.md b/site/doltlab/src/content/older/pre-installer-administrator-guide.md index 0db1df3f..2ee59ecc 100644 --- a/site/doltlab/src/content/older/pre-installer-administrator-guide.md +++ b/site/doltlab/src/content/older/pre-installer-administrator-guide.md @@ -29,7 +29,7 @@ DoltLab persists all data to local disk using Docker volumes. To backup or resto DoltLab <= `v0.8.4` uses PostgreSQL as its database and DoltLab `v1.0.0`+ uses Dolt. To backup the PostgreSQL server we recommend dumping the database with `pg_dump` and restoring the database from the dump using `psql`. To backup the Dolt server we recommend using Docker's volume backup and restore process, or Dolt's built-in backup and restore features. -## Backing up and restoring remote data, user uploaded data, and Dolt server data with Docker +### Backing up and restoring remote data, user uploaded data, and Dolt server data with Docker To backup DoltLab's remote data, the database data for all database on a given DoltLab instance, leave DoltLab's services up and run: @@ -141,7 +141,7 @@ docker run --rm --volumes-from doltlab_doltlabdb_1 -v $(pwd):/backup ubuntu bash You can now restart DoltLab, and should see all data restored from the `tar` files. -## Backing up and restoring PostgreSQL data +### Backing up and restoring PostgreSQL data For DoltLab versions <= `v0.8.4`, to backup data from DoltLab's postgres server, we recommend executing a data dump with `pg_dump`. To do so, keep DoltLab's services up and run: @@ -197,7 +197,7 @@ SET session_replication_role = replica; docker run --rm --network doltlab_doltlab -e PGPASSWORD= -v $(pwd):/doltlab-db-dumps postgres:13-bullseye bash -c "psql --host=doltlab_doltlabdb_1 --port=5432 --username=dolthubadmin dolthubapi < /doltlab-db-dumps/postgres-dump.sql" ``` -## Backing up and restoring the Dolt server with `dolt backup` +### Backing up and restoring the Dolt server with `dolt backup` DoltLab `v1.0.0`+ uses Dolt as its database. To back up the Dolt server of DoltLab using Dolt's built-in backup and restore features, keep DoltLab's services up and open a connection to the database. The quickest way to do this is with the `./shell-db.sh` script included with DoltLab: @@ -592,7 +592,7 @@ Starting with DoltLab `v1.0.0`, DoltLab can be configured to use a [Hosted Dolt] To configure a DoltLab to use a Hosted Dolt, follow the steps below as we create a sample DoltLab Hosted Dolt instance called `my-doltlab-db-1`. -## Create a Hosted Dolt deployment +### Create a Hosted Dolt deployment To begin, you'll need to create a Hosted Dolt deployment that your DoltLab instance will connect to. We've created a [video tutorial](https://www.dolthub.com/blog/2022-05-20-hosted-dolt-howto/) for how to create your first Hosted Dolt deployment, but briefly, you'll need to create an account on [hosted.doltdb.com](https://hosted.doltdb.com) and then click the "Create Deployment" button. @@ -639,7 +639,7 @@ You can do this by running these statements from the Hosted workbench SQL consol This instance is now ready for a DoltLab connection. -## Edit DoltLab's Docker Compose file +### Edit DoltLab's Docker Compose file To connect DoltLab to `my-doltlab-db-1`, ensure that your DoltLab instance is stopped, and remove references to `doltlabdb` in DoltLab's `docker-compose.yaml` file.