Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Every setting is an environment variable, and [.env.example](.env.example) lists
| `DB_HOST`, `DB_NAME`, `DB_USERNAME`, `DB_PASSWORD` | required | The PostgreSQL connection. `DB_PORT` is 5432 unless it is set |
| `DB_SSL_MODE` | `require`, and `disable` in the example | The mode of the connection. Use `disable` only on a private Docker network or on your own machine |
| `DB_MAX_CONNS` | `8` | The pool size of one copy |
| `SMTP_HOST`, `SMTP_PORT`, `SMTP_SECURITY`, `SMTP_FROM_ADDRESS` | see the file | The mail server for the verification and reset mails |
| `SMTP_HOST`, `SMTP_PORT`, `SMTP_SECURITY`, `SMTP_FROM_ADDRESS` | see the file | The mail server for the verification, reset and account deleted mails |
| `SENTRY_DSN` | empty | An empty value switches error reports off |
| `METRICS_ADDRESS` | empty | The address of the Prometheus listener. Empty switches it off |

Expand Down
2 changes: 1 addition & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The contract is `openapi.yaml` in the root of the repository. The job `OpenAPI l

## Deleting the account

`DELETE /v1/account` takes `{"password": "..."}` and deletes the business with everything under it, then clears the cookie and answers 204. The password is checked again so that a stolen session cannot do it alone, a wrong one answers 403 `invalid_credentials` and not 401, and a user has 5 wrong passwords an hour, shared with `DELETE /v1/gateways/{id}`. Only a wrong password uses up a try, so a call that ends in 503 `busy` or with the right password does not. After the fifth wrong one the route answers 429 `rate_limited` with `Retry-After` for every call, also with the right password. The budget is read again after the password check, so a burst of parallel calls cannot get a right password through once the budget is spent. The hubs of the business are cut off first, so they stop sending and cannot add readings during the delete, and a call that stops half way leaves them cut. Then the readings, the hourly sums and the anomalies go in batches, and the business row goes last. A 503 `busy` can therefore come after a part of the readings was deleted, or after the whole delete when the time ran out just after the commit. The same call can be repeated until it ends with 204. A 401 on the repeat means that the session is gone, which a finished delete causes, and so does a session that ended in the meantime. The app cannot tell the two apart from the answer, so it sends the user to the sign in page. A sign in with the right password works only while the account still exists, and a failed sign in proves nothing, because a wrong password gives the same answer. It cannot be undone.
`DELETE /v1/account` takes `{"password": "..."}` and deletes the business with everything under it, then clears the cookie and answers 204. Once the delete call returns without an error, the service tries to queue one short email to the address of the owner that says the account is gone. Only the call that deleted the business sends it, so a double click sends one. A 503 `busy` that comes after the commit sends none, and a full mail queue or a shutdown drops the email without a new try. None of this changes the 204. The password is checked again so that a stolen session cannot do it alone, a wrong one answers 403 `invalid_credentials` and not 401, and a user has 5 wrong passwords an hour, shared with `DELETE /v1/gateways/{id}`. Only a wrong password uses up a try, so a call that ends in 503 `busy` or with the right password does not. After the fifth wrong one the route answers 429 `rate_limited` with `Retry-After` for every call, also with the right password. The budget is read again after the password check, so a burst of parallel calls cannot get a right password through once the budget is spent. The hubs of the business are cut off first, so they stop sending and cannot add readings during the delete, and a call that stops half way leaves them cut. Then the readings, the hourly sums and the anomalies go in batches, and the business row goes last. A 503 `busy` can therefore come after a part of the readings was deleted, or after the whole delete when the time ran out just after the commit. The same call can be repeated until it ends with 204. A 401 on the repeat means that the session is gone, which a finished delete causes, and so does a session that ended in the meantime. The app cannot tell the two apart from the answer, so it sends the user to the sign in page. A sign in with the right password works only while the account still exists, and a failed sign in proves nothing, because a wrong password gives the same answer. It cannot be undone.

## Errors

Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The backend is one Go program that serves a JSON API for the dashboard under `/v
| API server | A Fiber v3 HTTP server that handles routing, security headers and errors |
| PostgreSQL | The only store. It holds accounts, sessions, readings, tips and every rate limit counter, and the program talks to it through pgx |
| Migrations | SQL files that are embedded in the binary and applied when the program starts, see [Data model](data-model.md) |
| Mail | Sends the verification and reset mails over SMTP. A request never waits for the mail server |
| Mail | Sends the verification, reset and account deleted mails over SMTP. A request never waits for the mail server |
| Metrics listener | An optional second listener that serves the Prometheus text format, see [Monitoring](monitoring.md) |
| Error reports | Optional reports of a 5xx and a panic to Sentry, with a scrubbed text |

Expand Down
4 changes: 2 additions & 2 deletions docs/privacy.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The service never stores a password in clear text. A request body is not written

| Where | What | When |
|:---|:---|:---|
| The SMTP server that the operator configures | The email address and the link of a verification or reset mail | Every mail the service sends. When `SMTP_HOST` is empty, which production refuses, the service writes the address and the link to its log instead |
| The SMTP server that the operator configures | The email address and the link of a verification or reset mail, and the address alone for the one mail that says an account was deleted | Every mail the service sends. When `SMTP_HOST` is empty, which production refuses, the service writes the address and the link to its log instead |
| Sentry | The error text, the stack, the route pattern, the status, the request id, the id of the account, the release, the environment and the server name. The text is scrubbed of emails, tokens, addresses and long numbers, but the scrubbing is best effort, so a name that sits in a database error text can pass | Only when `SENTRY_DSN` is set, and only for a failure of the service |
| Grafana Cloud | The metrics, and the logs of the API containers. A log line of a request has the method, the path without its query, the status, the time it took, the client address and the error code. A route with an id in its path is logged by its pattern, such as `/v1/devices/:id`, so the id is not in the log. A line about a failed route has the type of the error and not its text | Only when the operator turns on the monitoring profile, see [Monitoring](monitoring.md). The operator sets how long Grafana Cloud keeps the logs |
| The container log files of the host | The same log lines | Always. The production example keeps up to 5 files of 10 MB for each container by default (`LOG_MAX_SIZE` and `LOG_MAX_FILES`), cut by size and not by age, and Docker removes them when a container is recreated |
Expand All @@ -52,7 +52,7 @@ There is no analytics script and no advertising. The service sets one cookie, th

## Deleting an account

The owner deletes the account in the app with `DELETE /v1/account`, which asks for the password again. The route first cuts off the hubs of the business, so they stop sending, and then deletes the readings, the hourly sums and the anomalies in batches that commit one by one. Then it deletes the business row, and every other table under the business goes with it in that one statement, because the keys cascade, see [Data model](data-model.md). That covers every table under the business, for example the user, the sessions, the gateways, the devices, the tips and the relay data. A request that stops half way leaves the account in place with part of its readings gone, and the next call goes on. It cannot be undone. The operator can still delete a business row by hand for an owner who cannot sign in. The logs, the backups, the reverse proxy of the host, Sentry, Grafana Cloud and the mail provider keep their own copies and follow their own retention.
The owner deletes the account in the app with `DELETE /v1/account`, which asks for the password again. The route first cuts off the hubs of the business, so they stop sending, and then deletes the readings, the hourly sums and the anomalies in batches that commit one by one. Then it deletes the business row, and every other table under the business goes with it in that one statement, because the keys cascade, see [Data model](data-model.md). That covers every table under the business, for example the user, the sessions, the gateways, the devices, the tips and the relay data. A request that stops half way leaves the account in place with part of its readings gone, and the next call goes on. It cannot be undone. When the delete call returns without an error, the service tries to queue one short mail to the address of the owner that says the account is gone. It keeps the address in memory for that mail only, until the mail is sent or dropped. A failed send is tried twice more, and a mail that the queue drops is not queued a second time. The operator can still delete a business row by hand for an owner who cannot sign in. The logs, the backups, the reverse proxy of the host, Sentry, Grafana Cloud and the mail provider keep their own copies and follow their own retention.

The owner can also remove one gateway with `DELETE /v1/gateways/{id}`, which asks for the password again in the same way. It cuts the key first, deletes the readings, the hourly sums and the anomalies of that gateway in batches, and then deletes the gateway row, and its nodes, devices, tips, relay commands and schedules go with it because the keys cascade. A request that stops half way leaves a gateway with a cut key and fewer rows, and the next call goes on. It cannot be undone. The copies that the paragraph above names keep their own retention.

Expand Down
2 changes: 1 addition & 1 deletion docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The `Tests` job of [ci.yaml](../.github/workflows/ci.yaml) builds the program an

A route test needs PostgreSQL. `internal/database/dbtest` gives each test its own empty database with every migration applied, and drops it when the test ends. The server comes from the variables `TEST_DB_HOST`, `TEST_DB_PORT`, `TEST_DB_USER` and `TEST_DB_PASSWORD`. Without `TEST_DB_HOST` a test on a machine skips itself, so a plain `go test` still passes. A run with the variable `CI` set fails instead, so CI can never pass with every database test skipped.

`internal/testutil` builds the app in process, signs up an owner, seeds a gateway, a plug and a session, and sends requests through the app with `app.Test`. It starts no network port, and the tests send no real mail. The tests replace the mailer with an outbox that keeps the verification and reset links, and a test reads the link from it when it needs one.
`internal/testutil` builds the app in process, signs up an owner, seeds a gateway, a plug and a session, and sends requests through the app with `app.Test`. It starts no network port, and the tests send no real mail. The tests replace the mailer with an outbox that keeps the verification and reset links and counts the account deleted mails, and a test reads the link or the count from it when it needs one.

To run them on a machine, point the variables at a throwaway PostgreSQL. A killed run leaves its database behind, so never use a server that holds real data.

Expand Down
35 changes: 35 additions & 0 deletions internal/auth/auth_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1149,6 +1149,41 @@ func TestOnlyAWrongPasswordUsesUpATryToDeleteAnAccount(t *testing.T) {
}
}

func TestDeletingTheAccountSendsOneEmailAndAWrongPasswordSendsNone(t *testing.T) {
e := testutil.New(t)
session := e.SignUp(t, "budi@example.com")

if r := deleteAccount(t, e, `{"password":"salah"}`, session); r.Status != http.StatusForbidden {
t.Fatalf("a wrong password answered %d", r.Status)
}

if n := e.Mail.DeletedEmails("budi@example.com"); n != 0 {
t.Fatalf("a refused delete sent %d emails", n)
}

if r := deleteAccount(t, e, `{"password":"kata sandi baik"}`, session); r.Status != http.StatusNoContent {
t.Fatalf("the right password answered %d %v", r.Status, r.Body)
}

if n := e.Mail.DeletedEmails("budi@example.com"); n != 1 {
t.Fatalf("a delete sent %d emails, want 1", n)
}
}

func TestOnlyTheFirstDeleteOfABusinessReportsThatItDeletedTheRow(t *testing.T) {
e := testutil.New(t)
budi := e.SeedDevice(t, "budi@example.com", "ESP32-AABBCC")
store := auth.NewStore(e.Pool)

if deleted, err := store.DeleteBusiness(t.Context(), budi.Business); err != nil || !deleted {
t.Fatalf("the first delete answered %v %v, want true", deleted, err)
}

if deleted, err := store.DeleteBusiness(t.Context(), budi.Business); err != nil || deleted {
t.Fatalf("the second delete answered %v %v, want false", deleted, err)
}
}

func TestCuttingOffTheGatewaysOfABusinessStopsOnlyItsOwnHubs(t *testing.T) {
e := testutil.New(t)
budi := e.SeedDevice(t, "budi@example.com", "ESP32-AABBCC")
Expand Down
14 changes: 13 additions & 1 deletion internal/auth/service.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ const (
type Mailer interface {
Verification(to, link string)
PasswordReset(to, link string)
AccountDeleted(to string)
}

// The limits of the account routes. A key is an address of a client or an email.
Expand Down Expand Up @@ -356,7 +357,18 @@ func (s *Service) DeleteAccount(ctx context.Context, user *User, password string
return err
}

return s.store.DeleteBusiness(ctx, user.BusinessID)
deleted, err := s.store.DeleteBusiness(ctx, user.BusinessID)
if err != nil {
return err
}

// only the call that deleted the row sends the email, so a double click sends one. The address lives only in memory for this email,
// and a mail that cannot be queued does not undo a delete that is done.
if deleted {
s.mailer.AccountDeleted(user.Email)
}

return nil
}

// ConfirmPassword asks the user for the password again before an action that cannot be undone, so a stolen session cannot do it alone.
Expand Down
14 changes: 8 additions & 6 deletions internal/auth/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -490,16 +490,17 @@ func (s *Store) CutOffGateways(ctx context.Context, businessID string) error {
// of readings, and one statement for all of them could run past the deadline of the request every time, so the big tables go first in
// batches that each commit by themselves, as the janitor does. The hubs are cut off first. A call that stops half way has cut the hubs and
// deleted a part of the readings and nothing else, and the next call goes on from there. The business row goes last, in one statement.
func (s *Store) DeleteBusiness(ctx context.Context, businessID string) error {
// It reports whether this call deleted the row, which is false when another call got there first.
func (s *Store) DeleteBusiness(ctx context.Context, businessID string) (bool, error) {
if err := s.CutOffGateways(ctx, businessID); err != nil {
return err
return false, err
}

for _, query := range bigTablesOfABusiness {
for {
tag, err := s.pool.Exec(ctx, query, businessID, deleteBatch)
if err != nil {
return fmt.Errorf("delete a batch of rows of the business %w", err)
return false, fmt.Errorf("delete a batch of rows of the business %w", err)
}

// a batch can come back short when a concurrent writer took rows of it, so only an empty batch ends the loop
Expand All @@ -510,11 +511,12 @@ func (s *Store) DeleteBusiness(ctx context.Context, businessID string) error {
}
}

if _, err := s.pool.Exec(ctx, "DELETE FROM businesses WHERE id = $1", businessID); err != nil {
return fmt.Errorf("delete the business %w", err)
tag, err := s.pool.Exec(ctx, "DELETE FROM businesses WHERE id = $1", businessID)
if err != nil {
return false, fmt.Errorf("delete the business %w", err)
}

return nil
return tag.RowsAffected() == 1, nil
}

// DeleteSession ends one session
Expand Down
28 changes: 28 additions & 0 deletions internal/mail/mail_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -172,3 +172,31 @@ func TestAPanicOfTheSenderIsContainedAndTheMessageIsTriedAgain(t *testing.T) {

q.Close(ctx)
}

func TestTheEmailOfADeletedAccountHasNoLinkAndGoesToTheAddress(t *testing.T) {
f := &flaky{sent: make(chan Message, 1)}
q := NewQueue(f, 1, 4)

NewOutbox(q).AccountDeleted("budi@example.com")

var m Message

select {
case m = <-f.sent:
case <-time.After(5 * time.Second):
t.Fatal("the email of a deleted account was not sent")
}

if m.To != "budi@example.com" || m.Subject == "" || m.Text == "" || m.HTML == "" {
t.Fatalf("the message is %+v", m)
}

if strings.Contains(m.Text, "http") || strings.Contains(m.HTML, "http") || strings.Contains(m.HTML, "<a ") {
t.Fatalf("the message of a deleted account holds a link %+v", m)
}

ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()

q.Close(ctx)
}
17 changes: 17 additions & 0 deletions internal/mail/outbox.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ var resetHTML = template.Must(template.New("reset").Parse(`<p>Ketuk tautan ini u
<p>Tautan ini berlaku 1 jam. Jika Anda tidak memintanya, abaikan email ini dan kata sandi Anda tetap sama.</p>
`))

const deletedHTML = `<p>Akun Aethera Anda dan semua data bisnis Anda sudah dihapus. Penghapusan ini tidak bisa dibatalkan.</p>
<p>Jika bukan Anda yang melakukannya, segera hubungi pengelola Aethera.</p>
`

// Verification queues the email with the link that verifies an address. The plain text part holds the plain URL.
func (o *Outbox) Verification(to, link string) {
o.send(Message{
Expand All @@ -46,6 +50,16 @@ func (o *Outbox) PasswordReset(to, link string) {
})
}

// AccountDeleted queues the short email that says the account is gone, so an owner who did not do it hears about it
func (o *Outbox) AccountDeleted(to string) {
o.send(Message{
To: to,
Subject: "Akun Aethera Anda sudah dihapus",
Text: "Akun Aethera Anda dan semua data bisnis Anda sudah dihapus. Penghapusan ini tidak bisa dibatalkan.\r\n\r\nJika bukan Anda yang melakukannya, segera hubungi pengelola Aethera.\r\n",
HTML: deletedHTML,
})
}

func (o *Outbox) send(m Message) {
if !o.queue.Enqueue(m) {
slog.Warn("an email was not queued")
Expand All @@ -71,3 +85,6 @@ func (Log) Verification(to, link string) { slog.Info("verification email", "to",

// PasswordReset logs the link
func (Log) PasswordReset(to, link string) { slog.Info("password reset email", "to", to, "link", link) }

// AccountDeleted logs that the account is gone
func (Log) AccountDeleted(to string) { slog.Info("account deleted email", "to", to) }
19 changes: 18 additions & 1 deletion internal/testutil/testutil.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ type Outbox struct {
mu sync.Mutex
verify map[string]string
reset map[string]string
gone map[string]int
}

// Verification keeps the link of a verification email
Expand All @@ -54,6 +55,22 @@ func (o *Outbox) PasswordReset(to, link string) {
o.reset[to] = link
}

// AccountDeleted counts the emails that say an account is gone
func (o *Outbox) AccountDeleted(to string) {
o.mu.Lock()
defer o.mu.Unlock()

o.gone[to]++
}

// DeletedEmails returns how many emails told the address that the account is gone
func (o *Outbox) DeletedEmails(to string) int {
o.mu.Lock()
defer o.mu.Unlock()

return o.gone[to]
}

// VerifyLink returns the last link of a verification email to the address, and whether there is one
func (o *Outbox) VerifyLink(to string) (string, bool) {
o.mu.Lock()
Expand Down Expand Up @@ -108,7 +125,7 @@ func New(t *testing.T) *Env {
func NewOn(t *testing.T, pool *pgxpool.Pool) *Env {
t.Helper()

mail := &Outbox{verify: map[string]string{}, reset: map[string]string{}}
mail := &Outbox{verify: map[string]string{}, reset: map[string]string{}, gone: map[string]int{}}

cfg := config.Config{AppEnv: config.EnvProduction, AppOrigin: Origin}

Expand Down
Loading
Loading