diff --git a/docs/site/Testing-your-application.md b/docs/site/Testing-your-application.md index 6055d3dcc817..0e02574f4e13 100644 --- a/docs/site/Testing-your-application.md +++ b/docs/site/Testing-your-application.md @@ -1,1020 +1,1053 @@ ---- -lang: en -title: 'Testing your application' -keywords: - LoopBack 4.0, LoopBack 4, Node.js, TypeScript, OpenAPI, Node.js, TypeScript, - OpenAPI -sidebar: lb4_sidebar -permalink: /doc/en/lb4/Testing-your-application.html ---- - -## Overview - -A thorough automated test suite is important because it: - -- Ensures your application works as expected. -- Prevents regressions when new features are added and bugs are fixed. -- Helps new and existing developers understand different parts of the codebase - (knowledge sharing). -- Speeds up development over the long run (the code writes itself!). - -### Types of tests - -We encourage writing tests from a few perspectives, mainly -[black-box testing](https://en.wikipedia.org/wiki/Black-box_testing) -(acceptance) and -[white-box testing](https://en.wikipedia.org/wiki/White-box_testing) -(integration and unit). Tests are usually written using typical patterns such as -[`arrange/act/assert`](https://msdn.microsoft.com/en-us/library/hh694602.aspx#Anchor_3) -or [`given/when/then`](https://martinfowler.com/bliki/GivenWhenThen.html). Both -styles work well, so pick one that you're comfortable with and start writing -tests! - -For an introduction to automated testing, see -[Define your testing strategy](Defining-your-testing-strategy.md). - -{% include important.html content=" A great test suite requires you to think -smaller and favor fast and focused unit tests over slow end-to-end tests. -" %} - -This article is a reference guide for common types of tests and test helpers. - -## Project setup - -An automated test suite requires a test runner to execute all the tests and -produce a summary report. We use and recommend [Mocha](https://mochajs.org). - -In addition to a test runner, the test suites generally require: - -- An assertion library (we recommend [Should.js](https://shouldjs.github.io)). -- A library for making HTTP calls and verifying their results (we recommend - [supertest](https://github.com/visionmedia/supertest)). -- A library for creating test doubles (we recommend - [Sinon.JS](http://sinonjs.org/)). - -The [@loopback/testlab](https://www.npmjs.com/package/@loopback/testlab) module -integrates these packages and makes them easy to use together with LoopBack. - -### Set up testing infrastructure with LoopBack CLI - -LoopBack applications that have been generated using the `lb4 app` command from -`@loopback/cli` come with `@loopback/testlab` and `mocha` as a default, so no -other testing infrastructure setup is needed. - -### Setup testing infrastructure manually - -If you have an existing application install `mocha` and `@loopback/testlab`: - -```shell -npm install --save-dev mocha @loopback/testlab -``` - -Your `package.json` should then look something like this: - -```js -{ - // ... - "devDependencies": { - "@loopback/testlab": "^", - "@types/mocha": "^", - "mocha": "^" - }, - "scripts": { - "test": "mocha --recursive \"dist/test\"" - } - // ... -} -``` - -## Data handling - -Tests accessing a real database often require existing data. For example, a -method listing all products needs some products in the database; a method to -create a new product instance must determine which properties are required and -any restrictions on their values. There are various approaches to address this -issue. Many of them unfortunately make the test suite difficult to understand, -difficult to maintain, and prone to test failures unrelated to the changes made. - -Our approach to data handling, based on our experience, is described in this -section. - -### Create a test datasource - -As we would prefer full control of the database and even the development -database is not something we want to clean before each test it's handy to use an -independent in-memory datasource which is filled appropriately using -[test data builders](#use-test-data-builders) before each test run. - -{% include code-caption.html content="src/__tests__/fixtures/datasources/testdb.datasource.ts" %} - -```ts -import {juggler} from '@loopback/repository'; - -export const testdb: juggler.DataSource = new juggler.DataSource({ - name: 'db', - connector: 'memory', -}); -``` - -{% include important.html content="The `name` property of your test datasource must match your real datasource's name." %} - -### Clean the database before each test - -Start with a clean database before each test. This may seem counter-intuitive: -why not reset the database after the test has finished? When a test fails and -the database is cleaned after the test has finished, then it's difficult to -observe what was stored in the database and why the test failed. When the -database is cleaned in the beginning, then any failing test will leave the -database in the state that caused the test to fail. - -To clean the database before each test, set up a `beforeEach` hook to call a -helper method; for example: - -{% include code-caption.html content="src/__tests__/helpers/database.helpers.ts" %} - -```ts -import {ProductRepository, CategoryRepository} from '../../repositories'; -import {testdb} from '../fixtures/datasources/testdb.datasource'; - -export async function givenEmptyDatabase() { - await new ProductRepository(testdb).deleteAll(); - await new CategoryRepository(testdb).deleteAll(); -} -``` - -In case a repository includes a relation to another repository, ie. Product -belongs to Category, include it in the repository call, for example: - -{% include code-caption.html content="src/__tests__/helpers/database.helpers.ts" %} - -```ts -import {Getter} from '@loopback/core'; -import {ProductRepository, CategoryRepository} from '../../repositories'; -import {testdb} from '../fixtures/datasources/testdb.datasource'; - -export async function givenEmptyDatabase() { - let categoryRepository: CategoryRepository; - let productRepository: ProductRepository; - - categoryRepository = new CategoryRepository( - testdb, - async () => productRepository, - ); - - productRepository = new ProductRepository( - testdb, - async () => categoryRepository, - ); - - await productRepository.deleteAll(); - await categoryRepository.deleteAll(); -} -``` - -{% include code-caption.html content="src/__tests__/integration/controllers/product.controller.integration.ts" %} - -```ts -// in your test file -import {givenEmptyDatabase} from '../../helpers/database.helpers'; - -describe('ProductController (integration)', () => { - before(givenEmptyDatabase); - // etc. -}); -``` - -### Use test data builders - -To avoid duplicating code for creating model data that is complete with required -properties, use shared -[test data builders](http://www.natpryce.com/articles/000714.html). This enables -tests to provide the small subset of properties that is strictly required by the -tested scenario. Using shared test builders will help your tests to be: - -- Easier to understand, since it's immediately clear what model properties are - relevant to the tests. If the tests set the required properties, it is - difficult to tell whether the properties are actually relevant to the tested - scenario. - -- Easier to maintain. As your data model evolves, you will need to add more - required properties. If the tests build the model instance data manually, all - the tests must be manually updated to set a new required property. With a - shared test data builder, you update a single location with the new property. - -See -[@loopback/openapi-spec-builder](https://www.npmjs.com/package/@loopback/openapi-spec-builder) -for an example of how to apply this design pattern for building OpenAPI Spec -documents. - -In practice, a simple function that adds missing required properties is -sufficient. - -{% include code-caption.html content="src/__tests__/helpers/database.helpers.ts" %} - -```ts -// ... -export function givenProductData(data?: Partial) { - return Object.assign( - { - name: 'a-product-name', - slug: 'a-product-slug', - price: 1, - description: 'a-product-description', - available: true, - }, - data, - ); -} - -export async function givenProduct(data?: Partial) { - return new ProductRepository(testdb).create(givenProductData(data)); -} -// ... -``` - -### Avoid sharing the same data for multiple tests - -It's tempting to define a small set of data to be shared by all tests. For -example, in an e-commerce application, you might pre-populate the database with -a few categories, some products, an admin user and a customer. This approach has -several downsides: - -- When trying to understand any individual test, it's difficult to tell what - part of the pre-populated data is essential for the test and what's - irrelevant. For example, in a test checking the method counting the number of - products in a category using a pre-populated category "Stationery", is it - important that "Stationery" contains nested sub-categories or is that fact - irrelevant? If it's irrelevant, then what are the other tests that depend on - it? - -- As the application grows and new features are added, it's easier to add more - properties to existing model instances rather than create new instances using - only the properties required by the new features. For example, when adding a - category image, it's easier to add image to an existing category "Stationery" - and perhaps keep another category "Groceries" without any image, rather than - creating two new categories "CategoryWithAnImage" and "CategoryMissingImage". - This further amplifies the previous problem, because it's not clear that - "Groceries" is the category that should be used by tests requiring a category - with no image - the category name does not provide any hints on that. - -- As the shared dataset grows (together with the application), the time required - to bring the database into its initial state grows too. Instead of running a - few "DELETE ALL" queries before each test (which is relatively fast), you may - have to run tens or hundreds of different commands used to create different - model instances, thus triggering slow index rebuilds along the way and slowing - down the test suite considerably. - -Use the test data builders described in the previous section to populate your -database with the data specific to your test only. - - - - - -Write higher-level helpers to share the code for re-creating common scenarios. -For example, if your application has two kinds of users (admins and customers), -then you may write the following helpers to simplify writing acceptance tests -checking access control: - -```ts -async function givenAdminAndCustomer() { - return { - admin: await givenUser({role: Roles.ADMIN}), - customer: await givenUser({role: Roles.CUSTOMER}), - }; -} -``` - -## Unit testing - -Unit tests are considered "white-box" tests because they use an "inside-out" -approach where the tests know about the internals and control all the variables -of the system being tested. Individual units are tested in isolation and their -dependencies are replaced with -[Test doubles](https://en.wikipedia.org/wiki/Test_double). - -### Use test doubles - -Test doubles are functions or objects that look and behave like the real -variants used in production, but are actually simplified versions that give the -test more control of the behavior. For example, reproducing the situation where -reading from a file failed because of a hard-drive error is pretty much -impossible. However, using a test double to simulate the file-system API will -provide control over what each call returns. - -[Sinon.JS](http://sinonjs.org/) has become the de-facto standard for test -doubles in Node.js and JavaScript/TypeScript in general. The `@loopback/testlab` -package comes with Sinon preconfigured with TypeScript type definitions and -integrated with Should.js assertions. - -There are three kinds of test doubles provided by Sinon.JS: - -- [Test spies](http://sinonjs.org/releases/v4.0.1/spies/) are functions that - record arguments, the return value, the value of `this`, and exceptions thrown - (if any) for all its calls. There are two types of spies: Some are anonymous - functions, while others wrap methods that already exist in the system under - test. - -- [Test stubs](http://sinonjs.org/releases/v4.0.1/stubs/) are functions (spies) - with pre-programmed behavior. As spies, stubs can be either anonymous, or wrap - existing functions. When wrapping an existing function with a stub, the - original function is not called. - -- [Test mocks](http://sinonjs.org/releases/v4.0.1/mocks/) (and mock - expectations) are fake methods (like spies) with pre-programmed behavior (like - stubs) as well as pre-programmed expectations. A mock will fail your test if - it is not used as expected. - -{% include note.html content=" -We recommend against using test mocks. With test -mocks, the expectations must be defined before the tested scenario is executed, -which breaks the recommended test layout 'arrange-act-assert' (or -'given-when-then') and also produces code that's difficult to comprehend. -" %} - -#### Create a stub Repository - -When writing an application that accesses data in a database, the best practice -is to use [repositories](Repository.md) to encapsulate all -data-access/persistence-related code. Other parts of the application (typically -[controllers](Controller.md)) can then depend on these repositories for data -access. To test Repository dependents (for example, Controllers) in isolation, -we need to provide a test double, usually as a test stub. - -In traditional object-oriented languages like Java or C#, to enable unit tests -to provide a custom implementation of the repository API, the controller needs -to depend on an interface describing the API, and the repository implementation -needs to implement this interface. The situation is easier in JavaScript and -TypeScript. Thanks to the dynamic nature of the language, it’s possible to -mock/stub entire classes. - -Creating a test double for a repository class is very easy using the Sinon.JS -utility function `createStubInstance`. It's important to create a new stub -instance for each unit test in order to prevent unintended re-use of -pre-programmed behavior between (unrelated) tests. - -```ts -describe('ProductController', () => { - let repository: ProductRepository; - beforeEach(givenStubbedRepository); - - // your unit tests - - function givenStubbedRepository() { - repository = sinon.createStubInstance(ProductRepository); - } -}); -``` - -In your unit tests, you will usually want to program the behavior of stubbed -methods (what they should return) and then verify that the Controller (unit -under test) called the right method with the correct arguments. - -Configure stub's behavior at the beginning of your unit test (in the "arrange" -or "given" section): - -```ts -// repository.find() will return a promise that -// will be resolved with the provided array -const findStub = repository.find as sinon.SinonStub; -findStub.resolves([{id: 1, name: 'Pen'}]); -``` - -Verify how was the stubbed method executed at the end of your unit test (in the -"assert" or "then" section): - -```ts -// expect that repository.find() was called with the first -// argument deeply-equal to the provided object -sinon.assert.calledWithMatch(findStub, {where: {id: 1}}); -``` - -See [Unit test your controllers](#unit-test-your-controllers) for a full -example. - -#### Create a stub Service - -If your controller relies on service proxy for service oriented backends such as -REST APIs, SOAP Web Services, or gRPC microservices, then we can create stubs -for the service akin to the steps outlined in the above -[stub repository](#create-a-stub-repository) section. Consider a dependency on a -`GeoCoder` service that relies on a remote REST API for returning coordinates -for a specific address. - -```ts -export interface GeocoderService { - geocode(address: string): Promise; -} -``` - -The first step is to create a mocked instance of the `GeocoderService` API and -configure its `geocode` method as a Sinon stub: - -```ts -describe('GeocoderController', () => { - let geoService: GeoCoderService; - let geocode: sinon.SinonStub; - - beforeEach(givenMockGeoCoderService); - - // your unit tests - - function givenMockGeoCoderService() { - // this creates a stub with GeocoderService API - // in a way that allows the compiler to verify type correctness - geoService = {geocode: sinon.stub()}; - - // this creates a reference to the stubbed "geocode" method - // because "geoService.geocode" has type from GeocoderService - // and does not provide Sinon APIs - geocode = geoService.geocode as sinon.SinonStub; - } -}); -``` - -Afterwards, we can configure the `geocode` stub's behaviour before the `act` -phase of our test(s): - -```ts -// geoService.geocode() will return a promise that -// will be resolved with the provided array -geocode.resolves([{y: 41.109653, x: -73.72467}]); -``` - -Lastly, we'll verify how the stub was executed: - -```ts -// expect that geoService.geocode() was called with the first -// argument equal to the provided address string -sinon.assert.calledWithMatch(geocode, '1 New Orchard Road, Armonk, 10504'); -``` - -Check out -[TodoController unit tests](https://github.com/loopbackio/loopback-next/blob/bd0c45033503f631a533ad6176620354d9cf6768/examples/todo/src/__tests__/unit/controllers/todo.controller.unit.ts#L53-L71) -illustrating the above points in action for more information. - -### Unit test your Controllers - -Unit tests should apply to the smallest piece of code possible to ensure that -other variables and state changes do not pollute the result. A typical unit test -creates a controller instance with dependencies replaced by test doubles and -directly calls the tested method. The example below gives the controller a stub -implementation of its repository dependency using the `testlab` -`createStubInstance` function, ensures the controller calls the repository's -`find()` method with a correct query, and returns back the query results. See -[Create a stub repository](#create-a-stub-repository) for a detailed -explanation. - -{% include code-caption.html content="src/__tests__/unit/controllers/product.controller.unit.ts" %} - -```ts -import { - createStubInstance, - expect, - sinon, - StubbedInstanceWithSinonAccessor, -} from '@loopback/testlab'; -import {ProductRepository} from '../../../repositories'; -import {ProductController} from '../../../controllers'; - -describe('ProductController (unit)', () => { - let repository: StubbedInstanceWithSinonAccessor; - beforeEach(givenStubbedRepository); - - describe('getDetails()', () => { - it('retrieves details of a product', async () => { - const controller = new ProductController(repository); - repository.stubs.find.resolves([{name: 'Pen', slug: 'pen'}]); - - const details = await controller.getDetails('pen'); - - expect(details).to.containEql({name: 'Pen', slug: 'pen'}); - sinon.assert.calledWithMatch(repository.stubs.find, { - where: {slug: 'pen'}, - }); - }); - }); - - function givenStubbedRepository() { - repository = createStubInstance(ProductRepository); - } -}); -``` - -### Unit test your models and repositories - -In a typical LoopBack application, models and repositories rely on behavior -provided by the framework (`@loopback/repository` package) and there is no need -to test LoopBack's built-in functionality. However, any additional -application-specific APIs do need new unit tests. - -For example, if the `Person` Model has properties `firstname`, `middlename` and -`surname` and provides a function to obtain the full name, then you should write -unit tests to verify the implementation of this additional method. - -Remember to use [Test data builders](#use-test-data-builders) whenever you need -valid data to create a new model instance. - -{% include code-caption.html content="src/__tests__/unit/models/person.model.unit.ts" %} - -```ts -import {Person} from '../../../models'; -import {givenPersonData} from '../../helpers/database.helpers'; -import {expect} from '@loopback/testlab'; - -describe('Person (unit)', () => { - // we recommend to group tests by method names - describe('getFullName()', () => { - it('uses all three parts when present', () => { - const person = givenPerson({ - firstname: 'Jane', - middlename: 'Smith', - surname: 'Brown', - }); - - const fullName = person.getFullName(); - expect(fullName).to.equal('Jane Smith Brown'); - }); - - it('omits middlename when not present', () => { - const person = givenPerson({ - firstname: 'Mark', - surname: 'Twain', - }); - - const fullName = person.getFullName(); - expect(fullName).to.equal('Mark Twain'); - }); - }); - - function givenPerson(data: Partial) { - return new Person(givenPersonData(data)); - } -}); -``` - -Writing a unit test for custom repository methods is not as straightforward -because `CrudRepository` is based on -[loopback-datasource-juggler](https://github.com/loopbackio/loopback-datasource-juggler) -which was not designed with dependency injection in mind. Instead, use -integration tests to verify the implementation of custom repository methods. For -more information, refer to -[Test your repositories against a real database](#test-your-repositories-against-a-real-database) -in [Integration Testing](#integration-testing). - -### Unit test your Sequence - -While it's possible to test a custom Sequence class in isolation, it's better to -rely on acceptance-level tests in this exceptional case. The reason is that a -custom Sequence class typically has many dependencies (which can make test setup -long and complex), and at the same time it provides very little functionality on -top of the injected sequence actions. Bugs are much more likely to be caused by -the way the real sequence action implementations interact together (which is not -covered by unit tests), instead of the Sequence code itself (which is the only -thing covered). - -See [Test Sequence customizations](#test-sequence-customizations) in -[Acceptance Testing](#acceptance-end-to-end-testing). - -## Integration testing - -Integration tests are considered "white-box" tests because they use an -"inside-out" approach that tests how multiple units work together or with -external services. You can use test doubles to isolate tested units from -external variables/state that are not part of the tested scenario. - -### Test your repositories against a real database - -There are two common reasons for adding repository tests: - -- Your models are using an advanced configuration, for example, custom column - mappings, and you want to verify this configuration is correctly picked up by - the framework. -- Your repositories have additional methods. - -Integration tests are one of the places to put the best practices in -[Data handling](#data-handling) to work: - -- Clean the database before each test -- Use test data builders -- Avoid sharing the same data for multiple tests - -Here is an example showing how to write an integration test for a custom -repository method `findByName`: - -{% include code-caption.html content="src/__tests__/integration/repositories/category.repository.integration.ts" %} - -```ts -import { - givenEmptyDatabase, - givenCategory, -} from '../../helpers/database.helpers'; -import {CategoryRepository} from '../../../repositories'; -import {expect} from '@loopback/testlab'; -import {testdb} from '../../fixtures/datasources/testdb.datasource'; - -describe('CategoryRepository (integration)', () => { - beforeEach(givenEmptyDatabase); - - describe('findByName(name)', () => { - it('return the correct category', async () => { - const stationery = await givenCategory({name: 'Stationery'}); - const repository = new CategoryRepository(testdb); - - const found = await repository.findByName('Stationery'); - - expect(found).to.deepEqual(stationery); - }); - }); -}); -``` - -### Test controllers and repositories together - -Integration tests running controllers with real repositories are important to -verify that the controllers use the repository API correctly, and that the -commands and queries produce expected results when executed on a real database. -These tests are similar to repository tests with controllers added as another -ingredient. - -{% include code-caption.html content="src/__tests__/integration/controllers/product.controller.integration.ts" %} - -```ts -import {expect} from '@loopback/testlab'; -import {givenEmptyDatabase, givenProduct} from '../../helpers/database.helpers'; -import {ProductController} from '../../../controllers'; -import {ProductRepository} from '../../../repositories'; -import {testdb} from '../../fixtures/datasources/testdb.datasource'; - -describe('ProductController (integration)', () => { - beforeEach(givenEmptyDatabase); - - describe('getDetails()', () => { - it('retrieves details of the given product', async () => { - const pencil = await givenProduct({name: 'Pencil', slug: 'pencil'}); - const controller = new ProductController(new ProductRepository(testdb)); - - const details = await controller.getDetails('pencil'); - - expect(details).to.containEql(pencil); - }); - }); -}); -``` - -### Test your Services against real backends - -When integrating with other services (including our own microservices), it's -important to verify that our client configuration is correct and the client -(service proxy) API is matching the actual service implementation. Ideally, -there should be at least one integration test for each endpoint (operation) -consumed by the application. - -To write an integration test, we need to: - -- Obtain an instance of the tested service proxy. Optionally modify the - connection configuration, for example change the target URL or configure a - caching proxy to speed up tests. -- Execute service proxy methods and verify that expected results were returned - by the backend service. - -#### Obtain a Service Proxy instance - -In -[Make service proxies easier to test](./Calling-other-APIs-and-Web-Services.md#make-service-proxies-easier-to-test), -we are suggesting to leverage Providers as a tool allowing both the IoC -framework and the tests to access service proxy instances. - -In the integration tests, a test helper should be written to obtain an instance -of the service proxy by invoking the provider. This helper should be typically -invoked once before the integration test suite begins. - -```ts -import {GeoService, GeoServiceProvider} from '../../services/geo.service'; -import {GeoDataSource} from '../../datasources/geo.datasource'; - -describe('GeoService', () => { - let service: GeoService; - before(givenGeoService); - - // to be done: add tests here - - function givenGeoService() { - const dataSource = new GeoDataSource(); - service = new GeoServiceProvider(dataSource).value(); - } -}); -``` - -If needed, you can tweak the datasource config before creating the service -instance: - -```ts -import {merge} from 'lodash'; -import {GeocoderDataSource} from '../datasources/geocoder.datasource'; - -function givenGeoService() { - const config = merge({}, GeocoderDataSource.defaultConfig, { - // your config overrides - }); - const dataSource = new GeoDataSource(config); - service = new GeoServiceProvider(dataSource).value(); -} -``` - -#### Test individual service methods - -With the service proxy instance available, integration tests can focus on -executing individual methods with the right set of input parameters; and -verifying the outcome of those calls. - -```ts -it('resolves an address to a geo point', async () => { - const points = await service.geocode('1 New Orchard Road, Armonk, 10504'); - - expect(points).to.deepEqual([ - { - lat: 41.109653, - lng: -73.72467, - }, - ]); -}); -``` - -## Acceptance (end-to-end) testing - -Automated acceptance (end-to-end) tests are considered "black-box" tests because -they use an "outside-in" approach that is not concerned about the internals of -the system. Acceptance tests perform the same actions (send the same HTTP -requests) as the clients and consumers of your API will do, and verify that the -results returned by the system match the expected results. - -Typically, acceptance tests start the application, make HTTP requests to the -server, and verify the returned response. LoopBack uses -[supertest](https://github.com/visionmedia/supertest) to create test code that -simplifies both the execution of HTTP requests and the verification of -responses. Remember to follow the best practices from -[Data handling](#data-handling) when setting up your database for tests: - -- Clean the database before each test -- Use test data builders -- Avoid sharing the same data for multiple tests - -### Validate your OpenAPI specification - -The OpenAPI specification is a cornerstone of applications that provide REST -APIs. It enables API consumers to leverage a whole ecosystem of related tooling. -To make the spec useful, you must ensure it's a valid OpenAPI Spec document, -ideally in an automated way that's an integral part of regular CI builds. -LoopBack's [testlab](https://www.npmjs.com/package/@loopback/testlab) module -provides a helper method `validateApiSpec` that builds on top of the popular -[swagger-parser](https://www.npmjs.com/package/swagger-parser) package. - -Example usage: - -{% include code-caption.html content= "src/__tests__/acceptance/api-spec.acceptance.ts" %} - -```ts -import {HelloWorldApplication} from '../..'; -import {RestServer} from '@loopback/rest'; -import {validateApiSpec} from '@loopback/testlab'; - -describe('API specification', () => { - it('api spec is valid', async () => { - const app = new HelloWorldApplication(); - const server = await app.getServer(RestServer); - const spec = server.getApiSpec(); - await validateApiSpec(spec); - }); -}); -``` - -### Perform an auto-generated smoke test of your REST API - -{% include important.html content=" The top-down approach for building LoopBack -applications is not yet fully supported. Therefore, the code outlined in this -section is outdated and may not work out of the box. Check out -https://github.com/loopbackio/loopback-next/issues/1882 for the epic tracking -the feature and [OpenAPI generator](OpenAPI-generator.md) page for artifact -generation from OpenAPI specs. -" %} - -The formal validity of your application's spec does not guarantee that your -implementation is actually matching the specified behavior. To keep your spec in -sync with your implementation, you should use an automated tool like -[Dredd](https://www.npmjs.com/package/dredd) to run a set of smoke tests to -verify your app conforms to the spec. - -Automated testing tools usually require hints in your specification to tell them -how to create valid requests or what response data to expect. Dredd in -particular relies on response -[examples](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#exampleObject) -and request parameter -[x-example](http://dredd.org/en/latest/how-to-guides.html#example-values-for-request-parameters) -fields. Extending your API spec with examples is a good thing on its own, since -developers consuming your API will find them useful too. - -Here is an example showing how to run Dredd to test your API against the spec: - -{% include code-caption.html content= "src/__tests__/acceptance/api-spec.acceptance.ts" %} - -```ts -import {expect} from '@loopback/testlab'; -import {HelloWorldApplication} from '../..'; -import {RestServer, RestBindings} from '@loopback/rest'; -import {spec} from '../../apidefs/openapi'; -const Dredd = require('dredd'); - -describe('API (acceptance)', () => { - let app: HelloWorldApplication; - /* eslint-disable @typescript-eslint/no-explicit-any */ - let dredd: any; - before(initEnvironment); - after(async () => { - await app.stop(); - }); - - it('conforms to the specification', done => { - dredd.run((err: Error, stats: object) => { - if (err) return done(err); - expect(stats).to.containDeep({ - failures: 0, - errors: 0, - skipped: 0, - }); - done(); - }); - }); - - async function initEnvironment() { - app = new HelloWorldApplication(); - const server = await app.getServer(RestServer); - // For testing, we'll let the OS pick an available port by setting - // RestBindings.PORT to 0. - server.bind(RestBindings.PORT).to(0); - // app.start() starts up the HTTP server and binds the acquired port - // number to RestBindings.PORT. - await app.boot(); - await app.start(); - // Get the real port number. - const port = await server.get(RestBindings.PORT); - const baseUrl = `http://localhost:${port}`; - const config: object = { - server: baseUrl, // base path to the end points - options: { - level: 'fail', // report 'fail' case only - silent: false, // false for helpful debugging info - path: [`${baseUrl}/openapi.json`], // to download apiSpec from the service - }, - }; - dredd = new Dredd(config); - } -}); -``` - -The user experience needs improvement and we are looking into better solutions. -See [GitHub issue #644](https://github.com/loopbackio/loopback-next/issues/644). -Let us know if you have any recommendations! - -### Test your individual REST API endpoints - -You should have at least one acceptance (end-to-end) test for each of your REST -API endpoints. Consider adding more tests if your endpoint depends on (custom) -sequence actions to modify the behavior when the corresponding controller method -is invoked via REST, compared to behavior observed when the controller method is -invoked directly via JavaScript/TypeScript API. For example, if your endpoint -returns different responses to regular users and to admin users, then you should -two tests (one test for each user role). - -Here is an example of an acceptance test: - -{% include code-caption.html content= "src/__tests__/acceptance/product.acceptance.ts" %} - -```ts -import {HelloWorldApplication} from '../..'; -import {Client, createRestAppClient, expect} from '@loopback/testlab'; -import {givenEmptyDatabase, givenProduct} from '../helpers/database.helpers'; -import {RestServer, RestBindings} from '@loopback/rest'; -import {testdb} from '../fixtures/datasources/testdb.datasource'; - -describe('Product (acceptance)', () => { - let app: HelloWorldApplication; - let client: Client; - - before(givenEmptyDatabase); - before(givenRunningApp); - after(async () => { - await app.stop(); - }); - - it('retrieves product details', async () => { - // arrange - const product = await givenProduct({ - name: 'Ink Pen', - slug: 'ink-pen', - price: 1, - category: 'Stationery', - description: 'The ultimate ink-powered pen for daily writing', - label: 'popular', - available: true, - endDate: null, - }); - const expected = Object.assign({id: product.id}, product); - - // act - const response = await client.get('/product/ink-pen'); - - // assert - expect(response.body).to.containEql(expected); - }); - - async function givenRunningApp() { - app = new HelloWorldApplication({ - rest: { - port: 0, - }, - }); - await app.boot(); - - // change to use the test datasource after the app has been booted so that - // it is not overridden - app.dataSource(testdb); - await app.start(); - - client = createRestAppClient(app); - } -}); -``` - -### Test Sequence customizations - -Custom sequence behavior is best tested by observing changes in behavior of the -affected endpoints. For example, if your sequence has an authentication step -that rejects anonymous requests for certain endpoints, then you can write a test -making an anonymous request to those endpoints to verify that it's correctly -rejected. These tests are essentially the same as the tests verifying -implementation of individual endpoints as described in the previous section. - -# Code coverage - -`@loopback/build` contains a command line tool (`lb-nyc`) that acts as a wrapper -for [`nyc`](https://github.com/istanbuljs/nyc). - -To set up code coverage: - -- Create `.nycrc` in your project's root directory - - ```json - { - "include": ["dist"], - "exclude": ["dist/__tests__/"], - "extension": [".js", ".ts"], - "reporter": ["text", "html"], - "exclude-after-remap": false - } - ``` - -- Update your `package.json` scripts: - - ```json - "precoverage": "npm test", - "coverage": "open coverage/index.html", - "coverage:ci": "lb-nyc report --reporter=text-lcov | coveralls", - "test": "lb-nyc npm run mocha", - "test:ci": "lb-nyc npm run mocha" - ``` - - `coverage:ci` sets up integration with [Coveralls](https://coveralls.io/). +--- +lang: en +title: 'Testing your application' +keywords: + LoopBack 4.0, LoopBack 4, Node.js, TypeScript, OpenAPI, Node.js, TypeScript, + OpenAPI +sidebar: lb4_sidebar +permalink: /doc/en/lb4/Testing-your-application.html +--- + +## Overview + +A thorough automated test suite is important because it: + +- Ensures your application works as expected. +- Prevents regressions when new features are added and bugs are fixed. +- Helps new and existing developers understand different parts of the codebase + (knowledge sharing). +- Speeds up development over the long run (the code writes itself!). + +### Types of tests + +We encourage writing tests from a few perspectives, mainly +[black-box testing](https://en.wikipedia.org/wiki/Black-box_testing) +(acceptance) and +[white-box testing](https://en.wikipedia.org/wiki/White-box_testing) +(integration and unit). Tests are usually written using typical patterns such as +[`arrange/act/assert`](https://msdn.microsoft.com/en-us/library/hh694602.aspx#Anchor_3) +or [`given/when/then`](https://martinfowler.com/bliki/GivenWhenThen.html). Both +styles work well, so pick one that you're comfortable with and start writing +tests! + +For an introduction to automated testing, see +[Define your testing strategy](Defining-your-testing-strategy.md). + +{% include important.html content=" A great test suite requires you to think +smaller and favor fast and focused unit tests over slow end-to-end tests. +" %} + +This article is a reference guide for common types of tests and test helpers. + +## Project setup + +An automated test suite requires a test runner to execute all the tests and +produce a summary report. We use and recommend [Mocha](https://mochajs.org). + +In addition to a test runner, the test suites generally require: + +- An assertion library (we recommend [Should.js](https://shouldjs.github.io)). +- A library for making HTTP calls and verifying their results (we recommend + [supertest](https://github.com/visionmedia/supertest)). +- A library for creating test doubles (we recommend + [Sinon.JS](http://sinonjs.org/)). + +The [@loopback/testlab](https://www.npmjs.com/package/@loopback/testlab) module +integrates these packages and makes them easy to use together with LoopBack. + +### Set up testing infrastructure with LoopBack CLI + +LoopBack applications that have been generated using the `lb4 app` command from +`@loopback/cli` come with `@loopback/testlab` and `mocha` as a default, so no +other testing infrastructure setup is needed. + +### Setup testing infrastructure manually + +If you have an existing application install `mocha` and `@loopback/testlab`: + +```shell +npm install --save-dev mocha @loopback/testlab +``` + +Your `package.json` should then look something like this: + +```js +{ + // ... + "devDependencies": { + "@loopback/testlab": "^", + "@types/mocha": "^", + "mocha": "^" + }, + "scripts": { + "test": "mocha --recursive \"dist/test\"" + } + // ... +} +``` + +## Data handling + +Tests accessing a real database often require existing data. For example, a +method listing all products needs some products in the database; a method to +create a new product instance must determine which properties are required and +any restrictions on their values. There are various approaches to address this +issue. Many of them unfortunately make the test suite difficult to understand, +difficult to maintain, and prone to test failures unrelated to the changes made. + +Our approach to data handling, based on our experience, is described in this +section. + +### Create a test datasource + +As we would prefer full control of the database and even the development +database is not something we want to clean before each test it's handy to use an +independent in-memory datasource which is filled appropriately using +[test data builders](#use-test-data-builders) before each test run. + +{% include code-caption.html content="src/__tests__/fixtures/datasources/testdb.datasource.ts" %} + +```ts +import {juggler} from '@loopback/repository'; + +export const testdb: juggler.DataSource = new juggler.DataSource({ + name: 'db', + connector: 'memory', +}); +``` + +{% include important.html content="The `name` property of your test datasource must match your real datasource's name." %} + +### Clean the database before each test + +Start with a clean database before each test. This may seem counter-intuitive: +why not reset the database after the test has finished? When a test fails and +the database is cleaned after the test has finished, then it's difficult to +observe what was stored in the database and why the test failed. When the +database is cleaned in the beginning, then any failing test will leave the +database in the state that caused the test to fail. + +To clean the database before each test, set up a `beforeEach` hook to call a +helper method; for example: + +{% include code-caption.html content="src/__tests__/helpers/database.helpers.ts" %} + +```ts +import {ProductRepository, CategoryRepository} from '../../repositories'; +import {testdb} from '../fixtures/datasources/testdb.datasource'; + +export async function givenEmptyDatabase() { + await new ProductRepository(testdb).deleteAll(); + await new CategoryRepository(testdb).deleteAll(); +} +``` + +In case a repository includes a relation to another repository, ie. Product +belongs to Category, include it in the repository call, for example: + +{% include code-caption.html content="src/__tests__/helpers/database.helpers.ts" %} + +```ts +import {Getter} from '@loopback/core'; +import {ProductRepository, CategoryRepository} from '../../repositories'; +import {testdb} from '../fixtures/datasources/testdb.datasource'; + +export async function givenEmptyDatabase() { + let categoryRepository: CategoryRepository; + let productRepository: ProductRepository; + + categoryRepository = new CategoryRepository( + testdb, + async () => productRepository, + ); + + productRepository = new ProductRepository( + testdb, + async () => categoryRepository, + ); + + await productRepository.deleteAll(); + await categoryRepository.deleteAll(); +} +``` + +{% include code-caption.html content="src/__tests__/integration/controllers/product.controller.integration.ts" %} + +```ts +// in your test file +import {givenEmptyDatabase} from '../../helpers/database.helpers'; + +describe('ProductController (integration)', () => { + before(givenEmptyDatabase); + // etc. +}); +``` + +### Use test data builders + +To avoid duplicating code for creating model data that is complete with required +properties, use shared +[test data builders](http://www.natpryce.com/articles/000714.html). This enables +tests to provide the small subset of properties that is strictly required by the +tested scenario. Using shared test builders will help your tests to be: + +- Easier to understand, since it's immediately clear what model properties are + relevant to the tests. If the tests set the required properties, it is + difficult to tell whether the properties are actually relevant to the tested + scenario. + +- Easier to maintain. As your data model evolves, you will need to add more + required properties. If the tests build the model instance data manually, all + the tests must be manually updated to set a new required property. With a + shared test data builder, you update a single location with the new property. + +See +[@loopback/openapi-spec-builder](https://www.npmjs.com/package/@loopback/openapi-spec-builder) +for an example of how to apply this design pattern for building OpenAPI Spec +documents. + +In practice, a simple function that adds missing required properties is +sufficient. + +{% include code-caption.html content="src/__tests__/helpers/database.helpers.ts" %} + +```ts +// ... +export function givenProductData(data?: Partial) { + return Object.assign( + { + name: 'a-product-name', + slug: 'a-product-slug', + price: 1, + description: 'a-product-description', + available: true, + }, + data, + ); +} + +export async function givenProduct(data?: Partial) { + return new ProductRepository(testdb).create(givenProductData(data)); +} +// ... +``` + +### Avoid sharing the same data for multiple tests + +It's tempting to define a small set of data to be shared by all tests. For +example, in an e-commerce application, you might pre-populate the database with +a few categories, some products, an admin user and a customer. This approach has +several downsides: + +- When trying to understand any individual test, it's difficult to tell what + part of the pre-populated data is essential for the test and what's + irrelevant. For example, in a test checking the method counting the number of + products in a category using a pre-populated category "Stationery", is it + important that "Stationery" contains nested sub-categories or is that fact + irrelevant? If it's irrelevant, then what are the other tests that depend on + it? + +- As the application grows and new features are added, it's easier to add more + properties to existing model instances rather than create new instances using + only the properties required by the new features. For example, when adding a + category image, it's easier to add image to an existing category "Stationery" + and perhaps keep another category "Groceries" without any image, rather than + creating two new categories "CategoryWithAnImage" and "CategoryMissingImage". + This further amplifies the previous problem, because it's not clear that + "Groceries" is the category that should be used by tests requiring a category + with no image - the category name does not provide any hints on that. + +- As the shared dataset grows (together with the application), the time required + to bring the database into its initial state grows too. Instead of running a + few "DELETE ALL" queries before each test (which is relatively fast), you may + have to run tens or hundreds of different commands used to create different + model instances, thus triggering slow index rebuilds along the way and slowing + down the test suite considerably. + +Use the test data builders described in the previous section to populate your +database with the data specific to your test only. + +### Avoid interdependent tests + +A related anti-pattern is making tests rely on the **order** they run in, where +one test fills the database with data that later tests consume: + +```ts +it('creates a todo', async () => { + item = (await client.post('/todo').send(payload).expect(200)).body; +}); + +// ... + +it('successfully deletes todos', async () => { + await client.del(`/todo/${item.id}`).send(); + await client.get(`/todo/${item.id}`).send().expect(404); +}); +``` + +The second test passes only because the first test ran before it and created +`item`. This is bad for two reasons: + +- If the first test fails, every subsequent test fails too — not because of a + real defect in what they exercise, but because the data they expect was never + created. The resulting error messages point at the wrong code and are hard to + diagnose. +- Individual tests can no longer run on their own, e.g. with `it.only()` or + `mocha -g "test name"`. A test that depends on earlier tests is impossible to + re-run in isolation, which makes debugging and CI bisecting much harder. + +Keep every test independent: set up exactly the data it needs in +`beforeEach()` (or test-data builders) and tear it down after, so the suite +passes regardless of order, filtering, or parallelism. + + + + + +Write higher-level helpers to share the code for re-creating common scenarios. +For example, if your application has two kinds of users (admins and customers), +then you may write the following helpers to simplify writing acceptance tests +checking access control: + +```ts +async function givenAdminAndCustomer() { + return { + admin: await givenUser({role: Roles.ADMIN}), + customer: await givenUser({role: Roles.CUSTOMER}), + }; +} +``` + +## Unit testing + +Unit tests are considered "white-box" tests because they use an "inside-out" +approach where the tests know about the internals and control all the variables +of the system being tested. Individual units are tested in isolation and their +dependencies are replaced with +[Test doubles](https://en.wikipedia.org/wiki/Test_double). + +### Use test doubles + +Test doubles are functions or objects that look and behave like the real +variants used in production, but are actually simplified versions that give the +test more control of the behavior. For example, reproducing the situation where +reading from a file failed because of a hard-drive error is pretty much +impossible. However, using a test double to simulate the file-system API will +provide control over what each call returns. + +[Sinon.JS](http://sinonjs.org/) has become the de-facto standard for test +doubles in Node.js and JavaScript/TypeScript in general. The `@loopback/testlab` +package comes with Sinon preconfigured with TypeScript type definitions and +integrated with Should.js assertions. + +There are three kinds of test doubles provided by Sinon.JS: + +- [Test spies](http://sinonjs.org/releases/v4.0.1/spies/) are functions that + record arguments, the return value, the value of `this`, and exceptions thrown + (if any) for all its calls. There are two types of spies: Some are anonymous + functions, while others wrap methods that already exist in the system under + test. + +- [Test stubs](http://sinonjs.org/releases/v4.0.1/stubs/) are functions (spies) + with pre-programmed behavior. As spies, stubs can be either anonymous, or wrap + existing functions. When wrapping an existing function with a stub, the + original function is not called. + +- [Test mocks](http://sinonjs.org/releases/v4.0.1/mocks/) (and mock + expectations) are fake methods (like spies) with pre-programmed behavior (like + stubs) as well as pre-programmed expectations. A mock will fail your test if + it is not used as expected. + +{% include note.html content=" +We recommend against using test mocks. With test +mocks, the expectations must be defined before the tested scenario is executed, +which breaks the recommended test layout 'arrange-act-assert' (or +'given-when-then') and also produces code that's difficult to comprehend. +" %} + +#### Create a stub Repository + +When writing an application that accesses data in a database, the best practice +is to use [repositories](Repository.md) to encapsulate all +data-access/persistence-related code. Other parts of the application (typically +[controllers](Controller.md)) can then depend on these repositories for data +access. To test Repository dependents (for example, Controllers) in isolation, +we need to provide a test double, usually as a test stub. + +In traditional object-oriented languages like Java or C#, to enable unit tests +to provide a custom implementation of the repository API, the controller needs +to depend on an interface describing the API, and the repository implementation +needs to implement this interface. The situation is easier in JavaScript and +TypeScript. Thanks to the dynamic nature of the language, it’s possible to +mock/stub entire classes. + +Creating a test double for a repository class is very easy using the Sinon.JS +utility function `createStubInstance`. It's important to create a new stub +instance for each unit test in order to prevent unintended re-use of +pre-programmed behavior between (unrelated) tests. + +```ts +describe('ProductController', () => { + let repository: ProductRepository; + beforeEach(givenStubbedRepository); + + // your unit tests + + function givenStubbedRepository() { + repository = sinon.createStubInstance(ProductRepository); + } +}); +``` + +In your unit tests, you will usually want to program the behavior of stubbed +methods (what they should return) and then verify that the Controller (unit +under test) called the right method with the correct arguments. + +Configure stub's behavior at the beginning of your unit test (in the "arrange" +or "given" section): + +```ts +// repository.find() will return a promise that +// will be resolved with the provided array +const findStub = repository.find as sinon.SinonStub; +findStub.resolves([{id: 1, name: 'Pen'}]); +``` + +Verify how was the stubbed method executed at the end of your unit test (in the +"assert" or "then" section): + +```ts +// expect that repository.find() was called with the first +// argument deeply-equal to the provided object +sinon.assert.calledWithMatch(findStub, {where: {id: 1}}); +``` + +See [Unit test your controllers](#unit-test-your-controllers) for a full +example. + +#### Create a stub Service + +If your controller relies on service proxy for service oriented backends such as +REST APIs, SOAP Web Services, or gRPC microservices, then we can create stubs +for the service akin to the steps outlined in the above +[stub repository](#create-a-stub-repository) section. Consider a dependency on a +`GeoCoder` service that relies on a remote REST API for returning coordinates +for a specific address. + +```ts +export interface GeocoderService { + geocode(address: string): Promise; +} +``` + +The first step is to create a mocked instance of the `GeocoderService` API and +configure its `geocode` method as a Sinon stub: + +```ts +describe('GeocoderController', () => { + let geoService: GeoCoderService; + let geocode: sinon.SinonStub; + + beforeEach(givenMockGeoCoderService); + + // your unit tests + + function givenMockGeoCoderService() { + // this creates a stub with GeocoderService API + // in a way that allows the compiler to verify type correctness + geoService = {geocode: sinon.stub()}; + + // this creates a reference to the stubbed "geocode" method + // because "geoService.geocode" has type from GeocoderService + // and does not provide Sinon APIs + geocode = geoService.geocode as sinon.SinonStub; + } +}); +``` + +Afterwards, we can configure the `geocode` stub's behaviour before the `act` +phase of our test(s): + +```ts +// geoService.geocode() will return a promise that +// will be resolved with the provided array +geocode.resolves([{y: 41.109653, x: -73.72467}]); +``` + +Lastly, we'll verify how the stub was executed: + +```ts +// expect that geoService.geocode() was called with the first +// argument equal to the provided address string +sinon.assert.calledWithMatch(geocode, '1 New Orchard Road, Armonk, 10504'); +``` + +Check out +[TodoController unit tests](https://github.com/loopbackio/loopback-next/blob/bd0c45033503f631a533ad6176620354d9cf6768/examples/todo/src/__tests__/unit/controllers/todo.controller.unit.ts#L53-L71) +illustrating the above points in action for more information. + +### Unit test your Controllers + +Unit tests should apply to the smallest piece of code possible to ensure that +other variables and state changes do not pollute the result. A typical unit test +creates a controller instance with dependencies replaced by test doubles and +directly calls the tested method. The example below gives the controller a stub +implementation of its repository dependency using the `testlab` +`createStubInstance` function, ensures the controller calls the repository's +`find()` method with a correct query, and returns back the query results. See +[Create a stub repository](#create-a-stub-repository) for a detailed +explanation. + +{% include code-caption.html content="src/__tests__/unit/controllers/product.controller.unit.ts" %} + +```ts +import { + createStubInstance, + expect, + sinon, + StubbedInstanceWithSinonAccessor, +} from '@loopback/testlab'; +import {ProductRepository} from '../../../repositories'; +import {ProductController} from '../../../controllers'; + +describe('ProductController (unit)', () => { + let repository: StubbedInstanceWithSinonAccessor; + beforeEach(givenStubbedRepository); + + describe('getDetails()', () => { + it('retrieves details of a product', async () => { + const controller = new ProductController(repository); + repository.stubs.find.resolves([{name: 'Pen', slug: 'pen'}]); + + const details = await controller.getDetails('pen'); + + expect(details).to.containEql({name: 'Pen', slug: 'pen'}); + sinon.assert.calledWithMatch(repository.stubs.find, { + where: {slug: 'pen'}, + }); + }); + }); + + function givenStubbedRepository() { + repository = createStubInstance(ProductRepository); + } +}); +``` + +### Unit test your models and repositories + +In a typical LoopBack application, models and repositories rely on behavior +provided by the framework (`@loopback/repository` package) and there is no need +to test LoopBack's built-in functionality. However, any additional +application-specific APIs do need new unit tests. + +For example, if the `Person` Model has properties `firstname`, `middlename` and +`surname` and provides a function to obtain the full name, then you should write +unit tests to verify the implementation of this additional method. + +Remember to use [Test data builders](#use-test-data-builders) whenever you need +valid data to create a new model instance. + +{% include code-caption.html content="src/__tests__/unit/models/person.model.unit.ts" %} + +```ts +import {Person} from '../../../models'; +import {givenPersonData} from '../../helpers/database.helpers'; +import {expect} from '@loopback/testlab'; + +describe('Person (unit)', () => { + // we recommend to group tests by method names + describe('getFullName()', () => { + it('uses all three parts when present', () => { + const person = givenPerson({ + firstname: 'Jane', + middlename: 'Smith', + surname: 'Brown', + }); + + const fullName = person.getFullName(); + expect(fullName).to.equal('Jane Smith Brown'); + }); + + it('omits middlename when not present', () => { + const person = givenPerson({ + firstname: 'Mark', + surname: 'Twain', + }); + + const fullName = person.getFullName(); + expect(fullName).to.equal('Mark Twain'); + }); + }); + + function givenPerson(data: Partial) { + return new Person(givenPersonData(data)); + } +}); +``` + +Writing a unit test for custom repository methods is not as straightforward +because `CrudRepository` is based on +[loopback-datasource-juggler](https://github.com/loopbackio/loopback-datasource-juggler) +which was not designed with dependency injection in mind. Instead, use +integration tests to verify the implementation of custom repository methods. For +more information, refer to +[Test your repositories against a real database](#test-your-repositories-against-a-real-database) +in [Integration Testing](#integration-testing). + +### Unit test your Sequence + +While it's possible to test a custom Sequence class in isolation, it's better to +rely on acceptance-level tests in this exceptional case. The reason is that a +custom Sequence class typically has many dependencies (which can make test setup +long and complex), and at the same time it provides very little functionality on +top of the injected sequence actions. Bugs are much more likely to be caused by +the way the real sequence action implementations interact together (which is not +covered by unit tests), instead of the Sequence code itself (which is the only +thing covered). + +See [Test Sequence customizations](#test-sequence-customizations) in +[Acceptance Testing](#acceptance-end-to-end-testing). + +## Integration testing + +Integration tests are considered "white-box" tests because they use an +"inside-out" approach that tests how multiple units work together or with +external services. You can use test doubles to isolate tested units from +external variables/state that are not part of the tested scenario. + +### Test your repositories against a real database + +There are two common reasons for adding repository tests: + +- Your models are using an advanced configuration, for example, custom column + mappings, and you want to verify this configuration is correctly picked up by + the framework. +- Your repositories have additional methods. + +Integration tests are one of the places to put the best practices in +[Data handling](#data-handling) to work: + +- Clean the database before each test +- Use test data builders +- Avoid sharing the same data for multiple tests + +Here is an example showing how to write an integration test for a custom +repository method `findByName`: + +{% include code-caption.html content="src/__tests__/integration/repositories/category.repository.integration.ts" %} + +```ts +import { + givenEmptyDatabase, + givenCategory, +} from '../../helpers/database.helpers'; +import {CategoryRepository} from '../../../repositories'; +import {expect} from '@loopback/testlab'; +import {testdb} from '../../fixtures/datasources/testdb.datasource'; + +describe('CategoryRepository (integration)', () => { + beforeEach(givenEmptyDatabase); + + describe('findByName(name)', () => { + it('return the correct category', async () => { + const stationery = await givenCategory({name: 'Stationery'}); + const repository = new CategoryRepository(testdb); + + const found = await repository.findByName('Stationery'); + + expect(found).to.deepEqual(stationery); + }); + }); +}); +``` + +### Test controllers and repositories together + +Integration tests running controllers with real repositories are important to +verify that the controllers use the repository API correctly, and that the +commands and queries produce expected results when executed on a real database. +These tests are similar to repository tests with controllers added as another +ingredient. + +{% include code-caption.html content="src/__tests__/integration/controllers/product.controller.integration.ts" %} + +```ts +import {expect} from '@loopback/testlab'; +import {givenEmptyDatabase, givenProduct} from '../../helpers/database.helpers'; +import {ProductController} from '../../../controllers'; +import {ProductRepository} from '../../../repositories'; +import {testdb} from '../../fixtures/datasources/testdb.datasource'; + +describe('ProductController (integration)', () => { + beforeEach(givenEmptyDatabase); + + describe('getDetails()', () => { + it('retrieves details of the given product', async () => { + const pencil = await givenProduct({name: 'Pencil', slug: 'pencil'}); + const controller = new ProductController(new ProductRepository(testdb)); + + const details = await controller.getDetails('pencil'); + + expect(details).to.containEql(pencil); + }); + }); +}); +``` + +### Test your Services against real backends + +When integrating with other services (including our own microservices), it's +important to verify that our client configuration is correct and the client +(service proxy) API is matching the actual service implementation. Ideally, +there should be at least one integration test for each endpoint (operation) +consumed by the application. + +To write an integration test, we need to: + +- Obtain an instance of the tested service proxy. Optionally modify the + connection configuration, for example change the target URL or configure a + caching proxy to speed up tests. +- Execute service proxy methods and verify that expected results were returned + by the backend service. + +#### Obtain a Service Proxy instance + +In +[Make service proxies easier to test](./Calling-other-APIs-and-Web-Services.md#make-service-proxies-easier-to-test), +we are suggesting to leverage Providers as a tool allowing both the IoC +framework and the tests to access service proxy instances. + +In the integration tests, a test helper should be written to obtain an instance +of the service proxy by invoking the provider. This helper should be typically +invoked once before the integration test suite begins. + +```ts +import {GeoService, GeoServiceProvider} from '../../services/geo.service'; +import {GeoDataSource} from '../../datasources/geo.datasource'; + +describe('GeoService', () => { + let service: GeoService; + before(givenGeoService); + + // to be done: add tests here + + function givenGeoService() { + const dataSource = new GeoDataSource(); + service = new GeoServiceProvider(dataSource).value(); + } +}); +``` + +If needed, you can tweak the datasource config before creating the service +instance: + +```ts +import {merge} from 'lodash'; +import {GeocoderDataSource} from '../datasources/geocoder.datasource'; + +function givenGeoService() { + const config = merge({}, GeocoderDataSource.defaultConfig, { + // your config overrides + }); + const dataSource = new GeoDataSource(config); + service = new GeoServiceProvider(dataSource).value(); +} +``` + +#### Test individual service methods + +With the service proxy instance available, integration tests can focus on +executing individual methods with the right set of input parameters; and +verifying the outcome of those calls. + +```ts +it('resolves an address to a geo point', async () => { + const points = await service.geocode('1 New Orchard Road, Armonk, 10504'); + + expect(points).to.deepEqual([ + { + lat: 41.109653, + lng: -73.72467, + }, + ]); +}); +``` + +## Acceptance (end-to-end) testing + +Automated acceptance (end-to-end) tests are considered "black-box" tests because +they use an "outside-in" approach that is not concerned about the internals of +the system. Acceptance tests perform the same actions (send the same HTTP +requests) as the clients and consumers of your API will do, and verify that the +results returned by the system match the expected results. + +Typically, acceptance tests start the application, make HTTP requests to the +server, and verify the returned response. LoopBack uses +[supertest](https://github.com/visionmedia/supertest) to create test code that +simplifies both the execution of HTTP requests and the verification of +responses. Remember to follow the best practices from +[Data handling](#data-handling) when setting up your database for tests: + +- Clean the database before each test +- Use test data builders +- Avoid sharing the same data for multiple tests + +### Validate your OpenAPI specification + +The OpenAPI specification is a cornerstone of applications that provide REST +APIs. It enables API consumers to leverage a whole ecosystem of related tooling. +To make the spec useful, you must ensure it's a valid OpenAPI Spec document, +ideally in an automated way that's an integral part of regular CI builds. +LoopBack's [testlab](https://www.npmjs.com/package/@loopback/testlab) module +provides a helper method `validateApiSpec` that builds on top of the popular +[swagger-parser](https://www.npmjs.com/package/swagger-parser) package. + +Example usage: + +{% include code-caption.html content= "src/__tests__/acceptance/api-spec.acceptance.ts" %} + +```ts +import {HelloWorldApplication} from '../..'; +import {RestServer} from '@loopback/rest'; +import {validateApiSpec} from '@loopback/testlab'; + +describe('API specification', () => { + it('api spec is valid', async () => { + const app = new HelloWorldApplication(); + const server = await app.getServer(RestServer); + const spec = server.getApiSpec(); + await validateApiSpec(spec); + }); +}); +``` + +### Perform an auto-generated smoke test of your REST API + +{% include important.html content=" The top-down approach for building LoopBack +applications is not yet fully supported. Therefore, the code outlined in this +section is outdated and may not work out of the box. Check out +https://github.com/loopbackio/loopback-next/issues/1882 for the epic tracking +the feature and [OpenAPI generator](OpenAPI-generator.md) page for artifact +generation from OpenAPI specs. +" %} + +The formal validity of your application's spec does not guarantee that your +implementation is actually matching the specified behavior. To keep your spec in +sync with your implementation, you should use an automated tool like +[Dredd](https://www.npmjs.com/package/dredd) to run a set of smoke tests to +verify your app conforms to the spec. + +Automated testing tools usually require hints in your specification to tell them +how to create valid requests or what response data to expect. Dredd in +particular relies on response +[examples](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#exampleObject) +and request parameter +[x-example](http://dredd.org/en/latest/how-to-guides.html#example-values-for-request-parameters) +fields. Extending your API spec with examples is a good thing on its own, since +developers consuming your API will find them useful too. + +Here is an example showing how to run Dredd to test your API against the spec: + +{% include code-caption.html content= "src/__tests__/acceptance/api-spec.acceptance.ts" %} + +```ts +import {expect} from '@loopback/testlab'; +import {HelloWorldApplication} from '../..'; +import {RestServer, RestBindings} from '@loopback/rest'; +import {spec} from '../../apidefs/openapi'; +const Dredd = require('dredd'); + +describe('API (acceptance)', () => { + let app: HelloWorldApplication; + /* eslint-disable @typescript-eslint/no-explicit-any */ + let dredd: any; + before(initEnvironment); + after(async () => { + await app.stop(); + }); + + it('conforms to the specification', done => { + dredd.run((err: Error, stats: object) => { + if (err) return done(err); + expect(stats).to.containDeep({ + failures: 0, + errors: 0, + skipped: 0, + }); + done(); + }); + }); + + async function initEnvironment() { + app = new HelloWorldApplication(); + const server = await app.getServer(RestServer); + // For testing, we'll let the OS pick an available port by setting + // RestBindings.PORT to 0. + server.bind(RestBindings.PORT).to(0); + // app.start() starts up the HTTP server and binds the acquired port + // number to RestBindings.PORT. + await app.boot(); + await app.start(); + // Get the real port number. + const port = await server.get(RestBindings.PORT); + const baseUrl = `http://localhost:${port}`; + const config: object = { + server: baseUrl, // base path to the end points + options: { + level: 'fail', // report 'fail' case only + silent: false, // false for helpful debugging info + path: [`${baseUrl}/openapi.json`], // to download apiSpec from the service + }, + }; + dredd = new Dredd(config); + } +}); +``` + +The user experience needs improvement and we are looking into better solutions. +See [GitHub issue #644](https://github.com/loopbackio/loopback-next/issues/644). +Let us know if you have any recommendations! + +### Test your individual REST API endpoints + +You should have at least one acceptance (end-to-end) test for each of your REST +API endpoints. Consider adding more tests if your endpoint depends on (custom) +sequence actions to modify the behavior when the corresponding controller method +is invoked via REST, compared to behavior observed when the controller method is +invoked directly via JavaScript/TypeScript API. For example, if your endpoint +returns different responses to regular users and to admin users, then you should +two tests (one test for each user role). + +Here is an example of an acceptance test: + +{% include code-caption.html content= "src/__tests__/acceptance/product.acceptance.ts" %} + +```ts +import {HelloWorldApplication} from '../..'; +import {Client, createRestAppClient, expect} from '@loopback/testlab'; +import {givenEmptyDatabase, givenProduct} from '../helpers/database.helpers'; +import {RestServer, RestBindings} from '@loopback/rest'; +import {testdb} from '../fixtures/datasources/testdb.datasource'; + +describe('Product (acceptance)', () => { + let app: HelloWorldApplication; + let client: Client; + + before(givenEmptyDatabase); + before(givenRunningApp); + after(async () => { + await app.stop(); + }); + + it('retrieves product details', async () => { + // arrange + const product = await givenProduct({ + name: 'Ink Pen', + slug: 'ink-pen', + price: 1, + category: 'Stationery', + description: 'The ultimate ink-powered pen for daily writing', + label: 'popular', + available: true, + endDate: null, + }); + const expected = Object.assign({id: product.id}, product); + + // act + const response = await client.get('/product/ink-pen'); + + // assert + expect(response.body).to.containEql(expected); + }); + + async function givenRunningApp() { + app = new HelloWorldApplication({ + rest: { + port: 0, + }, + }); + await app.boot(); + + // change to use the test datasource after the app has been booted so that + // it is not overridden + app.dataSource(testdb); + await app.start(); + + client = createRestAppClient(app); + } +}); +``` + +### Test Sequence customizations + +Custom sequence behavior is best tested by observing changes in behavior of the +affected endpoints. For example, if your sequence has an authentication step +that rejects anonymous requests for certain endpoints, then you can write a test +making an anonymous request to those endpoints to verify that it's correctly +rejected. These tests are essentially the same as the tests verifying +implementation of individual endpoints as described in the previous section. + +# Code coverage + +`@loopback/build` contains a command line tool (`lb-nyc`) that acts as a wrapper +for [`nyc`](https://github.com/istanbuljs/nyc). + +To set up code coverage: + +- Create `.nycrc` in your project's root directory + + ```json + { + "include": ["dist"], + "exclude": ["dist/__tests__/"], + "extension": [".js", ".ts"], + "reporter": ["text", "html"], + "exclude-after-remap": false + } + ``` + +- Update your `package.json` scripts: + + ```json + "precoverage": "npm test", + "coverage": "open coverage/index.html", + "coverage:ci": "lb-nyc report --reporter=text-lcov | coveralls", + "test": "lb-nyc npm run mocha", + "test:ci": "lb-nyc npm run mocha" + ``` + + `coverage:ci` sets up integration with [Coveralls](https://coveralls.io/).