Mark slow tests to allow for quicker iteration

[ekohl]

Why are you introducing these changes? (Problem description, related links)

While working on #511 I wanted to quickly run tests again and again. The full service restart is slow and not really something I was interested in.

What are the changes introduced in this pull request?

You can now skip slow tests with:

pytest -k "not slow"

How to test this pull request

Steps to reproduce:

• Watch CI and verify it still runs the slow tests
• Manually run pytest to skip the slow tests

Checklist

• Tests added/updated (if applicable)
• Documentation updated (if applicable)

[arvind4501]

I wonder if these are the only tests which are slow, I was thinking if there is a more definative way to figure out slow tests and mark them accordingly? and seems like pytest has a way to figure that out https://docs.pytest.org/en/stable/how-to/usage.html#profiling-test-execution-duration

[ekohl]

We even have that enabled in our test playbook:

foremanctl/development/playbooks/test/test.yaml

Line 46 in 57966bd

cmd: "python -m pytest --durations=10 -vv {{ pytest_args | default() }}"

[ekohl]

Migration is probably #661, the other is failing on:

RUNNING HANDLER [iop_kafka : Restart kafka] ************************************
Error: : Task failed: Failed to connect to the host via ssh: ssh: connect to host 192.168.121.253 port 22: No route to host

I don't see how that could be caused by my changes.

[arvind4501]

I don't see how that could be caused by my changes.

It not i think, failure seems unrelated

[ehelms]

A note in our developer documentation about our use of slow and when to apply it, or at least how to think about when and when not to apply it would be helpful.

[ekohl]

I added some documentation, but wasn't quite sure about how to describe when to use the marker. I'm not sure if we need to provide that context now.

[ehelms]

I added some documentation, but wasn't quite sure about how to describe when to use the marker. I'm not sure if we need to provide that context now.

I was kinda hoping for that description. Ideally something like "tests longer than 1 minute can (or should) be marked as slow" for example.

I can see it both as being a good heuristic to have a "time" based component, but also a way for us to grab the list of slow tests for analysis sometime in the future.

This is not a hard requirement on my part, just thinking out loud.

[ekohl]

Slow is always relative. I wanted to avoid that.

[ehelms]

You'll need a rebase here to fix tests.

[ekohl]

I tried to be a bit more creative now in the docs. Also couldn't resist adding links to the section above it.