Quick Start

Commandline

First of all, install tbot. Instructions are here: Installation.

Let’s get started! To check if the installation went smoothly, try running tbot’s selftests:

bash-4.4$ tbot selftest
tbot starting ...
├─Calling selftest ...
│   ├─Calling testsuite ...
│   │   ├─Calling selftest_failing ...
│   │   │   ├─Calling inner ...
│   │   │   │   └─Fail. (0.000s)
│   │   │   └─Done. (0.001s)
│   │   ├─Calling selftest_uname ...
│   │   │   └─Done. (0.003s)
│   │   ├─Calling selftest_user ...
│   │   │   └─Done. (0.001s)
[...]
│   │   ├─Calling selftest_board_linux ...
│   │   │   ├─Skipped because no board available.
│   │   │   └─Done. (0.000s)
[...]
│   │   ├─Calling selftest_board_linux_bad_console ...
│   │   │   ├─POWERON (test)
│   │   │   ├─LINUX (test-linux)
│   │   │   ├─POWEROFF (test)
│   │   │   └─Done. (0.460s)
│   │   ├─────────────────────────────────────────
│   │   │ Success: 17/17 tests passed
│   │   └─Done. (4.675s)
│   └─Done. (4.742s)
├─────────────────────────────────────────
└─SUCCESS (4.845s)

If you feel adventurous, there are even more selftests that check if the built-in testcases work as intended:

bash-4.4$ tbot selftest_tc
tbot starting ...
├─Calling selftest_tc ...
│   ├─Calling testsuite ...
│   │   ├─Calling selftest_tc_git_checkout ...
│   │   │   ├─Calling git_prepare ...
│   │   │   │   ├─Setting up test repo ...
│   │   │   │   ├─Calling commit ...
│   │   │   │   │   └─Done. (0.009s)
│   │   │   │   ├─Creating test patch ...
│   │   │   │   ├─Calling commit ...
│   │   │   │   │   └─Done. (0.004s)
│   │   │   │   ├─Resetting repo ...
│   │   │   │   └─Done. (0.126s)
│   │   │   ├─Cloning repo ...
│   │   │   ├─Make repo dirty ...
│   │   │   ├─Add dirty commit ...
│   │   │   ├─Calling commit ...
│   │   │   │   └─Done. (0.004s)
│   │   │   └─Done. (0.321s)
[...]
│   │   ├─Calling selftest_tc_build_toolchain ...
│   │   │   ├─Creating dummy toolchain ...
│   │   │   ├─Attempt using it ...
│   │   │   └─Done. (0.153s)
│   │   ├─────────────────────────────────────────
│   │   │ Success: 6/6 tests passed
│   │   └─Done. (3.679s)
│   └─Done. (3.764s)
├─────────────────────────────────────────
└─SUCCESS (3.894s)

tbot also allows you to run multiple testcases at once:

bash-4.4$ tbot selftest selftest_tc
tbot starting ...
├─Calling selftest ...
│   ├─Calling testsuite ...
[...]
│   │   ├─────────────────────────────────────────
│   │   │ Success: 17/17 tests passed
│   │   └─Done. (4.788s)
│   └─Done. (4.873s)
├─Calling selftest_tc ...
│   ├─Calling testsuite ...
[...]
│   │   ├─────────────────────────────────────────
│   │   │ Success: 6/6 tests passed
│   │   └─Done. (3.390s)
│   └─Done. (3.459s)
├─────────────────────────────────────────
└─SUCCESS (8.453s)

If you want an overview of the available testcases, use this command:

$ tbot --list-testcases

The output you saw during the testcase runs was just a rough overview of what is going on. That might not be detailed enough for you. By adding -v, tbot will show all commands as they are executed. Add another one: -vv and you will also see command outputs!

bash-4.4$ tbot selftest_path_stat -vv
tbot starting ...
├─Calling selftest_path_stat ...
│   ├─Setting up test files ...
[...]
│   ├─[local] test -S /tmp/tbot-wd/nonexistent
│   ├─Checking stat results ...
│   ├─[local] stat -t /dev
│   │    ## /dev 4060 0 41ed 0 0 6 1025 20 0 0 1547723442 1547715500 1547715500 0 4096 system_u:object_r:device_t:s0
[...]
│   └─Done. (0.145s)
├─────────────────────────────────────────
└─SUCCESS (0.278s)

Note

There is one more verbosity level: -vvv. This is for debugging, if something doesn’t quite work. It shows you all communication happening, both directions. Try it if you want to, but be prepared: It will look quite messy!

One more commandline feature before we dive into python code: If you are afraid of a destructive command, you can run tbot with --interactive:

bash-4.4$ tbot selftest_uname -vi
tbot starting ...
├─Calling selftest_uname ...
│   ├─[local] uname -a
OK [Y/n]? Y
│   └─Done. (2.721s)
├─────────────────────────────────────────
└─SUCCESS (2.848s)

Now tbot will kindly ask you before running each command! (See? -emacs wouldn’t answer as nicely!)

Testcases

Ok, commandline isn’t all that fun. Let’s dive deeper! Some code please!

1
2
3
4
5
import tbot

@tbot.testcase
def hello_world():
    tbot.log.message("Hello World!")

This is tbot’s hello world. Stick this code into a file named tc.py. Now, if you check the list of testcases (tbot --list-testcases), hello_world pops up. Run it!

bash-4.4$ tbot hello_world
tbot starting ...
├─Calling hello_world ...
│   ├─Hello World!
│   └─Done. (0.000s)
├─────────────────────────────────────────
└─SUCCESS (0.127s)

Hello tbot!

Note

I am sure at least one person reading this will be offended by being told how to name their file. Why tc.py? I prefer calling it my_most_amazing_testcases.py!

Fear not, you can do just that! You just need to tell tbot about it. Instead of the above command, run:

$ tbot -t my_most_amazing_testcases.py hello_world

You can also include all python files in a directory with -T.

Well, before writing actual tests, I need to explain a few things: In tbot, testcases are basically python functions. This means you can call them just like python functions! From other testcases! How about the following?

1
2
3
4
5
6
7
8
9
import tbot

@tbot.testcase
def greet(name: str) -> None:
    tbot.log.message(f"Hello {name}!!")

@tbot.testcase
def greet_tbot() -> None:
    greet("tbot")

If you now call greet_tbot, you can see in the output that it calls greet.

But wait! If you try calling greet directly, it fails! Of course, because greet has a parameter. As previously mentioned, testcases are python functions, so naturally, they can also have parameters. There are two ways to “fix” this:

  1. Specifying a default value for the parameter:

    1
    2
    3
    4
    5
    import tbot
    
    @tbot.testcase
    def greet(name: str = "World") -> None:
        tbot.log.message(f"Hello {name}!!")
    
  2. Setting a value for the parameter! That’s right, you can set the parameter from the commandline. It looks like this:

    bash-4.4$ tbot greet -p name=\"tbot\"
    tbot starting ...
    ├─Parameters:
    │     name       = 'tbot'
    ├─Calling greet ...
    │   ├─Hello tbot!!
    │   └─Done. (0.000s)
    ├─────────────────────────────────────────
    └─SUCCESS (0.238s)

    Note the escaped quotes around \"tbot\". They are necessary because the value is eval()-uated internally. This is done to allow you to set values of any type with ease. Any python expression goes! (Also evil ones, be careful …)

As you’ll see later on, there are cases where you should have default values and ones where it doesn’t make sense. You’ll have to decide individually …

One more thing: You’d expect a testcase to somehow be able to show whether it succeeded. In tbot, a testcase that returns normally passes and one that raises an Exception has failed. This is pretty convenient: You can easily catch failures by using a try-block and your testcases will also automatically fail if anything unexpected happens.

Machines

Next up: Machines! Machines are what tbot is made for. Let’s take a look at the diagram from the landing page again:

_images/tbot1.svg

Lab-host? It’s a machine! Buildhost? Just as well! The boards you are testing? You guessed it!

Let’s start simple though: Just run a command on the lab-host:

1
2
3
4
5
6
7
8
import tbot

@tbot.testcase
def greet_user() -> None:
    with tbot.acquire_lab() as lh:
        name = lh.exec0("id", "--user", "--name").strip()

        tbot.log.message(f"Hello {name}!")

Now try:

bash-4.4$ tbot greet_user -v
tbot starting ...
├─Calling greet_user ...
│   ├─[local] id --user --name
│   ├─Hello hws!
│   └─Done. (0.070s)
├─────────────────────────────────────────
└─SUCCESS (0.173s)

As you can see, tbot ran id --user --name to find your name. You might be curious about the [local] part: That’s the machine tbot ran the command on. By default, the lab-host is your localhost. We’ll see later how to change that.

There are quite a few new things in the sample above. Let’s go through them one by one:

  • tbot.acquire_lab(): This is a function provided by tbot that returns the selected lab-host.

  • with tbot.acquire_lab() as lh:: Each machine is a context manager. To get access, you need to enter its context and as soon as you leave it the connection is destroyed. If you haven’t heard about context managers before, take a look at Python with Context Managers. They are really useful!

  • lh.exec0(): This is a function to run a command. Specifically exec-utes it and checks whether the return value is 0. There are also others, for example, lh.test() which returns True if the command succeeded and False otherwise.

  • All command executing methods take one parameter per commandline argument. Each one will be properly escaped: lh.exec0("echo", "!?#;>&<") would print !?#;>&<, no manual quoting needed!

  • lh.exec0() returns a string which I call .strip() on. The reason is that most commands include a trailing newline (\n). I don’t want that in the name so I remove it.

To learn more about the methods tbot provides for interacting with linux-machines, take a look at the docs for LinuxShell.

One more feature I want to mention in this quick guide: Most machines have an interactive() method. This method will connect the channel to the terminal and allows you to directly enter commands. You can use it to make tbot do some work, then do something manually. Like a symbiotic development process. It really makes you a lot more productive if you embrace this idea! There is also a testcase to call it from the commandline:

bash-4.4$ tbot interactive_lab
tbot starting ...
├─Calling interactive_lab ...
│   ├─Entering interactive shell ...

local: /tmp> whoami
hws
local: /tmp> exit

│   ├─Exiting interactive shell ...
│   └─Done. (49.746s)
├─────────────────────────────────────────
└─SUCCESS (49.851s)

Configuration

Up until now we did everything on our localhost. That’s boring! tbot allows you to easily use a lab-host that you can connect to via SSH for example. To do that you have to write a small config file. There’s a twist though! The config file is actually a python module. In this module, you create a class for your lab-host. If you have some special features on your lab-host you can add them in there just as well!

The simplest config (for a lab-host connected via SSH) looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import tbot
from tbot.machine import connector, linux

class AwesomeLab(
    connector.ParamikoConnector,
    linux.Bash,
    linux.Lab,
):
    name = "awesome-lab"
    hostname = "awesome.lab.com"

    @property
    def workdir(self):
        return linux.Workdir.athome(self, "tbot-workdir")


# tbot will check for `LAB`, don't forget to set it!
LAB = AwesomeLab

Of course, you’ll have to adjust this a little. tbot will try to connect to the host hostname. It will query ~/.ssh/config for a username and key. (You need to be able to connect to hostname without a password!)

Try using your config now!

$ tbot -l <name-of-lab-config>.py interactive_lab

Congratulations! You now have a remote session on your lab-host! You could also run some selftest to verify that tbot can run these commands on your new lab-host as well:

bash-4.4$ tbot -l lab.py selftest_path_integrity -vv
tbot starting ...
├─Calling selftest_path_integrity ...
│   ├─Logging in on hws@78.79.32.85:22 ...
│   ├─[awesome-lab] echo ${HOME}
│   │    ## /home/hws
│   ├─[awesome-lab] test -d /home/hws/tbot-workdir
│   ├─[awesome-lab] mkdir -p /home/hws/tbot-workdir
│   ├─Logging in on hws@78.79.32.85:22 ...
│   ├─[awesome-lab] mkdir -p /home/hws/tbot-workdir/folder
│   ├─[awesome-lab] test -d /home/hws/tbot-workdir/folder
│   ├─[awesome-lab] uname -a >/home/hws/tbot-workdir/folder/file.txt
│   ├─[awesome-lab] test -f /home/hws/tbot-workdir/folder/file.txt
│   ├─[awesome-lab] rm -r /home/hws/tbot-workdir/folder
│   ├─[awesome-lab] test -e /home/hws/tbot-workdir/folder/file.txt
│   ├─[awesome-lab] test -e /home/hws/tbot-workdir/folder
│   └─Done. (2.833s)
├─────────────────────────────────────────
└─SUCCESS (2.959s)

As you can see, now it says [awesome-lab] in front of the commands. tbot is running commands remotely!

This was just a simple example … Configs can get a lot bigger and a lot more complex. Take a look at the Configuration documentation for more info!

Machine-Classes

When configuring the lab-host we already saw the definition of a machine class, but up to now I did not really explain how those actually work. Before we can dive into the next chapter, I have to explain a bit about this:

tbot machines are classes which inherit from multiple components. This allows easy mix and matching to flexibly configure the machine for your needs. There are two main components which every machine needs and a number of mixins which allow further customization. First the big ones:

  1. Connectors define how a connection to the respective machine can be established. The easiest way is the SubprocessConnector which just spawns a shell as a subprocess. More complex examples include the ParamikoConnector which we saw above, or the ConsoleConnector. For more in-depth documentation of the connectors, take a look at the tbot.machine.connector module.

  2. Shells define the API for interacting with the machine. This varies quite drastically between the different machine-types as shells behave differently. Think how the U-Boot environment works completely different than the environment in Linux. The lab-host config above used the Bash shell and we will see a UBootShell in the next chapter. For further details, go to the tbot.machine.shell module.

Between those two, sometimes you need a third part, a so-called Initializer. An example for a situation where one is needed would be this: After opening the serial connection to your board, you want to wait for the login-prompt first and enter your credentials before the shell is available. For this, tbot provides a LinuxBootLogin initializer. If you have multiple initializers in a machine-class, you need to keep in mind in which order they should run.

As a final part, there are some mixins for certain uses. For example the Lab mixin which marks a machine as a lab-host or the Builder mixin which marks a machine as a build-host. These can be added whenever appropriate.

More details about machine-classes can be found in the tbot.machine module.

Hardware Interaction

We haven’t even talked to actual hardware yet! Let’s change that. Unfortunately, as each device is different, you’ll have to figure out a few things yourself.

First Step: Another config file. The board needs to be configured in a second file. Let’s start simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
import tbot
from tbot.machine import board, channel, connector, linux

class SomeBoard(
    connector.ConsoleConnector,
    board.PowerControl,
    board.Board,
):
    name = "some-board"

    def poweron(self) -> None:
        """Procedure to turn power on."""

        # You can access the labhost as `self.host` (if you use the
        # ConsoleConnector).  In this case I have a simple command to
        # toggle power.
        self.host.exec0("remote_power", "bbb", "on")

        # If you can't automatically toggle power,
        # you have to insert some marker here that reminds you
        # to manually toggle power.  How about:
        tbot.log.message("Turn power on now!")

    def poweroff(self) -> None:
        """Procedure to turn power off."""
        self.host.exec0("remote_power", "bbb", "off")

    def connect(self, mach) -> channel.Channel:
        """Connect to the boards serial interface."""

        # `mach.open_channel` 'steals' mach's channel and runs the
        # given command to connect to the serial console.  Your command
        # should just connect its tty to the serial console like rlogin,
        # telnet, picocom or kermit do.  The minicom behavior will not work.
        return mach.open_channel("picocom", "-b", "115200", "bbb")

# tbot will check for `BOARD`, don't forget to set it!
BOARD = SomeBoard

If you did everything correctly, this should be enough to get a serial connection running. Try this:

$ tbot -l lab.py -b my-board.py interactive_board -vv

You should see the board starting to boot. If not, go back and check manually if the commands by themselves work. You might also want to look at the Board Config documentation.

Next up we will add config for the Linux running on the board (in the same file for now). I’ll skip U-Boot in this quick guide for simplicity. Here’s the full new config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import tbot
from tbot.machine import board, connector, channel, linux

class SomeBoard(
    connector.ConsoleConnector,
    board.PowerControl,
    board.Board,
):
    name = "some-board"

    def poweron(self) -> None:
        self.host.exec0("remote_power", "bbb", "on")

    def poweroff(self) -> None:
        self.host.exec0("remote_power", "bbb", "off")

    def connect(self, mach) -> channel.Channel:
        return mach.open_channel("picocom", "-b", "115200", "bbb")

# Linux machine
#
# We use a config which boots directly to Linux without interaction
# with a bootloader for this example.
class SomeBoardLinux(
    board.Connector,
    board.LinuxBootLogin,
    linux.Bash,
):
    # Username for logging in once linux has booted
    username = "root"
    # Password.  If you don't need a password, set this to `None`
    password = "~ysu0dbi"

BOARD = SomeBoard
# You need to set `LINUX` now as well.
LINUX = SomeBoardLinux

Again, adjust it as necessary. If you are unsure about some parameters, you can check in the interactive_board session. To learn more about the individual parameters, look at the Board Config and Linux (without U-Boot) Config docs.

If you set everything correctly, you should be able to run:

$ tbot -l lab.py -b my-board.py interactive_linux -vv

You now have a shell on the board! As before, you can also try running a selftest:

$ tbot -l lab.py -b my-board.py selftest_board_linux -vv
bash-4.4$ tbot -l lab.py -b my-board.py selftest_board_linux -vv
tbot starting ...
├─Calling selftest_board_linux ...
│   ├─Logging in on hws@78.79.32.85:22 ...
│   ├─[awesome-lab] connect bbb
│   ├─[awesome-lab] remote_power bbb -l
│   │    ## bbb             off
│   ├─POWERON (bbb)
│   ├─[awesome-lab] remote_power bbb on
│   │    ## Power on   bbb: OK
│   ├─UBOOT (bbb-uboot)
│   │    <>
│   │    <> U-Boot 2018.11-00191-gd73d81fd85 (Nov 20 2018 - 06:01:01 +0100)
│   │    <>
│   │    <> CPU  : AM335X-GP rev 2.1
│   │    <> Model: TI AM335x BeagleBone Black
│   │    <> DRAM:  512 MiB
│   │    <> NAND:  0 MiB
│   │    <> MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
│   │    <> Loading Environment from FAT... ** No partition table - mmc 0 **
│   │    <> No USB device found
│   │    <> <ethaddr> not set. Validating first E-fuse MAC
│   │    <> Net:   eth0: ethernet@4a100000
│   ├─LINUX (bbb-linux)
│   ├─[bbb-uboot] setenv serverip 192.168.1.1
│   ├─[bbb-uboot] setenv netmask 255.255.255.0
│   ├─[bbb-uboot] setenv ipaddr 192.168.1.10
│   ├─[bbb-uboot] mw 0x81000000 0 0x4000
│   ├─[bbb-uboot] setenv rootpath /opt/core-image-lsb-sdk-generic-armv7a-hf
│   ├─[bbb-uboot] run netnfsboot
│   │    <> Booting from network ... with nfsargs ...
│   │    <> link up on port 0, speed 100, full duplex
│   │    <> TFTP from server 192.168.1.1; our IP address is 192.168.1.10
│   │    <> Load address: 0x82000000
│   │    <> Loading: #################################################################
│   │    <>    ########################
│   │    <>    2.9 MiB/s
│   │    <> done
│   │    <> Bytes transferred = 9883000 (96cd78 hex)
│   │    <> link up on port 0, speed 100, full duplex
│   │    <> TFTP from server 192.168.1.1; our IP address is 192.168.1.10
│   │    <> Load address: 0x88000000
│   │    <> Loading: #####
│   │    <>    1.1 MiB/s
│   │    <> done
│   │    <> Bytes transferred = 64051 (fa33 hex)
│   │    <> ## Flattened Device Tree blob at 88000000
│   │    <>    Booting using the fdt blob at 0x88000000
│   │    <>    Loading Device Tree to 8ffed000, end 8ffffa32 ... OK
│   │    <>
│   │    <> Starting kernel ...
│   │    <>
│   │    <> [    0.000000] Booting Linux on physical CPU 0x0
│   │    <> [    0.000000] Linux version 4.9.126 (build@denx) (gcc version 7.2.1 20171011 (Linaro GCC 7.2-2017.11) ) #1 SMP PREEMPT Wed Dec 12 03:12:29 CET 2018
│   │    <> [    0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
│   │    <> [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache                                              Hello there ;)
│   │    <> [    0.000000] OF: fdt:Machine model: TI AM335x BeagleBone Black
│   │    <> [    0.000000] efi: Getting EFI parameters from FDT:
│   │    <> [    0.000000] efi: UEFI not found.
│   │    <> [    0.000000] cma: Reserved 48 MiB at 0x9c800000
[...]
│   │    <> Poky (Yocto Project Reference Distro) 2.4 generic-armv7a-hf /dev/ttyS0
│   │    <>
│   │    <> generic-armv7a-hf login: root
│   ├─Calling selftest_machine_shell ...
│   │   ├─Testing command output ...
│   │   ├─[bbb-linux] echo 'Hello World'
│   │   │    ## Hello World
│   │   ├─[bbb-linux] echo '$?' '!#'
│   │   │    ## $? !#
[...]
│   │   └─Done. (3.355s)
│   ├─POWEROFF (bbb)
│   ├─[pollux] remote_power bbb off
│   │    ## Power off  bbb: OK
│   └─Done. (44.150s)
├─────────────────────────────────────────
└─SUCCESS (44.624s)

Hardware Use from Tests

Last part of this guide will be interacting with the board from a testcase. It’s pretty straight forward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
import tbot

@tbot.testcase
def test_board() -> None:
    # Get access to the lab-host as before
    with tbot.acquire_lab() as lh:
        # This context is for the "hardware".  Once you enter
        # it, the board will be powered on and as soon as
        # you exit it, it will be turned off again.
        with tbot.acquire_board(lh) as b:
            # This is the context for the "Linux machine".
            # Entering it means tbot will listen to the
            # board booting and give you a machine handle
            # as soon as the shell is available.
            with tbot.acquire_linux(b) as lnx:
                lnx.exec0("uname", "-a")

Those two additional indentation levels aren’t nice - We can refactor the code to look like this (I showed the explicit version first so you can see what is going on):

1
2
3
4
5
6
7
8
import tbot

@tbot.testcase
def test_board() -> None:
    with  tbot.acquire_lab() as lh,
          tbot.acquire_board(lh) as b,
          tbot.acquire_linux(b) as lnx:
        lnx.exec0("uname", "-a")

There is still one issue with this design: Let’s pretend this is a test to check some board functionality. Maybe you have quite a few testcases that each check different parts. Now, we want to call all of them from some “master” test, so we can test everything at once.

The issue we will run into is that each testcase will A) reconnect to the lab-host and B) powercycle the board. This will be very very slow! We can do better!

The idea is that testcases take the lab and board as optional parameters. This allows reusing the old connection and won’t powercycle the board for each test (if you need powercycling, you can of course do it like above). To make this as easy as possible, tbot provides the with_linux() decorator. You can use it like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import typing
import tbot
from tbot.machine import linux, board

@tbot.testcase
@tbot.with_linux
def test_board(
    lnx: linux.LinuxShell,
    param: str = "-a",
) -> None:
    lnx.exec0("uname", param)

@tbot.testcase
def call_it() -> None:
    with tbot.acquire_lab() as lh:
        test_board(lh, "-r")

@tbot.testcase
def call_it_prepared() -> None:
    with tbot.acquire_lab() as lh,
         tbot.acquire_board(lh) as b,
         tbot.acquire_linux(b) as lnx:
        test_board(lnx, "-n")

You can still call test_board from the commandline, but call_it and call_it_prepared work as well!

There is also with_lab() and with_uboot() for those two usecases.

That’s it for the quick-start guide. If you want to dive deeper, you might want to follow these links: