tbot.machine.linux

This module contains utilities for interacting with linux hosts. This includes:

• Linux Shells

• tbot’s own path type: tbot.machine.linux.Path

Linux Shells

The base-class tbot.machine.linux.LinuxShell defines the common interface all linux-shells should provide. This interface consists of the following methods:

• lnx.escape() - Escape args for the underlying shell.

• lnx.exec0() - Run command and ensure it succeeded.

• lnx.exec() - Run command and return output and return code.

• lnx.run() - Start a command and allow a testcase to interact with its stdio.

• lnx.test() - Run command and return boolean whether it succeeded.

• lnx.env() - Get/Set environment variables.

• lnx.subshell() - Start a subshell environment.

• lnx.interactive() - Start an interactive session for this machine.

And the following properties:

• lnx.username - Current user.

• lnx.fsroot - Path to the file-system root (just for convenience).

• lnx.workdir - Path to a directory which testcases can store files in.

tbot contains implementations of this interface for the following shells:

class tbot.machine.linux.Bash

Bases: tbot.machine.linux.linux_shell.LinuxShell

Bourne-again shell.

class tbot.machine.linux.LinuxShell

Bases: tbot.machine.shell.Shell

Base-class for linux shells.

This class defines the common interface for linux shells.

abstract escape(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → str

Escape a string according to this shell’s escaping rules.

If multiple arguments are given, .escape() returns a string containing each argument as a separate shell token. This means:

bash.escape("foo", "bar")
# foo bar

bash.escape("foo bar", "baz")
# "foo bar" baz

Parameters

*args – Arguments to be escaped. See Argument Types for details.

Returns

A string with quoted/escaped versions of the input arguments.

abstract exec(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → Tuple[int, str]

Run a command on this machine/shell.

Example:

retcode, output = mach.exec("uname", "-a")
assert retcode == 0

Parameters

*args – The command as separate arguments per command-line token. See Argument Types for more info.

Return type

tuple(int, str)

Returns

A tuple with the return code of the command and its console output. Note that the output is stdout and stderr merged. It will also contain a trailing newline in most cases.

abstract exec0(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → str

Run a command and assert its return code to be 0.

Example:

output = mach.exec0("uname", "-a")

# This will raise an exception!
mach.exec0("false")

Parameters

*args – The command as separate arguments per command-line token. See Argument Types for more info.

Return type

str

Returns

The command’s console output. Note that the output is stdout and stderr merged. It will also contain a trailing newline in most cases.

abstract test(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → bool

Run a command and return a boolean value whether it succeeded.

Example:

if lnx.test("which", "dropbear"):
    tbot.log.message("Dropbear is installed!")

Parameters

*args – The command as separate arguments per command-line token. See Argument Types for more info.

Return type

bool

Returns

Boolean representation of commands success. True if return code was 0, False otherwise.

abstract env(var: str, value: Optional[Union[str, tbot.machine.linux.path.Path[Self]]] = None) → str

Get or set an environment variable.

Example:

# Get the value of a var
value = lnx.env("PATH")

# Set the value of a var
lnx.env("DEBIAN_FRONTEND", "noninteractive")

Parameters

• var (str) – Environment variable name.

• value (tbot.machine.linux.Path,str) – Optional value to set the variable to.

Return type

str

Returns

Current (new) value of the environment variable.

run(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → ContextManager[tbot.machine.linux.util.RunCommandProxy]

Start an interactive command.

Interactive commands are started in a context-manager. Inside, direct interaction with the commands stdio is possible using a RunCommandProxy. You must call one of the terminate*() methods before leaving the context! The proxy object provides an interface similar to pexpect for inteaction (see the methods of the Channel class).

Example:

with lh.run("gdb", "-n", exe) as gdb:
    # Interact with gdb in this context
    # Wait for gdb to start up
    gdb.read_until_prompt("(gdb) ")

    # Better for automated interaction
    gdb.sendline("set confirm off")
    gdb.read_until_prompt("(gdb) ")

    # Necessary so output is not clobbered with escape sequences
    gdb.sendline("set style enabled off")
    gdb.read_until_prompt("(gdb) ")

    gdb.sendline("break main")
    gdb.read_until_prompt("(gdb) ")

    gdb.sendline("run")
    # We have hit the breakpoint
    gdb.read_until_prompt("(gdb) ")

    gdb.sendline("info locals", read_back=True)
    local_info = gdb.read_until_prompt("(gdb) ").strip()

    for line in local_info.split("\n"):
        var, val = line.split(" = ", 1)
        tbot.log.message(f"Local variable `{var}` has value `{val}`!")

    gdb.sendline("quit")
    gdb.terminate0()

abstract open_channel(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → tbot.machine.channel.channel.Channel

Transform this machine into a channel for something else by running a command.

This is meant to be used for tools like picocom which connect the terminal to a serial console.

Example:

ch = lnx.open_channel("picocom", "-b", "115200", "/dev/ttyUSB0")

# You can now interact with the channel for the serial console directly

Return type

tbot.machine.channel.Channel

abstract interactive() → None

Start an interactive session on this machine.

This method will connect tbot’s stdio to the machine’s channel so you can interactively run commands. This method is used by the interactive_lab and interactive_linux testcases.

abstract subshell(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → ContextManager[Self]

Start a subshell environment.

Sometimes you need to isolate certain tests into their own shell environment. This method returns a context manager which does this:

lnx.env("FOO", "bar")

with lnx.subshell():
    lnx.env("FOO", "baz")

assert lnx.env("FOO") == "bar"

You can also spawn a subshell with a custom command. This can be used, for example, to elevate privileges or switch user:

# Not root right now
assert int(lnx.env("EUID")) != 0

with lnx.subshell("sudo", "-ni", "bash", "--norc", "--noprofile"):
    # Root now!
    assert int(lnx.env("EUID")) == 0

Warning

tbot expects the shell inside the subshell environment to be the same shell as outside. This means, spawning a sudo environment which uses zsh instead of bash might lead to failures.

For bash, please spawn a bash --norc --noprofile for best compatibility.

For ash, an ash is good enough.

property username

Current username.

property fsroot

Path to the filesystem root of this machine, for convenience.

p = lnx.fsroot / "usr" / "lib"
assert p.is_dir()

property workdir

Path to a directory which testcases can use to store files in.

If configured properly, tbot will make sure this directory exists. Testcases should be able to deal with corrupt or missing files in this directory. Implementations should use tbot.machine.linux.Workdir.

Example:

# This is the defaut implementation
@property
def workdir(self):
    return linux.Workdir.xdg_data(self, "")

Argument Types

For a lot of methods defined in LinuxShell, a special set of types can be given as arguments. This protects against a lot of common errors. The allowed types are:

• str - Every string argument will be properly quoted so the shell picks it up as only one parameter. Example:

  lnx.exec0("echo", "Hello World!", "Is this tbot?")
  
  # Will run:
  #    echo "Hello World!" "Is this tbot?"
  

• tbot.machine.linux.Path - tbot’s path class keeps track of the machine the path is associated with. This prevents you from accidentally using a path from one host on another. The path class behaves like pathlib paths, so you can do things like:

  tftpdir = lnx.fsroot / "var" / "lib" / "tftpboot"
  lnx.exec0("ls", "-1", tftpdir)
  
  if (tftpdir / "u-boot.bin").is_file():
      tbot.log.message("Binary exists!")
  

• “Specials” - Sometimes you will need special shell-syntax for certain operations, for example to redirect output of a command. For these things, tbot provides special “tokens”. The full list can be found in Specials. As an example, redirecting to a file works like this:

  lnx.exec0("dmesg", linux.RedirStdout(lnx.workdir / "kernel.log"))
  # Will run:
  #    dmesg >/tmp/tbot-wd/kernel.log
  

Specials

Specials can be used as part of commands to use certain shell-syntaxes. This can be used to chain multiple commands or to redirect output.

tbot.machine.linux.AndThen

Chain commands using &&.

Example:

lnx.exec0("sleep", str(10), linux.AndThen, "echo", "Hello!")

tbot.machine.linux.OrElse

Chain commands using ||.

tbot.machine.linux.Then

Chain commands using ;.

tbot.machine.linux.Pipe

Pipe the output of one command into another:

lnx.exec0("dmesg", linux.Pipe, "grep", "usb")

class tbot.machine.linux.RedirStdout(file)

Redirect stdout (>...) to a file.

class tbot.machine.linux.RedirStderr(file)

Redirect stderr (2>...) to a file.

class tbot.machine.linux.RedirBoth(file)

Redirect both stdout and stderr (2>&1 >...) to a file.

tbot.machine.linux.Background

Tells the shell to run this command in the background (&).

By default all output from the command is suppressed (by redirection to /dev/null). If you do want to save command output, use the alternate form of Background:

f1 = lh.workdir / "stdout.txt"
f2 = lh.workdir / "stderr.txt"

# Just redirect stdout to a file, discard stderr
linux.Background(stdout=f1)
# Just redirect stderr to a file, discard stdout
linux.Background(stderr=f2)
# Redirect both to files
linux.Background(stdout=f1, stderr=f2)

Warning

Beware of the side-effects of running commands in the background! Never ever add linux.Background after a command that you expect will terminate at some point on its own! This will potentially clobber an unrelated command’s output which can have unexpected effects. Only use this special token with commands that will run forever until terminated manually. A good pattern to follow is this:

lh.exec0("some", "cmd", "that", "won't", "terminate", linux.Background)
pid = lh.env("!")

...

lh.exec0("kill", pid, linux.Then, "wait", pid)

class tbot.machine.linux.Raw(str)

Emits a raw string, bypassing shell quoting/escaping.

Warning

Only use this if nothing else works! linux.Raw can quickly lead to hard-to-find bugs.

RunCommandProxy

class tbot.machine.linux.RunCommandProxy(chan: tbot.machine.channel.channel.Channel, cmd_context: Callable[tbot.machine.linux.util.RunCommandProxy, Generator[str, None, Tuple[int, str]]])

Bases: tbot.machine.channel.channel.Channel

Proxy for interacting with a running command.

A RunCommandProxy is created with a context-manager and LinuxShell.run().

Example:

with lh.run("gdb", lh.workdir / "a.out") as gdb:
    gdb.sendline("target remote 127.0.0.1:3333")
    gdb.sendline("load")
    gdb.sendline("mon reset halt")

    gdb.sendline("quit")
    gbd.terminate0()

A RunCommandProxy has all methods of a Channel for interacting with the remote. Additionally, a few more methods exist which are necessary to end a command’s invokation properly. You must always call one of them before leaving the context-manager! These methods are:

terminate0() → str

Wait for the command to end successfully.

Asserts that the command returned with retcode 0. If it did not, an exception is raised.

Returns

Remaining output of the command until completion.

Return type

str

terminate() → Tuple[int, str]

Wait for the command to end.

Returns

A tuple of return code and remaining output.

Return type

tuple(int, str)

class tbot.machine.linux.CommandEndedException(match: Union[bytes, Match[bytes]] = b'')

Bases: tbot.machine.channel.channel.DeathStringException, tbot.machine.channel.channel.ChannelTakenException

The command which was run (interactively) ended prematurely.

This exception might be raised when reading from (or writing to) a RunCommandProxy and the remote command exited during the call. You can catch the exception but after receiving it, no more interaction with the command is allowed except the final terminate0() or terminate().

Example:

with lh.run("foo", "command") as foo:
    try:
        while True:
            foo.read_until_prompt("$ ")
            foo.sendline("echo some command")
    except linux.CommandEndedException:
        pass

    foo.terminate0()

Paths

class tbot.machine.linux.Path(host: H, *args: Any)

Bases: pathlib.PurePosixPath, Generic[typing.H]

A path that is associated with a tbot machine.

A path can only be used with its associated host. Using it with any other host will raise an exception and will be detected by a static typechecker.

Apart from that, Path behaves like a pathlib.Path:

from tbot.machine import linux

p = linux.Path(mach, "/foo/bar")
p2 = p / "bar" / "baz"
if not p2.exists():
    mach.exec0("mkdir", "-p", p2.parent)
    mach.exec0("touch", p2)
elif not p2.is_file():
    raise Exception(f"{p2} must be a normal file!")

Create a new path.

Parameters

• host (linux.LinuxShell) – Host this path should be associated with

• args – pathlib.PurePosixPath constructor arguments

property host

Host associated with this path.

stat() → os.stat_result

Return the result of stat on this path.

Tries to imitate the results of pathlib.Path.stat(), returns a os.stat_result.

exists() → bool

Whether this path exists.

is_dir() → bool

Whether this path points to a directory.

is_file() → bool

Whether this path points to a normal file.

is_symlink() → bool

Whether this path points to a symlink.

is_block_device() → bool

Whether this path points to a block device.

is_char_device() → bool

Whether this path points to a character device.

is_fifo() → bool

Whether this path points to a pipe(fifo).

is_socket() → bool

Whether this path points to a unix domain-socket.

property parent

Parent of this path.

glob(pattern: str) → Iterator[tbot.machine.linux.path.Path[H]]

Iterate over this subtree and yield all existing files (of any kind, including directories) matching the given relative pattern.

Example:

ubootdir = lh.workdir / "u-boot"

# .glob() returns a list which can be iterated.
for f in ubootdir.glob("common/*.c"):
    tbot.log.message(f"Found {f}.")

# To use the globs in another commandline (note the `*`!):
lh.exec0("ls", "-l", *ubootdir.glob("common/*.c"))

Warning

The glob pattern must not contain spaces or other special characters!

write_text(data: str, encoding: Optional[str] = None, errors: Optional[str] = None) → int

Write data into the file this path points to.

Example:

f = lnx.workdir / "foo.sh"

f.write_text('''\
#!/bin/sh
set -e

echo "Hello tbot!"
ps ax
''')

f.exec0("chmod", "+x", f)

Warning

This string must contain ‘text’ in the sense that some control characters are not allowed. Consult the documentation of LinuxShell.run() for details.

Additionally, line-endings might be transformed according to the tty’s settings. This function is not meant for byte-by-byte transfers, but for configuration files or small scripts. If you want to transfer a blob or a larger file, consider using tbot.tc.shell.copy() or (for small files) Path.write_bytes().

read_text(encoding: Optional[str] = None, errors: Optional[str] = None) → str

Read the contents of a text file, pointed to by this path.

Warning

This method is for ‘text’ content only as line-endings might not be transferred as contained in the file (Will always use a single \n). If you want to transfer a file byte-by-byte, consider using tbot.tc.shell.copy() instead. For small files, Path.write_bytes() might also be an option.

write_bytes(data: bytes) → int

Write binary data into the file this path points to.

Note

This method ensures exact byte-by-byte transfer. To do so, it encodes the data using base64 which makes console output less readable. If you intend to transfer text data, please use Path.write_text().

read_bytes() → bytes

Read the contents of a file, pointed to by this path.

Note

This method ensures exact byte-by-byte transfer. To do so, it encodes the data using base64 which makes console output less readable. If you intend to transfer text data, please use Path.read_text().

Workdir

class tbot.machine.linux.Workdir

classmethod static(host: H, pathstr: str) → tbot.machine.linux.workdir.Workdir[H]

Create a workdir in a static location, described by pathstr.

Example:

with tbot.acquire_lab() as lh:
    workdir = linux.Workdir.static(lh, "/tmp/tbot-my-workdir")

classmethod athome(host: H, subdir: str) → tbot.machine.linux.workdir.Workdir[H]

Create a workdir below the current users home directory.

Example:

with tbot.acquire_lab() as lh:
    # Use ~/tbot-foo-dir
    workdir = linux.Workdir.athome(lh, "tbot-foo-dir")

tbot will query the $HOME environment variable for the location of the current users home directory.

classmethod xdg_data(host: H, subdir: str) → tbot.machine.linux.workdir.Workdir[H]

Create a workdir in $XDG_DATA_HOME/tbot (usually ~/.local/share/tbot).

Example:

with tbot.acquire_lab() as lh:
    # Use ~/.local/share/tbot/foo-dir
    workdir = linux.Workdir.xdg_data(lh, "foo-dir")

classmethod xdg_runtime(host: H, subdir: str) → tbot.machine.linux.workdir.Workdir[H]

Create a workdir in $XDG_RUNTIME_DIR.

$XDG_RUNTIME_DIR is meant for non-essential runtime files and other file objects (such as sockets, named pipes, …) as specified by the XDG Base Directory Specification.

Example:

with tbot.acquire_lab() as lh:
    # Results in `$XDG_RUNTIME_DIR/tbot/tbot-pipes`
    workdir = linux.Workdir.xdg_runtime(lh, "tbot-pipes")

Lab-Host

class tbot.machine.linux.Lab

Bases: tbot.machine.linux.linux_shell.LinuxShell

Mixin for marking a machine as a lab-host.

build() → tbot.machine.linux.build.Builder

Return the default build-host for this lab.

If your lab does not contain a build-capable machine, just leave this method as is. tbot will raise an exception if a testcase attempts accessing the build-host anyway.

Builder

The Builder mixin allows marking a machine as a build-host. This means generic testcases like uboot.build can use it to automatically build projects. For this to work, a build-host needs to specify which toolchains it has installed and where tbot can find them.

class tbot.machine.linux.Builder

Bases: tbot.machine.linux.linux_shell.LinuxShell

Mixin to mark a machine as a build-host.

You need to define the toolchain() method when using this mixin. You can then use the enable() method to enable a toolchain and compile projects with it:

with MyBuildHost(lh) as bh:
    bh.exec0("uptime")

    with bh.enable("generic-armv7a-hf"):
        cc = bh.env("CC")
        bh.exec0(linux.Raw(cc), "main.c")

Note

If you look closely, I have used linux.Raw(cc) in the exec0() call. This is necessary because a lot of toolchains define $CC as something like

CC=arm-poky-linux-gnueabi-gcc -march=armv7-a -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a8

where some parameters are already included. Without the linux.Raw, tbot would run

$ "${CC}" main.c

where the arguments are interpreted as part of the path to the compiler. This will obviously fail so instead, with the linux.Raw, tbot will run

$ ${CC} main.c

where the shell expansion will do the right thing.

abstract property toolchains

Return a dictionary of all toolchains that exist on this buildhost.

Example:

@property
def toolchains(self) -> typing.Dict[str, linux.build.Toolchain]:
    return {
        "armv7a": linux.build.DistroToolchain("arm", "arm-linux-gnueabi-"),
        "armv8": linux.build.DistroToolchain("aarch64", "aarch64-linux-gnu-"),
        "mipsel": linux.build.DistroToolchain("mips", "mipsel-linux-gnu-"),
    }

enable(arch: str) → Iterator[None]

Enable the toolchain for arch on this BuildHost instance.

Example:

with lh.build() as bh:
    # Now we are on the buildhost

    with bh.enable("generic-armv7a-hf"):
        # Toolchain is enabled here
        cc = bh.env("CC")
        bh.exec0(linux.Raw(cc), "--version")

Toolchains

class tbot.machine.linux.build.Toolchain

Bases: abc.ABC

Generic toolchain type.

abstract enable(host: tbot.machine.linux.build.Builder) → None

Enable this toolchain on the given host.

class tbot.machine.linux.build.DistroToolchain(arch: str, prefix: str)

Bases: tbot.machine.linux.build.Toolchain

Toolchain that was installed from distribution repositories.

Example:

@property
def toolchains(self):
    return {
        "armv7a": linux.build.DistroToolchain("arm", "arm-linux-gnueabi-"),
        "armv8": linux.build.DistroToolchain("aarch64", "aarch64-linux-gnu-"),
    }

class tbot.machine.linux.build.EnvScriptToolchain(path: tbot.machine.linux.path.Path[H])

Bases: tbot.machine.linux.build.Toolchain

Toolchain that is initialized using an env script (e.g. yocto toolchain).

Example:

@property
def toolchains(self):
    yocto_root = self.fsroot / "opt" / "poky" / "2.6"

    return {
        "cortexa8hf": linux.build.EnvScriptToolchain(
            yocto_root / "environment-setup-cortexa8hf-neon-poky-linux-gnueabi",
        ),
    }

Create a new EnvScriptToolchain.

Parameters

path (linux.Path) – Path to the env script

Authenticators

For logging in via SSH using ParamikoConnector or SSHConnector, tbot provides the following ‘authenticators’:

class tbot.machine.linux.auth.NoneAuthenticator

Bases: tbot.machine.linux.auth.AuthenticatorBase

Most primitive authenticator.

Tries not passing any specific credentials and hopes ssh-config already contains all necessary infos. This is the default.

class tbot.machine.linux.auth.PrivateKeyAuthenticator(key_file: Union[str, pathlib.PurePath])

Bases: tbot.machine.linux.auth.AuthenticatorBase

Authenticate using a private-key file.

Example:

class MySSHHost(connector.SSHConnector, linux.Bash):
    username = "foouser"
    authenticator = linux.auth.PrivateKeyAuthenticator("/home/foo/.ssh/id_rsa_foo")

class tbot.machine.linux.auth.PasswordAuthenticator(password: str)

Bases: tbot.machine.linux.auth.AuthenticatorBase

Authenticate using a password.

Danger

This method is very insecure and might lead to PASSWORDS BEING STOLEN.

Example:

class MySSHHost(connector.SSHConnector, linux.Bash):
    username = "root"
    authenticator = linux.auth.PasswordAuthenticator("hunter2")