`tbot.machine.linux`¶

This module contains utilities for interacting with linux hosts. This includes:

Linux Shells
tbot’s own path type: tbot.machine.linux.Path

Linux Shells¶

The base-class tbot.machine.linux.LinuxShell defines the common interface all linux-shells should provide. This interface consists of the following methods:

lnx.escape() - Escape args for the underlying shell.
lnx.exec0() - Run command and ensure it succeeded.
lnx.exec() - Run command and return output and return code.
lnx.run() - Start a command and allow a testcase to interact with its stdio.
lnx.test() - Run command and return boolean whether it succeeded.
lnx.env() - Get/Set environment variables.
lnx.subshell() - Start a subshell environment.
lnx.interactive() - Start an interactive session for this machine.

And the following properties:

lnx.username - Current user.
lnx.fsroot - Path to the file-system root (just for convenience).
lnx.workdir - Path to a directory which testcases can store files in.

tbot contains implementations of this interface for the following shells:

class tbot.machine.linux.Bash[source]¶

Bases: tbot.machine.linux.linux_shell.LinuxShell

Bourne-again shell.

class tbot.machine.linux.LinuxShell[source]¶

Bases: tbot.machine.shell.Shell

Base-class for linux shells.

This class defines the common interface for linux shells.

abstract escape(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → str[source]¶

Escape a string according to this shell’s escaping rules.

If multiple arguments are given, .escape() returns a string containing each argument as a separate shell token. This means:

bash.escape("foo", "bar")
# foo bar

bash.escape("foo bar", "baz")
# "foo bar" baz

Parameters: *args – Arguments to be escaped. See Argument Types for details.
Returns: A string with quoted/escaped versions of the input arguments.

abstract exec(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → Tuple[int, str][source]¶

Run a command on this machine/shell.

Example:

retcode, output = mach.exec("uname", "-a")
assert retcode == 0

Parameters: *args – The command as separate arguments per command-line token. See Argument Types for more info.
Return type: tuple(int, str)
Returns: A tuple with the return code of the command and its console output. Note that the output is stdout and stderr merged. It will also contain a trailing newline in most cases.

abstract exec0(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → str[source]¶

Run a command and assert its return code to be 0.

Example:

output = mach.exec0("uname", "-a")

# This will raise an exception!
mach.exec0("false")

Parameters: *args – The command as separate arguments per command-line token. See Argument Types for more info.
Return type: str
Returns: The command’s console output. Note that the output is stdout and stderr merged. It will also contain a trailing newline in most cases.

abstract test(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → bool[source]¶

Run a command and return a boolean value whether it succeeded.

Example:

if lnx.test("which", "dropbear"):
    tbot.log.message("Dropbear is installed!")

Parameters: *args – The command as separate arguments per command-line token. See Argument Types for more info.
Return type: bool
Returns: Boolean representation of commands success. True if return code was 0, False otherwise.

abstract env(var: str, value: Optional[Union[str, tbot.machine.linux.path.Path[Self]]] = None) → str[source]¶

Get or set an environment variable.

Example:

# Get the value of a var
value = lnx.env("PATH")

# Set the value of a var
lnx.env("DEBIAN_FRONTEND", "noninteractive")

Parameters

var (str) – Environment variable name.
value (tbot.machine.linux.Path,str) – Optional value to set the variable to.

Return type

str

Returns

Current (new) value of the environment variable.

run(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → ContextManager[tbot.machine.linux.util.RunCommandProxy][source]¶

Start an interactive command.

Interactive commands are started in a context-manager. Inside, direct interaction with the commands stdio is possible using a RunCommandProxy. You must call one of the terminate*() methods before leaving the context! The proxy object provides an interface similar to pexpect for inteaction (see the methods of the Channel class).

Example:

with lh.run("gdb", "-n", exe) as gdb:
    # Interact with gdb in this context
    # Wait for gdb to start up
    gdb.read_until_prompt("(gdb) ")

    # Better for automated interaction
    gdb.sendline("set confirm off")
    gdb.read_until_prompt("(gdb) ")

    # Necessary so output is not clobbered with escape sequences
    gdb.sendline("set style enabled off")
    gdb.read_until_prompt("(gdb) ")

    gdb.sendline("break main")
    gdb.read_until_prompt("(gdb) ")

    gdb.sendline("run")
    # We have hit the breakpoint
    gdb.read_until_prompt("(gdb) ")

    gdb.sendline("info locals", read_back=True)
    local_info = gdb.read_until_prompt("(gdb) ").strip()

    for line in local_info.split("\n"):
        var, val = line.split(" = ", 1)
        tbot.log.message(f"Local variable `{var}` has value `{val}`!")

    gdb.sendline("quit")
    gdb.terminate0()

abstract open_channel(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → tbot.machine.channel.channel.Channel[source]¶

Transform this machine into a channel for something else by running a command.

This is meant to be used for tools like picocom which connect the terminal to a serial console.

Example:

ch = lnx.open_channel("picocom", "-b", "115200", "/dev/ttyUSB0")

# You can now interact with the channel for the serial console directly

Return type: tbot.machine.channel.Channel

abstract interactive() → None[source]¶

Start an interactive session on this machine.

This method will connect tbot’s stdio to the machine’s channel so you can interactively run commands. This method is used by the interactive_lab and interactive_linux testcases.

abstract subshell(*args: Union[str, tbot.machine.linux.special.Special[Self], tbot.machine.linux.path.Path[Self]]) → ContextManager[Self][source]¶

Start a subshell environment.

Sometimes you need to isolate certain tests into their own shell environment. This method returns a context manager which does this:

lnx.env("FOO", "bar")

with lnx.subshell():
    lnx.env("FOO", "baz")

assert lnx.env("FOO") == "bar"

You can also spawn a subshell with a custom command. This can be used, for example, to elevate privileges or switch user:

# Not root right now
assert int(lnx.env("EUID")) != 0

with lnx.subshell("sudo", "-ni", "bash", "--norc", "--noprofile"):
    # Root now!
    assert int(lnx.env("EUID")) == 0

Warning

tbot expects the shell inside the subshell environment to be the same shell as outside. This means, spawning a sudo environment which uses zsh instead of bash might lead to failures.

For bash, please spawn a bash --norc --noprofile for best compatibility.

For ash, an ash is good enough.

property username: str¶: Current username.

property fsroot: tbot.machine.linux.path.Path[Self]¶

Path to the filesystem root of this machine, for convenience.

p = lnx.fsroot / "usr" / "lib"
assert p.is_dir()

property workdir: tbot.machine.linux.path.Path[Self]¶

Path to a directory which testcases can use to store files in.

If configured properly, tbot will make sure this directory exists. Testcases should be able to deal with corrupt or missing files in this directory. Implementations should use tbot.machine.linux.Workdir.

Example:

# This is the defaut implementation
@property
def workdir(self):
    return linux.Workdir.xdg_data(self, "")

Argument Types¶

For a lot of methods defined in LinuxShell, a special set of types can be given as arguments. This protects against a lot of common errors. The allowed types are:

str - Every string argument will be properly quoted so the shell picks it up as only one parameter. Example:

lnx.exec0("echo", "Hello World!", "Is this tbot?")

# Will run:
#    echo "Hello World!" "Is this tbot?"

tbot.machine.linux.Path - tbot’s path class keeps track of the machine the path is associated with. This prevents you from accidentally using a path from one host on another. The path class behaves like pathlib paths, so you can do things like:
```
tftpdir = lnx.fsroot / "var" / "lib" / "tftpboot"
lnx.exec0("ls", "-1", tftpdir)

if (tftpdir / "u-boot.bin").is_file():
    tbot.log.message("Binary exists!")
```
“Specials” - Sometimes you will need special shell-syntax for certain operations, for example to redirect output of a command. For these things, tbot provides special “tokens”. The full list can be found in Specials. As an example, redirecting to a file works like this:
```
lnx.exec0("dmesg", linux.RedirStdout(lnx.workdir / "kernel.log"))
# Will run:
#    dmesg >/tmp/tbot-wd/kernel.log
```

Specials¶

Specials can be used as part of commands to use certain shell-syntaxes. This can be used to chain multiple commands or to redirect output.

tbot.machine.linux.AndThen¶

Chain commands using &&.

Example:

lnx.exec0("sleep", str(10), linux.AndThen, "echo", "Hello!")

tbot.machine.linux.OrElse¶: Chain commands using ||.

tbot.machine.linux.Then¶: Chain commands using ;.

tbot.machine.linux.Pipe¶

Pipe the output of one command into another:

lnx.exec0("dmesg", linux.Pipe, "grep", "usb")

class tbot.machine.linux.RedirStdout(file)[source]¶: Redirect stdout (>...) to a file.

class tbot.machine.linux.RedirStderr(file)[source]¶: Redirect stderr (2>...) to a file.

class tbot.machine.linux.RedirBoth(file)[source]¶: Redirect both stdout and stderr (>... 2>&1) to a file.

class tbot.machine.linux.AppendStdout(file)[source]¶: Redirect stdout (>>...) to a file, appending to the file rather than overwriting its contents.

New in version 0.9.1.

class tbot.machine.linux.AppendStderr(file)[source]¶: Redirect stderr (2>>...) to a file, appending to the file rather than overwriting its contents.

New in version 0.9.1.

class tbot.machine.linux.AppendBoth(file)[source]¶: Redirect both stdout and stderr (>>... 2>&1) to a file, appending to the file rather than overwriting its contents.

New in version 0.9.1.

tbot.machine.linux.Background¶

Tells the shell to run this command in the background (&).

By default all output from the command is suppressed (by redirection to /dev/null). If you do want to save command output, use the alternate form of Background:

f1 = lh.workdir / "stdout.txt"
f2 = lh.workdir / "stderr.txt"

# Just redirect stdout to a file, discard stderr
linux.Background(stdout=f1)
# Just redirect stderr to a file, discard stdout
linux.Background(stderr=f2)
# Redirect both to files
linux.Background(stdout=f1, stderr=f2)

Warning

Beware of the side-effects of running commands in the background! Never ever add linux.Background after a command that you expect will terminate at some point on its own! This will potentially clobber an unrelated command’s output which can have unexpected effects. Only use this special token with commands that will run forever until terminated manually. A good pattern to follow is this:

lh.exec0("some", "cmd", "that", "won't", "terminate", linux.Background)
pid = lh.env("!")

...

lh.exec0("kill", pid, linux.Then, "wait", pid)

Changed in version 0.9.1: Background now always properly redirects stderr as well.

class tbot.machine.linux.Raw(str)[source]¶: Emits a raw string, bypassing shell quoting/escaping.

Warning

Only use this if nothing else works! linux.Raw can quickly lead to hard-to-find bugs.

RunCommandProxy¶

class tbot.machine.linux.RunCommandProxy(chan: tbot.machine.channel.channel.Channel, cmd_context: Callable[tbot.machine.linux.util.RunCommandProxy, Generator[str, None, Tuple[int, str]]])[source]¶

Bases: tbot.machine.channel.channel.Channel

Proxy for interacting with a running command.

A RunCommandProxy is created with a context-manager and LinuxShell.run().

Example:

with lh.run("gdb", lh.workdir / "a.out") as gdb:
    gdb.sendline("target remote 127.0.0.1:3333")
    gdb.sendline("load")
    gdb.sendline("mon reset halt")

    gdb.sendline("quit")
    gbd.terminate0()

A RunCommandProxy has all methods of a Channel for interacting with the remote. Additionally, a few more methods exist which are necessary to end a command’s invokation properly. You must always call one of them before leaving the context-manager! These methods are:

terminate0() → str[source]¶

Wait for the command to end successfully.

Asserts that the command returned with retcode 0. If it did not, an exception is raised.

Returns: Remaining output of the command until completion.
Return type: str

terminate() → Tuple[int, str][source]¶

Wait for the command to end.

Returns: A tuple of return code and remaining output.
Return type: tuple(int, str)

class tbot.machine.linux.CommandEndedException(match: Union[bytes, Match[bytes]] = b'')[source]¶

Bases: tbot.machine.channel.channel.DeathStringException, tbot.machine.channel.channel.ChannelTakenException

The command which was run (interactively) ended prematurely.

This exception might be raised when reading from (or writing to) a RunCommandProxy and the remote command exited during the call. You can catch the exception but after receiving it, no more interaction with the command is allowed except the final terminate0() or terminate().

Example:

with lh.run("foo", "command") as foo:
    try:
        while True:
            foo.read_until_prompt("$ ")
            foo.sendline("echo some command")
    except linux.CommandEndedException:
        pass

    foo.terminate0()

Paths¶

class tbot.machine.linux.Path(host: H, *args: Any)[source]¶

Bases: Generic[H]

A path that is associated with a tbot machine.

A path can only be used with its associated host. Using it with any other host will raise an exception and will be detected by a static typechecker.

Apart from that, Path behaves like a pathlib.Path:

from tbot.machine import linux

p = linux.Path(mach, "/foo/bar")
p2 = p / "bar" / "baz"
if not p2.exists():
    mach.exec0("mkdir", "-p", p2.parent)
    mach.exec0("touch", p2)
elif not p2.is_file():
    raise Exception(f"{p2} must be a normal file!")

Create a new path.

Parameters

host (linux.LinuxShell) – Host this path should be associated with
args – pathlib.PurePosixPath constructor arguments

Changed in version 0.9.1: Path no longer inherits from pathlib.PurePosixPath.

property host: H¶: Host associated with this path.

at_host(host: H) → str[source]¶

Convert this Path into a string representation assuming it should be valid for the machine host. An exception is raised if this Path is for a different machine instead.

This prevents accidentally using a Path with the wrong host as much as possible.

New in version 0.9.1.

stat() → os.stat_result[source]¶

Return the result of stat on this path.

Tries to imitate the results of pathlib.Path.stat(), returns a os.stat_result.

exists() → bool[source]¶: Whether this path exists.

is_dir() → bool[source]¶: Whether this path points to a directory.

is_file() → bool[source]¶: Whether this path points to a normal file.

is_symlink() → bool[source]¶: Whether this path points to a symlink.

is_block_device() → bool[source]¶: Whether this path points to a block device.

is_char_device() → bool[source]¶: Whether this path points to a character device.

is_fifo() → bool[source]¶: Whether this path points to a pipe(fifo).

is_socket() → bool[source]¶: Whether this path points to a unix domain-socket.

glob(pattern: str) → Iterator[tbot.machine.linux.path.Path[H]][source]¶

Iterate over this subtree and yield all existing files (of any kind, including directories) matching the given relative pattern.

Example:

ubootdir = lh.workdir / "u-boot"

# .glob() returns a list which can be iterated.
for f in ubootdir.glob("common/*.c"):
    tbot.log.message(f"Found {f}.")

# To use the globs in another commandline (note the `*`!):
lh.exec0("ls", "-l", *ubootdir.glob("common/*.c"))

Warning

The glob pattern must not contain spaces or other special characters!

write_text(data: str, encoding: Optional[str] = None, errors: Optional[str] = None) → int[source]¶

Write data into the file this path points to.

Example:

f = lnx.workdir / "foo.sh"

f.write_text('''\
#!/bin/sh
set -e

echo "Hello tbot!"
ps ax
''')

f.exec0("chmod", "+x", f)

Warning

This string must contain ‘text’ in the sense that some control characters are not allowed. Consult the documentation of LinuxShell.run() for details.

Additionally, line-endings might be transformed according to the tty’s settings. This function is not meant for byte-by-byte transfers, but for configuration files or small scripts. If you want to transfer a blob or a larger file, consider using tbot.tc.shell.copy() or (for small files) Path.write_bytes().

read_text(encoding: Optional[str] = None, errors: Optional[str] = None) → str[source]¶: Read the contents of a text file, pointed to by this path.

Warning

This method is for ‘text’ content only as line-endings might not be transferred as contained in the file (Will always use a single \n). If you want to transfer a file byte-by-byte, consider using tbot.tc.shell.copy() instead. For small files, Path.write_bytes() might also be an option.

write_bytes(data: bytes) → int[source]¶: Write binary data into the file this path points to.

Note

This method ensures exact byte-by-byte transfer. To do so, it encodes the data using base64 which makes console output less readable. If you intend to transfer text data, please use Path.write_text().

read_bytes() → bytes[source]¶: Read the contents of a file, pointed to by this path.

Note

This method ensures exact byte-by-byte transfer. To do so, it encodes the data using base64 which makes console output less readable. If you intend to transfer text data, please use Path.read_text().

rmdir() → None[source]¶: Remove the directory pointed to by this path. The directory must be empty.

New in version 0.9.1.

unlink(missing_ok: bool = False) → None[source]¶

Remove the file or symbolic link from the filesystem. If the path points to a directory, use Path.rmdir() instead.

If missing_ok is false (the default), FileNotFoundError is raised if the path does not exist.

New in version 0.9.1.

mkdir(parents: bool = False, exist_ok: bool = False) → None[source]¶

Create a directory at the path this object represents.

If parents is false (the default), a missing parent causes FileNotFoundError to be raised. If parents is true, missing parent directories are created as needed.

If exist_ok is false (the default), a FileExistsError is raised if the path already exists. If exist_ok is true, a FileExistsError exception be suppressed if the target path exists and is a directory.

New in version 0.9.1.

Workdir¶

class tbot.machine.linux.Workdir[source]¶

classmethod static(host: H, pathstr: str) → tbot.machine.linux.workdir.Workdir[H][source]¶

Create a workdir in a static location, described by pathstr.

Example:

with tbot.acquire_lab() as lh:
    workdir = linux.Workdir.static(lh, "/tmp/tbot-my-workdir")

classmethod athome(host: H, subdir: str) → tbot.machine.linux.workdir.Workdir[H][source]¶

Create a workdir below the current users home directory.

Example:

with tbot.acquire_lab() as lh:
    # Use ~/tbot-foo-dir
    workdir = linux.Workdir.athome(lh, "tbot-foo-dir")

tbot will query the $HOME environment variable for the location of the current users home directory.

classmethod xdg_data(host: H, subdir: str) → tbot.machine.linux.workdir.Workdir[H][source]¶

Create a workdir in $XDG_DATA_HOME/tbot (usually ~/.local/share/tbot).

Example:

with tbot.acquire_lab() as lh:
    # Use ~/.local/share/tbot/foo-dir
    workdir = linux.Workdir.xdg_data(lh, "foo-dir")

classmethod xdg_runtime(host: H, subdir: str) → tbot.machine.linux.workdir.Workdir[H][source]¶

Create a workdir in $XDG_RUNTIME_DIR.

$XDG_RUNTIME_DIR is meant for non-essential runtime files and other file objects (such as sockets, named pipes, …) as specified by the XDG Base Directory Specification.

Example:

with tbot.acquire_lab() as lh:
    # Results in `$XDG_RUNTIME_DIR/tbot/tbot-pipes`
    workdir = linux.Workdir.xdg_runtime(lh, "tbot-pipes")

Lab-Host¶

class tbot.machine.linux.Lab[source]¶

Bases: tbot.machine.linux.linux_shell.LinuxShell

Mixin for marking a machine as a lab-host.

build() → tbot.machine.linux.build.Builder[source]¶

Return the default build-host for this lab.

If your lab does not contain a build-capable machine, just leave this method as is. tbot will raise an exception if a testcase attempts accessing the build-host anyway.

Builder¶

The Builder mixin allows marking a machine as a build-host. This means generic testcases like uboot.build can use it to automatically build projects. For this to work, a build-host needs to specify which toolchains it has installed and where tbot can find them.

class tbot.machine.linux.Builder[source]¶

Bases: tbot.machine.linux.linux_shell.LinuxShell

Mixin to mark a machine as a build-host.

You need to define the toolchain() method when using this mixin. You can then use the enable() method to enable a toolchain and compile projects with it:

with MyBuildHost(lh) as bh:
    bh.exec0("uptime")

    with bh.enable("generic-armv7a-hf"):
        cc = bh.env("CC")
        bh.exec0(linux.Raw(cc), "main.c")

Note

If you look closely, I have used linux.Raw(cc) in the exec0() call. This is necessary because a lot of toolchains define $CC as something like

CC=arm-poky-linux-gnueabi-gcc -march=armv7-a -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a8

where some parameters are already included. Without the linux.Raw, tbot would run

$ "${CC}" main.c

where the arguments are interpreted as part of the path to the compiler. This will obviously fail so instead, with the linux.Raw, tbot will run

$ ${CC} main.c

where the shell expansion will do the right thing.

abstract property toolchains: Dict[str, tbot.machine.linux.build.Toolchain]¶

Return a dictionary of all toolchains that exist on this buildhost.

Example:

@property
def toolchains(self) -> typing.Dict[str, linux.build.Toolchain]:
    return {
        "armv7a": linux.build.DistroToolchain("arm", "arm-linux-gnueabi-"),
        "armv8": linux.build.DistroToolchain("aarch64", "aarch64-linux-gnu-"),
        "mipsel": linux.build.DistroToolchain("mips", "mipsel-linux-gnu-"),
    }

enable(arch: str) → Iterator[None][source]¶

Enable the toolchain for arch on this BuildHost instance.

Example:

with lh.build() as bh:
    # Now we are on the buildhost

    with bh.enable("generic-armv7a-hf"):
        # Toolchain is enabled here
        cc = bh.env("CC")
        bh.exec0(linux.Raw(cc), "--version")

Toolchains¶

class tbot.machine.linux.build.Toolchain[source]¶

Bases: abc.ABC

Generic toolchain type.

abstract enable(host: tbot.machine.linux.build.Builder) → None[source]¶: Enable this toolchain on the given host.

class tbot.machine.linux.build.DistroToolchain(arch: str, prefix: str)[source]¶

Bases: tbot.machine.linux.build.Toolchain

Toolchain that was installed from distribution repositories.

Example:

@property
def toolchains(self):
    return {
        "armv7a": linux.build.DistroToolchain("arm", "arm-linux-gnueabi-"),
        "armv8": linux.build.DistroToolchain("aarch64", "aarch64-linux-gnu-"),
    }

class tbot.machine.linux.build.EnvScriptToolchain(path: tbot.machine.linux.path.Path[H])[source]¶

Bases: tbot.machine.linux.build.Toolchain

Toolchain that is initialized using an env script (e.g. yocto toolchain).

Example:

@property
def toolchains(self):
    yocto_root = self.fsroot / "opt" / "poky" / "2.6"

    return {
        "cortexa8hf": linux.build.EnvScriptToolchain(
            yocto_root / "environment-setup-cortexa8hf-neon-poky-linux-gnueabi",
        ),
    }

Create a new EnvScriptToolchain.

Parameters: path (linux.Path) – Path to the env script

Authenticators¶

For logging in via SSH using ParamikoConnector or SSHConnector, tbot provides the following ‘authenticators’:

class tbot.machine.linux.auth.NoneAuthenticator[source]¶

Bases: tbot.machine.linux.auth.AuthenticatorBase

Most primitive authenticator.

Tries not passing any specific credentials and hopes ssh-config already contains all necessary infos. This is the default.

class tbot.machine.linux.auth.PrivateKeyAuthenticator(key_file: Union[str, tbot.machine.linux.path.Path, pathlib.PurePath])[source]¶

Bases: tbot.machine.linux.auth.AuthenticatorBase

Authenticate using a private-key file.

Example:

class MySSHHost(connector.SSHConnector, linux.Bash):
    username = "foouser"
    authenticator = linux.auth.PrivateKeyAuthenticator("/home/foo/.ssh/id_rsa_foo")

class tbot.machine.linux.auth.PasswordAuthenticator(password: str)[source]¶

Bases: tbot.machine.linux.auth.AuthenticatorBase

Authenticate using a password.

Danger

This method is very insecure and might lead to PASSWORDS BEING STOLEN.

Example:

class MySSHHost(connector.SSHConnector, linux.Bash):
    username = "root"
    authenticator = linux.auth.PasswordAuthenticator("hunter2")

tbot.machine.linux¶

Linux Shells¶

Argument Types¶

Specials¶

RunCommandProxy¶

Paths¶

Workdir¶

Lab-Host¶

Builder¶

Toolchains¶

Authenticators¶

`tbot.machine.linux`¶