Serial

OS in Rust

Your Task

We are trying to get automated testing working.
We can quickly open and close QEMU.
We want to blast QEMU to stdout
And ideally not open QEMU.
We just recently turned to the serial port.

Serial Port

Easy mode is a serial port
- An old interface standard which is no longer found in modern computers.
It is easy to program and…
QEMU can redirect the bytes sent over serial to the host.

UART

The chips implementing a serial interface are called UARTs
- Universal asynchronous receiver-transmitter

It was one of the earliest computer communication devices, used to attach teletypewriters for an operator console. It was also an early hardware system for the Internet.

Two Parts

We will proceed in two parts: 1. [ ] Configure Rust to print to the serial port. 2. [ ] Configure QEMU to display the messages relayed via the serial port to a terminal.

There is also a mysterious third part, panic handling.

Rust

Using UART

uart_16550 implements a SerialPort struct.
- It represents the UART registers.
Like println, I do this in a new file.

Add to Main

src/main.rs

mod serial;

Create Serial File

Make a new file, src/serial.rs, that will implement functions which print to the serial ports.
- Not to the VGA buffer!
- We will hope to eventually see the text in on host device.
  - That is, the same command line we invoke cargo t from.
To begin, we will use the same print and println macros from src/vga.rs, but prepend them with “serial” so know they are for special external usage.
- This will probably look roughly as expected.
- It’s a headache so I’m providing it.

src/serial.rs

/// Prints to the host through the serial interface.
#[macro_export]
macro_rules! serial_print {
    ($($arg:tt)*) => {
        $crate::serial::_print(format_args!($($arg)*));
    };
}

/// Prints to the host through the serial interface, appending a newline.
#[macro_export]
macro_rules! serial_println {
    () => ($crate::serial_print!("\n"));
    ($fmt:expr) => ($crate::serial_print!(concat!($fmt, "\n")));
    ($fmt:expr, $($arg:tt)*) => ($crate::serial_print!(
        concat!($fmt, "\n"), $($arg)*));
}

`_print`

With these macros furnished, all that remains is to implement the _print function which is called internally within the macros, namely in the most indented line of serial_print.
Vs. VGA, this will be much easier.
- We don’t need to worry about linbreaks.
- We don’t need to worry about newlines.
- We don’t need to worry about scrolling.
In fact, we don’t have to do any formatting at all!
- The uart_16550::SerialPort already has the .write_fmt Trait that we needed to implement println for the VGA buffer.
All that remains is to properly set up the serial port for printing (and to print to it).

Recall

My haters, of which there are many, may have been opinionated on how I got access to .write_fmt for the VGA buffer.

src/vga.rs

pub struct Dummy {}

impl core::fmt::Write for Dummy {
    fn write_str(&mut self, s: &str) -> core::fmt::Result {
        str_to_vga(s);
        Ok(())
    }
}

pub fn _print(args: core::fmt::Arguments) {
    use core::fmt::Write;
    let mut d = Dummy {};
    d.write_fmt(args).unwrap();
}

We will only need to borrow _print, as SerialPort is already a (1) struct that (2) impl core::fmt::Write.

So…

Something like this.

src/serial.rs

pub fn _print(args: core::fmt::Arguments) {
    use core::fmt::Write;
    let mut serial_port = Dummy {};
    serial_port.write_fmt(args).unwrap();
}

Instead of a Dummy we need a uart_16500::SerialPort.

Left as Exercise

Getting the rest of this to work is left as an exercise, and not too bad.
But you can’t test it yet.

Testing

Usage of Serial

Previously, we worked with the following test_runner

src/main.rs

#[cfg(test)]
fn test_runner(_tests: &[&dyn Fn()]) {
    let fs = [_ex];
    for i in 0..fs.len() {
        print!("Running test case {:0x}... ", i);
        fs[i]();
        println!("Success.");
    }
}

There would be some benefit to either of the following formulations.

Print to Serial Only

This works well once you know your serial printer is working as intended.

src/main.rs

#[cfg(test)]
fn test_runner(_tests: &[&dyn Fn()]) {
    let fs = [_ex];
    for i in 0..fs.len() {
        serial_print!("Running test case {:0x}... ", i);
        fs[i]();
        serial_println!("Success.");
    }
}

Double Print

When getting my code to work, I used both print and serial_print.
- This reduced my error handling cases (as I could check the VGA buffer for text as well as host device).
- I was only using this during debugging or I would’ve written a wrapper.

src/main.rs

#[cfg(test)]
fn test_runner(_tests: &[&dyn Fn()]) {
    let fs = [_ex];
    for i in 0..fs.len() {
        serial_print!("Running test case {:0x}... ", i);
        print!("Running test case {:0x}... ", i);
        fs[i]();
        println!("Success.");
        serial_println!("Success.");
    }
}

Takeaways

In any case, make sure you have some that (1) prints to serial in the (2) test case.

QEMU

A Fun Story

I spent several hours thinking I didn’t get serial ports working when, in fact, I had, I simply had configured QEMU to display the results of serial port writes, so I wasn’t see anything.
Do not be like me.

QEMU Options

So going forward, we assume the correctness of the serial_println macro.
- You won’t be able to test it until you’ve done this section!
- That is okay, we can do two things at once.

Return to Cargo

To see the serial output from QEMU, we need to use the -serial argument to redirect the output to “Standard I/O” stdio.
- For now, we are only using stdout but stdout $\in$ stdio

Cargo.toml

[package.metadata.bootimage]
test-args = ["-device", "isa-debug-exit,iobase=0xf4,iosize=0x04", "-serial", "stdio"]

Massaging Main

I also wanted to do multiple tests, so I updated test_runner.
It wasn’t too exciting, I just added some formatting to look nice.
I also run the test multiple times, to mimic running multiple tests.

src/main.rs

fn test_runner(_tests: &[&dyn Fn()]) {
    let fs = [_ex, _ex, _ex];
    for i in 0..fs.len() {
        serial_print!("Beginning test 0x{:02x}...", i);
        fs[i]();
        serial_println!(" [Pass]");
    }
    unsafe { x86_64::instructions::port::Port::new(0xf4).write(0xAu32) };
}

Testing Tests

When we run cargo test now, we see the test output directly in the console:

$ cargo t
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
     Running unittests src/main.rs (target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da)
Building bootloader
    Finished `release` profile [optimized + debuginfo] target(s) in 0.07s
Running: `qemu-system-x86_64 -drive format=raw,file=/home/calvin/tmp/work/target/x86_64-osirs/debug/deps/bootimage-osirs-f4313fd1ed2d08da.bin -no-reboot -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none`
Beginning test 0x00... [Pass]
Beginning test 0x01... [Pass]
Beginning test 0x02... [Pass]

Good Enough?

Surely that is a good enough… right?
- Wrong.
It is extremely annoying to have the graphics window rapidly open and close.
- It is visually distracting.
- I won’t work well in all environments, such as GitHub Actions
- It is resource intensive.

Hiding QEMU

We can prevent the display window launch by passing the -display none argument to QEMU…
- Though we would only want to do this when conducting automated testing.

Cargo.toml

[package.metadata.bootimage]
test-args = ["-device", "isa-debug-exit,iobase=0xf4,iosize=0x04", "-serial", "stdio", "-display", "none"]

Quoth Blog

This is not only less annoying, but also allows our test framework to run in environments without a graphical user interface, such as CI services or SSH connections.

Good Enough?

Imagine this test:

src/main.rs

fn _ex() {
    loop {}
}

Timeouts

cargo t waits until the test runner exits.
A test that never returns can block the test runner forever.
That’s unfortunate, but not a big problem in practice since it’s usually easy to avoid endless loops.
We have no cases of loop {} in our code, right?

Systems Design

In our case, however, endless loops can occur in various situations: - The bootloader fails to load our kernel - The BIOS/UEFI firmware fails to load the bootloader - The CPU enters a loop {} statement. - Perhaps after QEMU bug. - The hardware causes a system reset.

Set a Timeout

By default, apparently timeout is 300 seconds or 5 minutes.
- That is way too long.
- I give up on problems instantly.
We can manually set it to be shorter, like 1 second.
bootimage config

Cargo.toml

[package.metadata.bootimage]
test-timeout = 1

Test It

Try this now:

src/main.rs

fn _ex() {
    loop {}
}

What I see

The takeaway here is that I get my console back in a finite amount of time (5 minutes is infinite).

$ cargo t 
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
     Running unittests src/main.rs (target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da)
Building bootloader
    Finished `release` profile [optimized + debuginfo] target(s) in 0.07s
Running: `qemu-system-x86_64 -drive format=raw,file=/home/calvin/tmp/work/target/x86_64-osirs/debug/deps/bootimage-osirs-f4313fd1ed2d08da.bin -no-reboot -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none`
Beginning test 0x00...Error: Test timed out
error: test failed, to rerun pass `--bin osirs`

Caused by:
  process didn't exit successfully: `bootimage runner /home/calvin/tmp/work/target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da` (exit status: 1)
note: test exited abnormally; to see the full output pass --no-capture to the harness.

Panics

One More

Okay, so a test can pass or stall infinitely, but what about fail.

src/main.rs

fn _bad() {
    assert!(false);
}

Huh?

What is happening here?

Beginning test 0x00... [Pass]
Beginning test 0x01...Error: Test timed out

An assertion is made.
The assertion fails.
The failed assertion triggers a panic.
The panic handler broadcasts an error message to the VGA buffer, which is not being displayed.
The panic handler enters an infinite loop.
Eventually, the timeout occurs and terminates the test suite.

Instead

We should write a custom panic handler for the testing context.
We use conditional compilation.
I used the following template:

src/main.rs

#[cfg(not(test))]
#[panic_handler]
fn panic(info: &core::panic::PanicInfo) -> ! {
    println!("{}", info);
    loop {}
}

#[cfg(test)]
#[panic_handler]
fn panic(info: &core::panic::PanicInfo) -> ! {
    // Your code here
    loop {}
}

You should:
- Print a message denoting the test failed.
- Print a message decribing what the failure was.
- Exit QEMU.
  - With a status code other than success.
  - I used 0xF vs. 0xA which I used in my test_runner.
Here is an example output:

$ cargo t
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.10s
     Running unittests src/main.rs (target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da)
Building bootloader
    Finished `release` profile [optimized + debuginfo] target(s) in 0.07s
Running: `qemu-system-x86_64 -drive format=raw,file=/home/calvin/tmp/work/target/x86_64-osirs/debug/deps/bootimage-osirs-f4313fd1ed2d08da.bin -no-reboot -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none`
Beginning test 0x00... [Pass]
Beginning test 0x01... [Fail]
panicked at src/main.rs:51:5:
assertion failed: false
error: test failed, to rerun pass `--bin osirs`

Caused by:
  process didn't exit successfully: `bootimage runner /home/calvin/tmp/work/target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da` (exit status: 31)
note: test exited abnormally; to see the full output pass --no-capture to the harness.

Fin

Deliverables

Ensure you have a testing suite that can run without a graphics window and handle pass, fail, and divergence.

--- title: Serial format: html --- # Your Task - We are trying to get automated testing working. - We can quickly open and close QEMU. - We want to blast QEMU to `stdout` - And ideally not open QEMU. - We just recently turned to the *serial port*. ## Serial Port - Easy mode is a [serial port](https://en.wikipedia.org/wiki/Serial_port) - An *old* interface standard which is no longer found in modern computers. - It is easy to program and... - QEMU can redirect the bytes sent over serial to the host. ## UART - The chips implementing a serial interface are called UARTs - [Universal asynchronous receiver-transmitter](https://en.wikipedia.org/wiki/Universal_asynchronous_receiver-transmitter) > It was one of the earliest computer communication devices, used to attach teletypewriters for an operator console. It was also an early hardware system for the Internet. ## Two Parts We will proceed in two parts: 1. [ ] Configure Rust to print to the serial port. 2. [ ] Configure QEMU to display the messages relayed via the serial port to a terminal. There is also a mysterious third part, panic handling. # Rust ## Popular - UART 16550 is popular, nominally, so we'll use that. - The various UARTs are all basically the same, like ASCII vs. Code Page ??? - There is, naturally, a [`uart_165001`](https://docs.rs/uart_16550) crate. - Add it to your dependencies in the expect manner (in `Cargo.toml`). ```{.toml filename="cargo.toml" code-line-numbers="4"} [dependencies] bootloader = "0.9" x86_64 = "0.14.2" uart_16550 = "0.2.0" ``` ## Using UART - `uart_16550` implements a `SerialPort` struct. - It *represents* the UART registers. - Like `println`, I do this in a new file. ### Add to Main ```{.rs filename="src/main.rs"} mod serial; ``` ## Create Serial File - Make a new file, `src/serial.rs`, that will implement functions which print to the serial ports. - Not to the VGA buffer! - We will hope to eventually see the text in on host device. - That is, the same command line we invoke `cargo t` from. - To begin, we will use the same `print` and `println` macros from `src/vga.rs`, but prepend them with "serial" so know they are for special external usage. - This will probably look roughly as expected. - It's a headache so I'm providing it. ```{.rs filename="src/serial.rs"} /// Prints to the host through the serial interface. #[macro_export] macro_rules! serial_print { ($($arg:tt)*) => { $crate::serial::_print(format_args!($($arg)*)); }; } /// Prints to the host through the serial interface, appending a newline. #[macro_export] macro_rules! serial_println { () => ($crate::serial_print!("\n")); ($fmt:expr) => ($crate::serial_print!(concat!($fmt, "\n"))); ($fmt:expr, $($arg:tt)*) => ($crate::serial_print!( concat!($fmt, "\n"), $($arg)*)); } ``` ## `_print` - With these macros furnished, all that remains is to implement the `_print` function which is called internally within the macros, namely in the most indented line of `serial_print`. - Vs. VGA, this will be much easier. - We don't need to worry about linbreaks. - We don't need to worry about newlines. - We don't need to worry about scrolling. - In fact, we don't have to do any formatting at all! - The `uart_16550::SerialPort` *already has* the `.write_fmt` Trait that we needed to implement `println` for the VGA buffer. - All that remains is to properly set up the serial port for printing (and to print to it). ## Recall - My haters, of which there are many, may have been opinionated on how I got access to `.write_fmt` for the VGA buffer. ```{.rs filename="src/vga.rs"} pub struct Dummy {} impl core::fmt::Write for Dummy { fn write_str(&mut self, s: &str) -> core::fmt::Result { str_to_vga(s); Ok(()) } } pub fn _print(args: core::fmt::Arguments) { use core::fmt::Write; let mut d = Dummy {}; d.write_fmt(args).unwrap(); } ``` - We will only need to borrow `_print`, as `SerialPort` is already a (1) `struct` that (2) `impl core::fmt::Write`. ## So... - Something like this. ```{.rs filename="src/serial.rs"} pub fn _print(args: core::fmt::Arguments) { use core::fmt::Write; let mut serial_port = Dummy {}; serial_port.write_fmt(args).unwrap(); } ``` - Instead of a `Dummy` we need a `uart_16500::SerialPort`. ## Left as Exercise - Getting the rest of this to work is left as an exercise, and not too bad. - But you can't test it yet. # Testing ## Usage of Serial - Previously, we worked with the following `test_runner` ```{.rs filename="src/main.rs"} #[cfg(test)] fn test_runner(_tests: &[&dyn Fn()]) { let fs = [_ex]; for i in 0..fs.len() { print!("Running test case {:0x}... ", i); fs[i](); println!("Success."); } } ``` - There would be some benefit to either of the following formulations. ### Print to Serial Only - This works well once you know your serial printer is working as intended. ```{.rs filename="src/main.rs"} #[cfg(test)] fn test_runner(_tests: &[&dyn Fn()]) { let fs = [_ex]; for i in 0..fs.len() { serial_print!("Running test case {:0x}... ", i); fs[i](); serial_println!("Success."); } } ``` ### Double Print - When getting my code to work, I used both `print` and `serial_print`. - This reduced my error handling cases (as I could check the VGA buffer for text as well as host device). - I was only using this during debugging or I would've written a wrapper. ```{.rs filename="src/main.rs"} #[cfg(test)] fn test_runner(_tests: &[&dyn Fn()]) { let fs = [_ex]; for i in 0..fs.len() { serial_print!("Running test case {:0x}... ", i); print!("Running test case {:0x}... ", i); fs[i](); println!("Success."); serial_println!("Success."); } } ``` ### Takeaways - In any case, make sure you have some that (1) prints to serial in the (2) test case. # QEMU ## A Fun Story - I spent several hours thinking I didn't get serial ports working when, in fact, I had, I simply had configured QEMU to display the results of serial port writes, so I wasn't see anything. - Do not be like me. ## QEMU Options - So going forward, we assume the correctness of the `serial_println` macro. - You won't be able to test it until you've done this section! - That is okay, we can do two things at once. ## Return to Cargo - To see the serial output from QEMU, we need to use the `-serial` argument to redirect the output to "Standard I/O" `stdio`. - For now, we are only using `stdout` but `stdout` $\in$ `stdio` ```{.toml filename="Cargo.toml"} [package.metadata.bootimage] test-args = ["-device", "isa-debug-exit,iobase=0xf4,iosize=0x04", "-serial", "stdio"] ``` ## Massaging Main - I also wanted to do multiple tests, so I updated `test_runner`. - It wasn't *too* exciting, I just added some formatting to look nice. - I also run the test multiple times, to mimic running multiple tests. ```{.rs filename="src/main.rs"} fn test_runner(_tests: &[&dyn Fn()]) { let fs = [_ex, _ex, _ex]; for i in 0..fs.len() { serial_print!("Beginning test 0x{:02x}...", i); fs[i](); serial_println!(" [Pass]"); } unsafe { x86_64::instructions::port::Port::new(0xf4).write(0xAu32) }; } ``` ## Testing Tests - When we run `cargo test` now, we see the test output directly in the console: ```{.sh} $ cargo t Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s Running unittests src/main.rs (target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da) Building bootloader Finished `release` profile [optimized + debuginfo] target(s) in 0.07s Running: `qemu-system-x86_64 -drive format=raw,file=/home/calvin/tmp/work/target/x86_64-osirs/debug/deps/bootimage-osirs-f4313fd1ed2d08da.bin -no-reboot -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none` Beginning test 0x00... [Pass] Beginning test 0x01... [Pass] Beginning test 0x02... [Pass] ``` ## Good Enough? - Surely that is a good enough... right? - Wrong. - It is extremely annoying to have the graphics window rapidly open and close. - It is visually distracting. - I won't work well in all environments, such as GitHub Actions - It is resource intensive. ## Hiding QEMU - We can prevent the display window launch by passing the `-display none` argument to QEMU... - Though we would *only* want to do this when conducting automated testing. ```{.toml filename="Cargo.toml"} [package.metadata.bootimage] test-args = ["-device", "isa-debug-exit,iobase=0xf4,iosize=0x04", "-serial", "stdio", "-display", "none"] ``` ## Quoth Blog > This is not only less annoying, but also allows our test framework to run in environments without a graphical user interface, such as CI services or SSH connections. - [CI - Continuious integration with GitHub Actions](https://docs.github.com/en/actions/get-started/continuous-integration) - [SSH - Secure Shell](https://en.wikipedia.org/wiki/Secure_Shell) ## Good Enough? - Imagine this test: ```{.rs filename="src/main.rs"} fn _ex() { loop {} } ``` ## Timeouts - `cargo t` waits until the test runner exits. - A test that never returns can block the test runner forever. - That's unfortunate, but not a big problem in practice since it's usually easy to avoid endless loops. - We have no cases of `loop {}` in our code, right? ## Systems Design In our case, however, endless loops can occur in various situations: - The bootloader fails to load our kernel - The BIOS/UEFI firmware fails to load the bootloader - The CPU enters a `loop {}` statement. - Perhaps after QEMU bug. - The hardware causes a system reset. ## Set a Timeout - By default, apparently timeout is 300 seconds or 5 minutes. - That is *way* too long. - I give up on problems instantly. - We can manually set it to be shorter, like 1 second. - [bootimage config](https://github.com/rust-osdev/bootimage#configuration) ```{.toml filename="Cargo.toml"} [package.metadata.bootimage] test-timeout = 1 ``` ## Test It - Try this now: ```{.rs filename="src/main.rs"} fn _ex() { loop {} } ``` ## What I see - The takeaway here is that I get my console back in a finite amount of time (5 minutes is infinite). ```{.sh} $ cargo t Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s Running unittests src/main.rs (target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da) Building bootloader Finished `release` profile [optimized + debuginfo] target(s) in 0.07s Running: `qemu-system-x86_64 -drive format=raw,file=/home/calvin/tmp/work/target/x86_64-osirs/debug/deps/bootimage-osirs-f4313fd1ed2d08da.bin -no-reboot -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none` Beginning test 0x00...Error: Test timed out error: test failed, to rerun pass `--bin osirs` Caused by: process didn't exit successfully: `bootimage runner /home/calvin/tmp/work/target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da` (exit status: 1) note: test exited abnormally; to see the full output pass --no-capture to the harness. ``` # Panics ## One More - Okay, so a test can pass or stall infinitely, but what about fail. ```{.rs filename="src/main.rs"} fn _bad() { assert!(false); } ``` ## Huh? - What is happening here? ```{.sh} Beginning test 0x00... [Pass] Beginning test 0x01...Error: Test timed out ``` 1. An assertion is made. 2. The assertion fails. 3. The failed assertion triggers a panic. 4. The panic handler broadcasts an error message to the VGA buffer, which is not being displayed. 5. The panic handler enters an infinite loop. 6. Eventually, the timeout occurs and terminates the test suite. ## Instead - We should write a *custom panic handler* for the testing context. - We use *conditional compilation*. - I used the following template: ```{.rs filename="src/main.rs"} #[cfg(not(test))] #[panic_handler] fn panic(info: &core::panic::PanicInfo) -> ! { println!("{}", info); loop {} } #[cfg(test)] #[panic_handler] fn panic(info: &core::panic::PanicInfo) -> ! { // Your code here loop {} } ``` - You should: - Print a message denoting the test failed. - Print a message decribing what the failure was. - Exit QEMU. - With a status code *other* than success. - I used `0xF` vs. `0xA` which I used in my `test_runner`. - Here is an example output: ```{.sh} $ cargo t Finished `test` profile [unoptimized + debuginfo] target(s) in 0.10s Running unittests src/main.rs (target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da) Building bootloader Finished `release` profile [optimized + debuginfo] target(s) in 0.07s Running: `qemu-system-x86_64 -drive format=raw,file=/home/calvin/tmp/work/target/x86_64-osirs/debug/deps/bootimage-osirs-f4313fd1ed2d08da.bin -no-reboot -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none` Beginning test 0x00... [Pass] Beginning test 0x01... [Fail] panicked at src/main.rs:51:5: assertion failed: false error: test failed, to rerun pass `--bin osirs` Caused by: process didn't exit successfully: `bootimage runner /home/calvin/tmp/work/target/x86_64-osirs/debug/deps/osirs-f4313fd1ed2d08da` (exit status: 31) note: test exited abnormally; to see the full output pass --no-capture to the harness. ``` # Fin ## Deliverables - [ ] Ensure you have a testing suite that can run without a graphics window and handle pass, fail, and divergence.