10. Tests#
In this notebook, we cover the following subjects:
# To enable type hints for lists, dicts, tuples, and sets we need to import the following:
from typing import List, Dict, Tuple, Set
Testing is an essential skill in any programmer’s toolkit. It ensures our code runs as expected, catches bugs early, and gives us confidence in the software we create. In this notebook, we’re going to dive into two powerful tools in Python’s testing arsenal:
You’ve already worked with exceptions(Notebook 8), learning how to handle errors smoothly when they arise. Now, we’ll explore how to prevent them from happening in the first place by actively testing our code. By the end of this chapter, you’ll be comfortable writing, running, and understanding tests in Python.
One last thing before we start… This notebook will be special because it is split into 2 parts. In Part 1 we will be learning about the assert keyword in our usual, Jupyter Notebook environment. However, for Part 2, you will be required to follow along the tutorial about pytests (explained in this notebook) and practice in a seperate py files. Meaning, you will have to work in your choice of code editor (VS Code, Pycharm etc.) in multiple py files. We will get back to that in due course.
Ok, now all is set. Let’s dive in!
10.1. Part 1 - Using assert for Basic Testing#
The assert statement in Python is a simple yet powerful way to verify that a condition holds True. It’s commonly used for quick checks or to validate assumptions in code. If the condition you’re testing is True, the program continues as expected. However, if the condition is False, an AssertionError is raised, and the program stops. This is particularly useful when you want to catch potential bugs early on.
10.1.1. Syntax of assert#
The basic syntax of an assert statement is as follows:
assert <condition>, <optional error message>
<condition>: This is the expression you want to test. If it evaluates to True, the program continues running. If it’s False, Python raises anAssertionError.<optional error message>: This is an optional message that will display if the assertion fails, helping you understand what went wrong.
Now that we know the theory behind the syntax, let’s look at a few examples!
Let’s say we have a function that expects positive numbers only. We can use an assert statement to ensure this requirement:
def process_number(number: int):
assert number > 0, "Number must be positive"
return number * 2
print(process_number(5)) # works fine
print(process_number(-3)) # raises AssertionError: "Number must be positive"
10
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[2], line 6
3 return number * 2
5 print(process_number(5)) # works fine
----> 6 print(process_number(-3)) # raises AssertionError: "Number must be positive"
Cell In[2], line 2, in process_number(number)
1 def process_number(number: int):
----> 2 assert number > 0, "Number must be positive"
3 return number * 2
AssertionError: Number must be positive
In the above example, if n is positive, the function continues and returns the result. However, if n is negative, it raises an AssertionError with a helpful message.
10.1.2. Common keywords used with assert#
Bellow is a table of the most common keywords and expressions used with assert, along with brief explanations for each.
These keywords make it easier to perform specific checks, such as verifying types, values, and membership, which can help you catch issues early.
Keyword |
Explanation |
Example |
|---|---|---|
|
Checks if two objects are the same instance (identity). |
|
|
Checks if two values are equal. |
|
|
Checks if an element exists within a container (e.g., list, tuple, set, dictionary). |
|
|
Checks if an element does not exist within a container. |
|
|
Checks if an object is an instance of a specific class or tuple of classes. |
|
|
Checks if a class is a subclass of another class. |
|
|
Checks if all elements of one set are present in another set. |
|
|
Checks if a value is greater than or equal to, or less than or equal to, another value. |
|
|
Checks if a value is strictly greater than or less than another value. |
|
|
Verifies the length of a collection, often used with |
|
|
Checks if an object is exactly a specific type, useful when |
|
|
Verifies that at least one element in an iterable is true (used with a condition). |
|
|
Verifies that all elements in an iterable are true (used with a condition). |
|
Let’s look at one more example:
def validate_user(user: Dict[str, any]) -> str:
"""
Validates user information for sign-up.
Args:
user (Dict[str, any]): A dictionary containing user data with required keys:
- 'username': a string with at least 3 characters
- 'age': an integer greater than 12
- 'email': a string containing an '@' symbol
Returns:
str: A success message if all validation checks pass.
Raises:
AssertionError: If any validation check fails, with a message describing the error.
"""
# check if user is a dictionary
assert isinstance(user, dict), "User information must be a dictionary."
# check for required keys
required_keys = {"username", "age", "email"}
assert required_keys.issubset(user.keys()), f"User data must include {required_keys}"
# check if username is a string with at least 3 characters
assert isinstance(user["username"], str) and len(user["username"]) >= 3, "Username must be a string with at least 3 characters."
# check if age is an integer greater than 12
assert isinstance(user["age"], int) and user["age"] > 12, "Age must be an integer greater than 12."
# check if email contains an '@' symbol
assert "@" in user["email"], "Email must contain an '@' symbol."
# iff all assertions pass, return a success message
return f"User {user['username']} is successfully validated."
# test with valid user data
print(validate_user({"username": "Alice", "age": 25, "email": "alice@example.com"}))
# test with invalid user data
# print(validate_user({"username": "Al", "age": 10, "email": "aliceexample.com"})) # Raises AssertionError: "Username must be a string with at least 3 characters."
The above function (validate_user) checks if a user’s information meets certain requirements during sign-up.
It takes a dictionary with the user’s details and verifies that it includes the necessary keys: ‘username’, ‘age’, and ‘email’.
The function ensures the username is at least three characters long,
the age is an integer, greater than 12
and that the email contains an ‘@’ symbol.
If any of these checks fail, it raises an assertion error with a message explaining what went wrong. If everything is correct, it returns a success message confirming the user has been validated.
10.1.3. When to Use assert#
The assert statement is great for situations where you need quick, in-line checks. However, it’s generally used in scenarios where you’re confident the conditions should hold under normal circumstances. If you’re unsure whether a condition might fail in a production environment, consider handling the error with a try-except block instead.
Let’s look at an example:
Example with assert:
Let’s say we are working on a function to calculate the square root of a number. Normally, you expect that the number provided is positive. In such a case, you can use assert to make a quick, in-line check:
def calculate_square_root(x: int) -> int:
# sssert that the input should be positive
assert x >= 0, "Input must be a non-negative number"
return x ** 0.5
# example usage:
print(calculate_square_root(9)) # works fine
print(calculate_square_root(-4)) # raises AssertionError with message "Input must be a non-negative number"
Here, assert is ideal because, under normal circumstances, you expect the input to be positive.
What will happen if you change an assertion from assert x > 0 to assert x >= 0 in your code?
Changing the assertion to assert x >= 0 allows the value 0 to pass without raising an error, whereas assert x > 0 would fail if x were 0. This change could lead to unexpected behavior in your program if you rely on x being strictly greater than 0 for correct functionality. It’s essential to understand the specific requirements of your code to ensure it behaves as intended.
Example with try-except:
If the input is coming from an unpredictable source (e.g., user input in a production environment), it’s better to handle it with try-except so the program doesn’t crash. For instance:
def calculate_square_root(x: int) -> int:
try:
if x < 0:
raise ValueError("Input must be a non-negative number")
return x ** 0.5
except ValueError as e:
print(f"Error: {e}")
return None
# example usage:
print(calculate_square_root(9)) # outputs 3.0
print(calculate_square_root(-4)) # outputs "Error: Input must be a non-negative number" and returns None
In this version, if a negative number is entered, it smoothly handles the error without crashing, which is more suitable for production scenarios where unexpected input is possible.
Note: This is the END of Part 1. Before we begin with Part 2, please open the code editor of your choice and create two new .py files called test_example.py and example.py. This will be where you write your tests as you follow along.
10.2. Part 2 - Writing Your First pytest#
In your example.py file, let’s start with a basic function.
Open your
example.pyfile.Write the following code in it:
# test_example.py
def addition(num_1:int, num_2:int) -> int:
'''
This function adds 2 numbers together.
Params:
num_1:int -> the first number
num_2: int -> the second number
Return:
An integer which is the result of adding num_1 and num_2 together.
'''
return num_1 + num_2
In general its good practice if we create functions that start with test_ as test functions.
In your test_example.py file, let’s start with a few basic tests.
Open your
test_example.pyfile.Write the following code in it:
# so we can use the function we created
from example import addition
def test_addition() -> any:
'''
This function tests if the addition() function works as expected.
Params: None
Return:
A boolean (True) if the function works correctly or a string with an appropriate error message.
'''
assert addition(1,2) == 3, 'The function does not work correctly'
assert addition(3,4) == 7, 'The function does not work correctly'
assert addition(0,0) == 0, 'The function does not work correctly'
The test_addition() function is our first test. The assert statement checks if 1 + 2 equals 3, 3+4 equals 7 and so on. If it does, the test passes. Otherwise, pytest will raise an error, and the test will fail.
To run the tests, open your terminal (or command prompt), navigate to the directory containing test_example.py, and type:
pytest
If everything works as intended, you should see the following message:
=========================================================== test session starts ===========================================================
platform win32 -- Python 3.10.11, pytest-8.3.3, pluggy-1.5.0
rootdir: <your directory name>
plugins: anyio-4.0.0, typeguard-4.3.0
collected 1 item
test_example.py . [100%]
============================================================ 1 passed in 0.02s ============================================================
In case you run into any errors, check the next section (Error handling), otherwise skip the next section (Writing more tests).
10.2.1. Error handling (optional)#
In case you run into the following error:
pytest : The term 'pytest' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of
the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ pytest
+ ~~~~~~
+ CategoryInfo : ObjectNotFound: (pytest:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
Check if pytest is downloaded or not by writing this into your terminal: pip show pytest
If pytest is not found, use this command: pip install pytest
If pytest is indeed installed and you still have the same error, try this command: python -m pytest
Hopefully now all is good and you should see the following message:
=========================================================== test session starts ===========================================================
platform win32 -- Python 3.10.11, pytest-8.3.3, pluggy-1.5.0
rootdir: <your directory name>
plugins: anyio-4.0.0, typeguard-4.3.0
collected 1 item
test_example.py . [100%]
============================================================ 1 passed in 0.02s ============================================================
10.2.2. Writing more tests#
I hope I can congratule you on your first pytest and you did not just give up fixing the error :). Anyways, let’s extend our py file:
In example.py, add the following functions:
def subtraction(num_1: int, num_2: int) -> int:
"""
Subtracts the second integer from the first.
Params:
num_1 (int): The integer from which to subtract.
num_2 (int): The integer to subtract.
Returns:
int: The result of the subtraction.
"""
return num_1 - num_2
def multiplication(num_1: int, num_2: int) -> int:
"""
Multiplies two integers.
Params:
num_1 (int): The first integer to multiply.
num_2 (int): The second integer to multiply.
Returns:
int: The result of the multiplication.
"""
return num_1 * num_2
def division(num_1: int, num_2: int) -> float:
"""
Divides the first integer by the second.
Params:
num_1 (int): The integer to be divided.
num_2 (int): The integer to divide by.
Returns:
float: The result of the division as a float.
"""
return num_1 / num_2
Moreover, to test these functions, lets write into example.py:
def test_subtraction() -> None:
"""
Tests the subtraction function with various cases.
"""
assert subtraction(2, 3) == -1, "Expected subtraction(2, 3) to be -1"
assert subtraction(5, 5) == 0, "Expected subtraction(5, 5) to be 0"
assert subtraction(10, 4) == 6, "Expected subtraction(10, 4) to be 6"
assert subtraction(-3, -7) == 4, "Expected subtraction(-3, -7) to be 4"
def test_multiplication() -> None:
"""
Tests the multiplication function with various cases.
"""
assert multiplication(2, 3) == 6, "Expected multiplication(2, 3) to be 6"
assert multiplication(-2, 3) == -6, "Expected multiplication(-2, 3) to be -6"
assert multiplication(0, 5) == 0, "Expected multiplication(0, 5) to be 0"
assert multiplication(4, -1) == -4, "Expected multiplication(4, -1) to be -4"
def test_division() -> None:
"""
Tests the division function with various cases.
"""
assert division(6, 3) == 2.0, "Expected division(6, 3) to be 2.0"
assert division(5, 2) == 2.5, "Expected division(5, 2) to be 2.5"
assert division(-8, 4) == -2.0, "Expected division(-8, 4) to be -2.0"
assert division(0, 1) == 0.0, "Expected division(0, 1) to be 0.0"
As you can see, each function begins with test_ and contains a different assertion to verify basic mathematical operations. Pytest will automatically detect these tests when you run pytest again in the terminal.
Run pytest in the terminal to verify that all tests pass.
If all went well, you should see:
=========================================================== test session starts ===========================================================
platform win32 -- Python 3.10.11, pytest-8.3.3, pluggy-1.5.0
rootdir: <your directory name>
plugins: anyio-4.0.0, typeguard-4.3.0
collected 1 item
test_example.py . [100%]
============================================================ 4 passed in 0.02s ============================================================
Notice that now we passed 4 tests in contrats to the previous 1.
Hint: If you still only passed one test, try saving the py file you are working in, and then run the test again. Usually that does the trick.
At this point your example.py should look like this:
#############################
# Writing Your First pytest #
#############################
def addition(num_1:int, num_2:int) -> int:
'''
This function adds 2 numbers together.
Params:
num_1:int -> the first number
num_2: int -> the second number
Return:
An integer which is the result of adding num_1 and num_2 together.
'''
return num_1 + num_2
#############################
# Writing more tests #
#############################
def subtraction(num_1: int, num_2: int) -> int:
"""
Subtracts the second integer from the first.
Params:
num_1 (int): The integer from which to subtract.
num_2 (int): The integer to subtract.
Returns:
int: The result of the subtraction.
"""
return num_1 - num_2
def multiplication(num_1: int, num_2: int) -> int:
"""
Multiplies two integers.
Params:
num_1 (int): The first integer to multiply.
num_2 (int): The second integer to multiply.
Returns:
int: The result of the multiplication.
"""
return num_1 * num_2
def division(num_1: int, num_2: int) -> float:
"""
Divides the first integer by the second.
Params:
num_1 (int): The integer to be divided.
num_2 (int): The integer to divide by.
Returns:
float: The result of the division as a float.
"""
return num_1 / num_2
Whilst your test_example.py should look like this:
from example import addition, subtraction, multiplication, division
def test_addition() -> any:
'''
This function tests if the addition() function works as expected.
Params: None
Return:
A boolean (True) if the function works correctly or a string with an appropriate error message.
'''
assert addition(1,2) == 3, 'The function does not work correctly'
assert addition(3,4) == 7, 'The function does not work correctly'
assert addition(0,0) == 0, 'The function does not work correctly'
def test_subtraction() -> None:
"""
Tests the subtraction function with various cases.
"""
assert subtraction(2, 3) == -1, "Expected subtraction(2, 3) to be -1"
assert subtraction(5, 5) == 0, "Expected subtraction(5, 5) to be 0"
assert subtraction(10, 4) == 6, "Expected subtraction(10, 4) to be 6"
assert subtraction(-3, -7) == 4, "Expected subtraction(-3, -7) to be 4"
def test_multiplication() -> None:
"""
Tests the multiplication function with various cases.
"""
assert multiplication(2, 3) == 6, "Expected multiplication(2, 3) to be 6"
assert multiplication(-2, 3) == -6, "Expected multiplication(-2, 3) to be -6"
assert multiplication(0, 5) == 0, "Expected multiplication(0, 5) to be 0"
assert multiplication(4, -1) == -4, "Expected multiplication(4, -1) to be -4"
def test_division() -> None:
"""
Tests the division function with various cases.
"""
assert division(6, 3) == 2.0, "Expected division(6, 3) to be 2.0"
assert division(5, 2) == 2.5, "Expected division(5, 2) to be 2.5"
assert division(-8, 4) == -2.0, "Expected division(-8, 4) to be -2.0"
assert division(0, 1) == 0.0, "Expected division(0, 1) to be 0.0"
10.2.3. More assert Statements with pytest#
We can use assert statements to compare different data types and objects. Here are a few examples (add these to your example.py file):
def string_to_upper(example_string: str) -> str:
"""
Converts the input string to uppercase.
Args:
example_string (str): The string to convert to uppercase.
Returns:
str: The input string in uppercase.
"""
return example_string.upper()
def list_contains_elements(example_list: list, elements: list) -> bool:
"""
Checks if all elements in the `elements` list are contained within `example_list`.
Args:
example_list (list): The list to check within.
elements (list): The list of elements to check for.
Returns:
bool: True if all elements are found in `example_list`, otherwise False.
"""
return all(element in example_list for element in elements)
def dictionary_has_key_value(example_dict: dict, key: str, value: any) -> bool:
"""
Checks if the dictionary contains a specific key-value pair.
Args:
example_dict (dict): The dictionary to check.
key (str): The key to look for in the dictionary.
value (any): The value to match with the specified key.
Returns:
bool: True if the dictionary contains the specified key-value pair, otherwise False.
"""
return example_dict.get(key) == value
Moreover, add these tests to your test_example.py file:
def test_string_to_upper() -> None:
"""
Tests the string_to_upper function with various cases.
"""
assert string_to_upper("hello") == "HELLO", "Expected string_to_upper('hello') to be 'HELLO'"
assert string_to_upper("Test") == "TEST", "Expected string_to_upper('Test') to be 'TEST'"
assert string_to_upper("123abc") == "123ABC", "Expected string_to_upper('123abc') to be '123ABC'"
assert string_to_upper("") == "", "Expected string_to_upper('') to be ''"
def test_list_contains_elements() -> None:
"""
Tests the list_contains_elements function with various cases.
"""
assert list_contains_elements([1, 2, 3, 4], [1, 3]) is True, "Expected list_contains_elements([1, 2, 3, 4], [1, 3]) to be True"
assert list_contains_elements([1, 2, 3], [4]) is False, "Expected list_contains_elements([1, 2, 3], [4]) to be False"
assert list_contains_elements([1, 2, 3], []) is True, "Expected list_contains_elements([1, 2, 3], []) to be True"
assert list_contains_elements([], [1]) is False, "Expected list_contains_elements([], [1]) to be False"
def test_dictionary_has_key_value() -> None:
"""
Tests the dictionary_has_key_value function with various cases.
"""
assert dictionary_has_key_value({"key": "value"}, "key", "value") is True, "Expected dictionary_has_key_value({'key': 'value'}, 'key', 'value') to be True"
assert dictionary_has_key_value({"key": "other_value"}, "key", "value") is False, "Expected dictionary_has_key_value({'key': 'other_value'}, 'key', 'value') to be False"
assert dictionary_has_key_value({"another_key": "value"}, "key", "value") is False, "Expected dictionary_has_key_value({'another_key': 'value'}, 'key', 'value') to be False"
assert dictionary_has_key_value({}, "key", "value") is False, "Expected dictionary_has_key_value({}, 'key', 'value') to be False"
You should get something like this when your run the test:
=========================================================== test session starts ===========================================================
platform win32 -- Python 3.10.11, pytest-8.3.3, pluggy-1.5.0
rootdir: <your directory name>
plugins: anyio-4.0.0, typeguard-4.3.0
collected 1 item
test_example.py . [100%]
============================================================ 7 passed in 0.03s ============================================================
As we have seen in Part 1, assertions can be used to check string manipulations, list contents, dictionary key-value pairs and much more. Don’t be afraid to combine your knowledge about theassert keyword in your pytest experiments.
10.2.4. Using pytest.raises to Check for Exceptions#
One last thing, sometimes, you’ll want to ensure that specific errors are raised under certain conditions. Pytest provides a helpful feature called pytest.raises for this purpose. This lets us test that code raises the expected exception.
Let’s test a function that raises an error if a number is divided by zero.
Add this function to your example.py file:
def divide_by_zero(x: int, y: int) -> float:
"""
Divides x by y, raising a ValueError if y is zero.
Args:
x (int): The numerator.
y (int): The denominator.
Returns:
float: The result of the division.
Raises:
ValueError: If y is zero.
"""
if y == 0:
raise ValueError("Cannot divide by zero")
return x / y
Also, add this test into the test_example.py file:
import pytest
def test_divide_by_zero() -> None:
"""
Tests that dividing by zero raises a ValueError with the correct message.
"""
with pytest.raises(ValueError, match="Cannot divide by zero"):
divide(10, 0)
So what is happening here? The pytest.raises context manager checks if a ValueError is raised with the message "Cannot divide by zero". If divide_by_zero(10, 0) does not raise this error, the test will fail.
pytest.raises(ValueError)ensures that aValueErroris raised whendivide_by_zero(10, 0)is called.match="Cannot divide by zero"further checks that the error message matches the provided string.
If the ValueError with the expected message is raised, the test passes. If no exception is raised or if a different message appears, the test fails.
Note
This approach eliminates the need for an assert statement, as pytest.raises() handles the verification directly.
That’s it for testing. Let’s start practicing!
Exercises
Let’s practice! Mind that each exercise is designed with multiple levels to help you progressively build your skills. Level 1 is the foundational level, designed to be straightforward so that everyone can successfully complete it. In Level 2, we step it up a notch, expecting you to use more complex concepts or combine them in new ways. Finally, in Level 3, we get closest to exam level questions, but we may use some concepts that are not covered in this notebook. However, in programming, you often encounter situations where you’re unsure how to proceed. Fortunately, you can often solve these problems by starting to work on them and figuring things out as you go. Practicing this skill is extremely helpful, so we highly recommend completing these exercises.
For each of the exercises, make sure to add a docstring and type hints, and do not import any libraries unless specified otherwise.
Note: To get the most out of these exercises, it is recommended that you create 2 new py files (just as we did for the tutorial), called pytest_exercises.py and test_pytest_exercises.py, and solve the exercises bellow in these files.
10.2.5. Exercise 1#
Level 1: Write a function celsius_to_fahrenheit(celsius) that converts a temperature from Celsius to Fahrenheit. Create tests to verify that this function works for common values.
Hint: 0 Celsius equals 32 Fahrenheit.
Example input:
celsius_to_fahrenheit(0)
celsius_to_fahrenheit(100)
celsius_to_fahrenheit(-40)
Example output:
32
212
-40
# TODO.
Level 2: Modify celsius_to_fahrenheit() to handle a list of Celsius values and return a list of Fahrenheit conversions. Write tests for this new functionality.
Example input:
celsius_to_fahrenheit([0, 100, -40])
Example output:
[32, 212, -40]
# TODO
Level 3: Expand celsius_to_fahrenheit() to accept either a single value or a list of values. It should return a single Fahrenheit value if given a single Celsius input, or a list of values if given a list input. Write tests for both types of inputs.
Example input:
celsius_to_fahrenheit(0)
celsius_to_fahrenheit([0, 100, -40])
Example output:
32
[32, 212, -40]
# TODO.
10.2.6. Exercise 2#
Level 1: Write a function sum_even_numbers(lst) that returns the sum of all even numbers in a list. Write tests to verify the function with lists containing both even and odd numbers.
Example input:
sum_even_numbers([1, 2, 3, 4])
sum_even_numbers([10, 15, 20, 25])
Example output:
6
30
# TODO.
Level 2: Write a function count_primes(lst) that returns the count of prime numbers in a list. Write tests to verify the function works for lists with different values.
Example input:
count_primes([2, 3, 4, 5, 6])
count_primes([10, 11, 12, 13, 14, 15])
Example output:
3
2
# TODO.
Level 3: Write a function find_divisors(n) that returns a list of all divisors of a given number n. For example, find_divisors(6) should return [1, 2, 3, 6]. Write tests for numbers with different divisor structures.
Example input:
find_divisors(6)
find_divisors(12)
Example output:
[1, 2, 3, 6]
[1, 2, 3, 4, 6, 12]
# TODO.
10.2.7. Exercise 3#
Level 1: Write a function count_vowels(s) that returns the count of vowels (a, e, i, o, u) in a given string. Write tests to verify the function on different strings.
Example input:
count_vowels("hello")
count_vowels("pytest")
Example output:
2
1
# TODO.
Level 2: Write a function capitalize_words(s) that capitalizes the first letter of each word in a sentence. Write tests to verify correct capitalization for sentences with varying structures.
Example input:
capitalize_words("hello world")
capitalize_words("testing with pytest")
Example output:
"Hello World"
"Testing With Pytest"
# TODO.
Level 3: Write a function longest_word(s) that returns the longest word in a sentence. In case of a tie, return the first longest word encountered. Write tests to verify the function with sentences of varying word lengths.
Example input:
longest_word("find the longest word here")
longest_word("multiple words with same length")
Example output:
"longest"
"multiple"
# TODO.
Material for the VU Amsterdam course “Introduction to Python Programming” for BSc Artificial Intelligence students. These notebooks are created using the following sources:
Learning Python by Doing: This book, developed by teachers of TU/e Eindhoven and VU Amsterdam, is the main source for the course materials. Code snippets or text explanations from the book may be used in the notebooks, sometimes with slight adjustments.