Testing a user submitted python function

Question

I am creating a website with coding questions. When a user submits a code for a question I use the contents of the question to create a python file and then I send it to the piston API for evaluation.

I use this function to create the python files content as string

fn inject_code(question_id: i32, content: String, db_pool: DbPool) -> String {
    let question= get_single_question(question_id, db_pool).expect("Expected to find question");
    
    let imports= std::fs::read_to_string("injections/function_top.py").unwrap();
    let change_name = format!("__some_function = Solution.{}", question.function_name);
    let cases = question.cases;
    let py_runner = std::fs::read_to_string("injections/function.py").unwrap(); 

    format!("{imports}\n\n{content}\n\n{change_name}\n\n{cases}\n\n{py_runner}")
}

Where injection/function_top looks like:

from typing import *
from dataclasses import dataclass

@dataclass
class TestCase:
    inputs: tuple
    expected: Any

and injection/function looks like:

import io
from contextlib import redirect_stdout
import json

test_results = []
for case_id, test_case in enumerate(cases):
    args = test_case.inputs
    result = test_case.expected
    with io.StringIO() as buf, redirect_stdout(buf):
        error = None
        correct = False
        try:
            correct = __some_function(*args) == result
        except Exception as e:
            error = e

        function_stdout = buf.getvalue()
        case_signature = f"{str(args)} -> {str(result)}"
        test_results.append(
            {"is_correct": correct, "case_stdout": function_stdout, "error": error, "case_signature": case_signature}
        )
print(json.dumps(test_results, default=str))  # Ensures exceptions are stringified

The cases are stored as native python files such as this:

cases = [
    TestCase(inputs=([], []), expected=[]),
    TestCase(inputs=([1], [2]), expected=[1, 2]),
    TestCase(inputs=([2], [1]), expected=[2, 1]),
    TestCase(inputs=([3, 2, 1], []), expected=[3, 2, 1]),
    TestCase(inputs=([], [1, 2, 3]), expected=[1, 2, 3]),
    TestCase(inputs=([2, 3], [6, 4, 2]), expected=[2, 3, 6, 4, 2]),
]

and if we run this line of code for debug:

let injected_code = inject_code(id, content, pool);
std::fs::write("test.py", &injected_code).unwrap(); // debug the created python file 

we get test.py (this is the file that would be sent to the piston API)

from typing import *
from dataclasses import dataclass

@dataclass
class TestCase:
    inputs: tuple
    expected: Any

class Solution:
    def concat(arr_1: List[int], arr_2: List[int]):
        return arr_1 + arr_2

__some_function = Solution.concat

cases = [
    TestCase(inputs=([], []), expected=[]),
    TestCase(inputs=([1], [2]), expected=[1, 2]),
    TestCase(inputs=([2], [1]), expected=[2, 1]),
    TestCase(inputs=([3, 2, 1], []), expected=[3, 2, 1]),
    TestCase(inputs=([], [1, 2, 3]), expected=[1, 2, 3]),
    TestCase(inputs=([2, 3], [6, 4, 2]), expected=[2, 3, 6, 4, 2]),
]


import io
from contextlib import redirect_stdout
import json

test_results = []
for case_id, test_case in enumerate(cases):
    args = test_case.inputs
    result = test_case.expected
    with io.StringIO() as buf, redirect_stdout(buf):
        error = None
        correct = False
        try:
            correct = __some_function(*args) == result
        except Exception as e:
            error = e

        function_stdout = buf.getvalue()
        case_signature = f"{str(args)} -> {str(result)}"
        test_results.append(
            {"is_correct": correct, "case_stdout": function_stdout, "error": error, "case_signature": case_signature}
        )
print(json.dumps(test_results, default=str))  # Ensures exceptions are stringified

Some explanation: I am printing the final result because piston API only returns the program output. It is printed as JSON so that I can easily send it to my frontend. I am redirecting the stdout while the function is tested so that the user can see what was printed when the test case ran. I started by using a dictionary for my test cases but since the keys need to be hashable that turned out to be a problem.

I wanted the test cases to be written in native python so that any valid python object could be the answer or argument to the test case.

Is there any way I can improve this system? One mini problem that I have right now is that when creating a case my editor complains that "TestCase" is not defined.

Answer 1

name mangling

    let change_name = format!("__some_function = Solution.{}", question.function_name);

I don't understand the leading dunder there. Name mangling is seldom helpful. Recommend you just use _some_function -- a _private variable.

Also, instead of the rather vague "some", perhaps you'd prefer _target_function.

wildcard import

from typing import *

That module includes name, stat, and other innocuous sounding symbols that might cause confusion for a contestant. For example, typos could lead to surprising results. Recommend that you only import the several symbols that you actually want, and make contestants responsible for importing what they need in their submitted source code.

Also, please use $ isort *.py to organize your imports in the way that PEP 8 recommends.

old-style annotations

    def concat(arr_1: List[int], arr_2: List[int]):

Prefer the following. It is modern notation, and it doesn't need from typing import List.

    def concat(arr_1: list[int], arr_2: list[int]):

test module

This isn't the end of the world:

std::fs::write("test.py", &injected_code).unwrap();

But it's usually not a terrific idea to name a module test, since import test is already provided by the Batteries Included builtin libraries. Prefer test1 or some other identifier, to avoid confusion.

f-strings

I don't understand this line.

        case_signature = f"{str(args)} -> {str(result)}"

Note that formatting with an f-string will already invoke str(). What you wanted to write was simply:

        case_signature = f"{args} -> {result}"