Good morning. For CIS*2750, I've had the responsibility of working on the automarker. It's begun interfering with my other classes, such that I almost had to fail to submit two important pieces of work this week to complete the automarker (that about 70 students depend on). Yikes. But let's look at some of my more interesting work on the automarker.
CIS*2750: Isolating tests
I did some fun work ensuring our tests were more isolated from one another. Currently, I write a test plan and then implement a series of pass/fail tests for their code, divided across a few different suites. Previously, each suite and all its tests ran in a single process. (Each suite is run independently by some Perl code.) However, if a student's code messed up its memory accesses severely enough or had an infinite loop, we'd have to interrupt that whole suite. Some students might end up with lower score than they warranted, or 0, as they'd lost the subsequent tests in the suite. Our policy was to re-run their tests when they reported it.
For the first automarker, I alleviated the issue a bit. We use a nice C exception handling library writ by a former student. I modified the framework a little so it could catch interruptions (that are usually fatal to the process, like SIGINT) as an exception and we could handle that in the test code and recover. That only helps in some cases, like infinite loops or recursion, or when their code inexplicably finds some way to block that isn't a memory error. However, many problems involved serious memory handling errors that would prevent the subsequent tests from doing anything less than locking up anyway. (Crazy mallocness.)
The true solution was obvious. We really should run each test in its own process. And now for the 2nd assignment's automarker, we do. I wrote a two function test manager, to which I passed a test function and its data, and it forks the process into two, the one running the test and then quitting, the other waiting for the first to end, so it can resume and continue running the rest of the tests. The second one, the managing process, will wait 2 seconds for the test-running process to quit before assuming the student code has erred irretrievably and kills it. (Recorded as a fail.)
One of the main differences is that the automarker should now be truly automated. Before, it had to be supervised so you could interrupt hanging tests, and sometimes many times over. It was a pain for our professor. Now, you should be able to start it and come back after 5 minutes to a complete grade log for the whole course :) (It would be faster, but for the 2 second time out.)
CIS*2750: Tracing a stack
When student code fails a test, we want to give them the best idea as to why. This usually involves printing out a useful error message about how the output of their code differed from what we expected. But when their code crashes in the automarker, we currently give them nothing more than that. "You crashed," basically. The exception handling library mentioned above can catch most crashes (segmentation faults or often "bus error" on Mac OS X), and it actually has code to print a trace of the code stack, to show the student where exactly the crash occurred in their code.
However, we have an issue, and I wasted several hours on it before giving up. We link in their code from a static archive for A2 (for A1, we compiled it right in). For some reason, this confuses the stack tracer (the glibc backtrace () call and __built_in_return_address ()) which ends up returning a bizarre address offset from I don't know what into I don't know where. I googled around for a while, but it might be one of those esoteric cases that comes up so seldom there was no (or at least not easily found) prior discussion. ARGH (there's a whole post about the difference in problem solving between googling and dissecting it yourself). Sadly, my time is not infinite, so I could not fix this, and thus the students will have to run their code in gdb or something to find their crashes. (They shouldn't rely on a marking programme to show them anyway. It would just be a courtesy.)
CIS*6890: So Much Work
I've actually managed to get quite behind in CIS*6890 (Technical Communication and Research Methods) due to all my automarking work over the past 1.5 weeks (automarker 1 revisions, then automarker 2 test plan, then automarker 2 implementation). There was stress, and panic, and despair, but eventually understanding and mercy. I am fortunate. I am now still very busy though :) Hopefully I will get to write a little about all the papers I have to finish reading for this soon!
TAing for CIS*2750 has taken up much more time than I'd usually like. I ended up talking to the professor about it, and I feel a bit foolish for how that conversation went, but I just felt I had to do something. It's understood that there is a lot of up-front work for TAing this course, and I think I'd actually like to do it again at some point, as it would be much easier then, and I feel like I could have a better impact on the students in my lab.
CIS*6050 Neural Networks
Right, I need to not forget about this! Lectures are surprisingly riveting. I actually didn't sleep last night (finishing up automarker code), and I kept almost dozing off in lecture today. I survived, somehow, though. (And slept quite a bit after class, hence the early morning post.)
No comments:
Post a comment