steve3

Recently I was tasked with doing some load testing on Assignments 2 as we slowly increase the amount of usage our pilot is getting. And actually, Fall Term Instructors will have to choose between Assignments tools when they set up their sites, so it should see a fair amount of load next fall as well.

It seems like our load tests usually involve something basic like hitting 5 or 6 pages in Sakai and ramping it up until something breaks, and maybe there is some sort of interaction as different user roles. In the past with assignments we’ve had lots of issues with duplicated entries and other oddities that occurred during concurrent submissions, so I wanted to do something with that a bit more, and with truly original data for each test.

Each run of this test consists of:

1. Submitting an assignment as a student.
2. Grading that assignment as an instructor and leaving some feedback.
3. Going back and viewing that feedback as a student.

The Grinder

The last 9 months or so I’ve (well we’ve), been using The Grinder to do load testing when there was time. I originally picked it over JMeter because it seems slicker, more extensible, and the tests are generally written in Python. It is a Java based application though, so it’s really Jython, and you can call and use Java libraries pretty easily. One of the nice things is that all testing is done by instrumenting objects or single methods, meaning you can test/measure just about anything. It also has a proxy recorder like most of these sorts of tool suites do, but I haven’t used it too much since many of the things you need to test in Sakai have so much parameterization and weird fields and urls that it’s sometimes easier to hand craft the interaction for performing fine grained load.

However! Last month I actually gave a demo of The Grinder and some of our tests at the Indianapolis Python Meetup, and someone there told me that the newest release of Selenium Server can now use a headless Firefox so it might actually be possible for load testing. We had experimented with running recorded Selenium scripts in The Grinder last fall, but because they spawned an actual Firefox GUI it wouldn’t have been useful for much more than functional testing. ( Selenium allows you to export scripts in a large number of programming languages. 2 of those are Python and Java, so consequently you can use either of those exports as test cases to measure in The Grinder ).

All that aside, for this particular exercise I’m mostly interested in using programmatic Python code to carefully test the steps listed above.

The Grinder Installation and Architecture

Because the default Grinder package isn’t as turnkey as I’d like, I’ve been putting together a package in contrib that all you have to do is check it out, and you should be able to run it assuming you have Java6 and a JAVA_HOME property set.

( Until a recent version The Grinder only required Java5. However, after a recent upgrade to the newest version of Jython 2.5 (yay! This helps us run lots of extra libraries), they also changed some of the underlying mechanisms for measuring tests. You’ll see in more depth below, but basically, previously they created proxy objects that would measure how long test executions would take. In the latest major Grinder upgrade they also added new support for measuring test execution by actually instrumenting the code. One positive impact this may have in the future is the ability to use other JVM languages to write tests. Personally I really prefer Python, but it’s always a bonus to include more of the JVM Language community. )

So, to check out Sakai Grinder and fire it up you can basically go:

Grinder Console after running tests with a connected agent loaded with the basic sakaihttp.py script.
The Grinder uses a pretty standard client/server sort of architecture in the form of the Console and Agents. Although the console that is started above is actually a GUI, there is a headless version so you can set up an automated test farm.

Each Agent starts up with a grinder.properties that specifies a number of things such as:

1. The test script to run. The above script used the simple example sakaihttp.py in the projects folder.
2. Things such as number of processes/threads/trials the agent should run, as well as ramp up time for threads and things.
3. An optional port and IP address for the console to connect to. This is so you can set up a farm of agents and control them all from a console. We’ve actually done that in the past for some load tests where we had agents running in Bloomington and Indianapolis and all connected to the same console to control them.
4. A lot of other things I haven’t looked at, as well as any user specified properties you want to create (we use these for like sakai server url prefix, etc, etc)

So it ends up looking like:



You’ve noticed that we’re checking out a trunk version of Sakai Grinder. After a few things get cleaned up I do want to roll a proper tag of it and zip it up for some sort of release.

Hello World

A basic Hello World, from the screen shot above is:

This has one test. You can see the Test object is what will show up in the console has a timed method. You then use that Test object to “wrap” a method or object that you want to time. This example actually uses the old proxy version of timing. The more modern method (that was added a few months ago) is called record(), which I’ll eventually switch to. The method or object that is returned from the wrap/record is what you actually call when you want to test, and each usage of it is timed and statistics are collected for it. Obviously this is pretty great, because you can measure anything ranging from HTTP calls, to JDBC calls, to well any object or method thats running on your JVM. And it works nice because Python has enough language facilities to make method wrapping and argument packing convenient.

Each script must have one of these TestRunner classes to use as the scripts test start point. You can also set it up to use a more naturally callable method (rather than a class with __call__), but I’m usually happy to stick with the default example.

Sakai Logins

In order to make scripting Sakai a bit easier, there is a sakaigrinder module, that builds on top of Grinders HTTP connection library. The Grinder HTTP connections, while maybe not as pretty as Pythons httplib2 library, is actually really great because it does lots of homework for keeping track of your sessions, cookies, and other things. So I actually prefer using it now, though I have though about implementing some httplib2 signatures on top of it.

You create a connection as follows:

And then you get a HTTPConnection back. Under the covers it just uses portal/xlogin at the moment, but I’ve started looking at creating other options for it, such as using the CAS login mechanisms at IU.

Script Structure and Threads/Processes

Taking a look at the Assignments 2 test, asnn2.py (which is still pretty sloppy but works), we’ll see the interaction between threads.

Each Grinder Agent’s properties file dictates the number of processes and threads to start up. Each process starts it’s own invocation of the test script. Then each thread in that process creates it’s own instance of the TestRunner class. In this way you can share data between test threads by putting shared state outside the TestRunner class in the script file. I’m currently doing something like this to round robin between test users, so I can load in a huge list of test students and have each thread use a different student:

In the example we can see that I have 3 tests I’m measuring. Submitting an assignment, Grading it, and Reviewing it as a student again. Outside of the TestRunner function I have the little getNextUser function that is shared by all threads and gives each test thread a separate user.

This is a bit primitive still, and I’m looking at better and more reusable ways to create huge distributed tests that include students/instructors and Sites to go along with each. It’s worked fine for testing so far though. I’ll probably move it into the sakaigrinder library.

As far as I know, there isn’t really a best practice way to share information between processes so it might require a database or something else, but that may have changed since I looked at the docs and skimmed the mailing list.

Manual HTTP Requests

Because of how I wanted the tests to work I ended up handcrafting the requests and post keys, and because they aren’t new age JSON payloads or something it’s a little ugly, but works.

These do the actual work and are the methods that we wrap in order to measure elapsed time and stuff. For Assignments 2 implementation reasons I won’t go in to, we actually have to go through and do some tricky editing of POST keys, and screen scraping HTTP Responses in order to have the entire life cycle work with unique data for every single test.

Failing Tests

You can cause the currently executing test in any Grinder script to fail using:

You can see above in the TestRunner class that we generate UUID’s to put into the students submission text, and to insert into the Instructors feedback. This is so we can verify under load that the assignments are actually getting recorded correctly, graded correctly, and we are aren’t destroying any data and whatnot.

So in the grade feedback test, we check to see that the student submission is completely intact and containing the UUID. And then when the student checks their submission at the end, we ensure it contains the intact feedback from the Grader with the second UUID that was generated. If these don’t check out that particular test is set to fail and this shows up in the aggregated data in the Console.

Honestly, we should go even further and write these out to a data file, and then re-validate the results against the SQL tables afterwards. But you have to start somewhere…

Getting Properties

The only other really interesting thing here is adding your own properties and fetching them.

The urlprefix I store in the grinder.properties. We could probably actually dynamically merge this with a sakai.properties so we can use those properties, and for testing JDBC queries and things.

The grinder.passwd actually comes from launching the agents that we have in sakaigrinder.sh using the startagentpasswd command.

I added this after accidentally checking in the password for our test users and having to reset a hundred of them. :p So you start with the startagentpasswd option and it prompts you for a password, then you can use that password property in your scripts. Obviously this assumes you’re using the same password for all your test users. This bit of functionality deserves some more sophistication in the future as well.

Epilogue
I’m pretty happy with this so far, and hope to keep cleaning it up as well as creating sophisticated tests that cover more than just overloading the server with requests until it breaks. I am pretty excited to give another look at headless selenium servers. Because selenium has a great recorder, and the headless server actually contains a full browser/javascript environment it could be pretty cool to set up in a cloud. There has been work by folks to create Grinder setups to run on Amazon EC2 clouds and things, so that is cool. Also, because it is Java based, I could see actually embedding it as a library in a Sakai module so you could load test executions to various loaded components using Sash and other things.

I’ve done several runs of this with a few hundred concurrent users, students and instructors, with no errors, and am still looking at ways to easily scale this up by auto generating assignments and classes on the fly.

Holy crap! Non Linear Video Editing on Linux!
And I didn’t actually have to do anything to get it working!

I was pretty amazed after upgrading to Ubuntu 10.04 on several machines, that PiTiVi, the new bundled video editor, actually started up, let me load some clips, drag them around on the time line, and render the whole lot to a video… without Crashing! I’m sure it’s probably still buggy as hell, but to actually perform some basic tasks with no issues and not installing anything was honestly pretty amazing. If you use Win32 or OSX you’re probably rolling your eyes, but I think it really is a big deal. In the 8 or so years I’ve been using desktop linux it’s amazing the progress we’ve made. I hope to dedicate a little time to desktop linux this year. I’ll probably start making more screen casts now too using XVidCap (screen recorder) and PiTiVi.

When you’re grading assignments in Sakai you usually have the option to mark up the text that a student submitted so you can return it to them with comments and annotations.  In the past this has been done by surrounding text you want highlighted in brackets, and then that text will turn red.

An example of doing this ( you can click on the images for bigger readable versions):

Marking up comments with double brackets.

And then the contents in brackets are highlighed in red:

The red text was surrounded with {{ }} by the instructor.

We’ve been working on a ‘more new’ Assignments application for Sakai (imaginatively called ‘Assignments 2′), and have been looking at making this process of marking up student submissions more like the sort of annotation feature you’d see in something like OpenOffice, Word, or even something like viewing a code diff.

After searching around for an html diffing tool, so far I’ve been pretty satisfied with Daisy Diff, which was a Google Summer of Code project a few years back and is used in the stable production Daisy CMS. It’s built especially for HTML Diffing, so it keeps tracking of tricky things like adjacent text nodes that appear as one in a browser, tag issues, etc etc.

So, now the instructor basically marks up their stuff by just editing the submission text:

Text is marked up. In the backend we save the original student submission, the annotated instructor submission, and are currently doing the diffing on each request that requires it. It seems fast enough on initial test data, but needs some perf testing for really large submissions still, to see if maybe it should be saved to the db.

and the added/changed/removed regions are highlighted in the student view:

The added areas are in yellow highlight, and the removed areas are in yellow highlight strikethrough.

You can change the added/changed/removed sections with standard CSS styles. I actually haven’t gotten anything to be picked up as “Changed” yet, always just added and removed so I might need to look into that more, though even with added/removed it’s pretty good. Also, after a few tests, it seems like the Daisy styles really are the ones that take precedence, for instance say if you had highlighted something blue, and it was removed, it would still highlight yellow. From looking at the results, it appears to wrap a span as closely as possible around text nodes so to take effect even if it’s across several tags.

Using the actual library was a little more work than I wanted, but not too much. As is standard with Java, you end up making lots of factory code for reading Strings and whatnot. ( There may actually be a utility wrapper method in Daisy Diff for this, but I didn’t see one. )

Although I stripped it all out for our project, daisydiff includes support for interesting javascript things, such as tabbing through all the annotations on the page and viewing each annotation and it’s information in a little popup, but we just want a simple static view.

It’s likely we’ll need to work on streamlining and improving the mechanisms on the instructor side to aide in previewing the annotations they’re doing, but so far we’ve made a pretty good start with the initial submission annotation using daisydiff.