> library(rmr)
Loading required package: RJSONIO
Loading required package: itertools
Loading required package: iterators
Loading required package: digest
> small.ints = to.dfs(1:10)
Warning: $HADOOP_HOME is deprecated.

12/04/28 19:17:45 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/04/28 19:17:45 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
12/04/28 19:17:45 INFO compress.CodecPool: Got brand-new compressor
> out = mapreduce(input = small.ints, map = function(k,v) keyval(v, v^2))
Warning: $HADOOP_HOME is deprecated.

packageJobJar: [/tmp/RtmpXlELmY/rmr-local-env, /tmp/RtmpXlELmY/rmr-global-env, 
                              /tmp/RtmpXlELmY/rhstr.map2cf71bf8a3a9, 
                              /home/piers/hadoop/tmp/hadoop-unjar1509588906818235502/] []
                              /tmp/streamjob912555254031649512.jar tmpDir=null
12/04/28 19:18:04 INFO mapred.FileInputFormat: Total input paths to process : 1
12/04/28 19:18:05 INFO streaming.StreamJob: getLocalDirs(): [/home/piers/hadoop/tmp/mapred/local]
12/04/28 19:18:05 INFO streaming.StreamJob: Running job: job_201204281916_0001
12/04/28 19:18:05 INFO streaming.StreamJob: To kill this job, run:
12/04/28 19:18:05 INFO streaming.StreamJob: /usr/libexec/../bin/hadoop job 
                             -Dmapred.job.tracker=192.168.1.3:9001 -kill job_201204281916_0001
12/04/28 19:18:15 INFO streaming.StreamJob: Tracking URL: http://192.168.1.3:50030/jobdetails.jsp?jobid=job_201204281916_0001
12/04/28 19:18:16 INFO streaming.StreamJob:  map 0%  reduce 0%
12/04/28 19:18:45 INFO streaming.StreamJob:  map 100%  reduce 0%
12/04/28 19:18:54 INFO streaming.StreamJob:  map 100%  reduce 17%
12/04/28 19:18:57 INFO streaming.StreamJob:  map 100%  reduce 67%
12/04/28 19:19:06 INFO streaming.StreamJob:  map 100%  reduce 100%
12/04/28 19:19:21 INFO streaming.StreamJob: Job complete: job_201204281916_0001
12/04/28 19:19:21 INFO streaming.StreamJob: Output: /tmp/RtmpXlELmY/file2cf7546b881b
> from.dfs('/tmp/RtmpXlELmY/file2cf7546b881b')
Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

[[1]]
[[1]]$key
[1] 1

[[1]]$val
[1] 1

attr(,"rmr.keyval")
[1] TRUE

[[2]]
[[2]]$key
[1] 2

[[2]]$val
[1] 4
...

The key things that people want (from an administration point of view) are to manage people and courses. Often these activities are a tiresome bulk process at set times of the year with a relatively minor tweaking type of activity in between. For managing users - create/update/delete, and enrolments - we already have the built in functionality to do bulk user upload. I have added to this for Courses and for Course categories.
The course upload admin tool can be used to create and manage course outlines, but it can also populate courses using either a nominated course as a template (copies the course contents using the Moodle backup/restore facility), or populate the course from a Moodle backup file.

diff --git a/dumbo/util.py b/dumbo/util.py
index a57166d..cd35df3 100644
--- a/dumbo/util.py
+++ b/dumbo/util.py
@@ -267,6 +267,7 @@ def findjar(hadoop, name):
     hadoop home directory and component base name (e.g 'streaming')"""
 
     jardir_candidates = filter(os.path.exists, [
+        os.path.join(hadoop, 'share', 'hadoop', 'contrib', name),
         os.path.join(hadoop, 'mapred', 'build', 'contrib', name),
         os.path.join(hadoop, 'build', 'contrib', name),
         os.path.join(hadoop, 'mapred', 'contrib', name, 'lib'),

hadoop fs -copyFromLocal /var/log/apache2/access.log /user/hduser/access.log
hadoop fs -ls /user/hduser/
dumbo start ipcount.py -hadoop /usr -input /user/hduser/access.log -output ipcounts
dumbo cat ipcounts/part* -hadoop /usr | sort -k2,2nr | head -n 5

0970518767|"DJ Adams" |M|Andrew_Powis/RVSUK/FES/Rank@rank.com
...

gource --start-position 0.28 --stop-position 0.29 --title 'Communication sphere since 1996' -s 1 --log-format custom email-log.txt

You can record it as a video too:

gource --start-position 0.28 --stop-position 0.29 --title 'Communication sphere since 1996' -s 1 \
    --log-format custom email-log.txt  -1280x720 -o - | ffmpeg -y -r 60 \
    -f image2pipe -vcodec ppm -i - -vcodec libx264 -preset ultrafast -crf 1 -threads 0 -bf 0 gource-video-of-email.mp4

I have found that, in particular at the beginning, it is often harder to think of basic data munging concepts in R - typical tasks like sorting, grouping, data type conversion - often your language of choice (Perl, Python, or even bash) is just quicker for doing these things in the first instance when I'm paring the data down into what I want to apply some form of statistical analysis or charting too.

With this in mind - I basically wanted to be able to perform SQL against a CSV file, without the hassle of loading it into a database first. Enter a clever tool written in Haskell called txt-sushi. This enables you to do interesting things like:

cat test.csv | tssql -table x - 'select a,b, sum(hours) AS hours_sum from x group by a,b'

However, for my purposes tssql is too strict on handling data types, and is dependent on Haskell, so I've built my own simple CSV SQL processor - csvtable in Python using SQLite as a backend. This is surprisingly easy to do, and let's you have the benefit of the convenience and power of SQLite syntax:

python csvtable.py \
  --where="system_code != 'LEAVE'" \
  --convert='date_epoch:date,hours:int' \
  --list="*, sum(hours) AS hours_sum, min(date_epoch) AS date_epoch_min, 
               max(date_epoch) AS date_epoch_max, count(*) AS days,
               ROUND(AVG(hours), 2) AS avg_time, MIN(hours) AS hours_min,
               MAX(hours) AS hours_max" \
  --group='organisation_code, system_code, request_id' \
  --file=test1.csv | \
 python csvtable.py --list='*, ROUND(((date_epoch_max - date_epoch_min) / (60 * 60 * 24)) + 1, 2) AS duration' > test2.csv

This presented another problem - in Python, how do you access the HDFS API? There are two existing integrations that I can find - https://github.com/traviscrawford/python-hdfs, and http://code.google.com/p/libpyhdfs/. Travis Crawfords' is easy to get going, but as it uses a JNI binding I didn't relish the prospect of trying to make sure CLASSPATHs etc are right across all my Hadoop nodes (which for my purposes are any machine that I can beg, borrow or steal), in light of this I created my own cheap and cheerful library that uses subprocess to call the 'hadoop' executable for 'fs' - hdfsio .
I admit this isn't the height of efficiency (or possibly elegance), but it is surprisingly robust and very simple.

Hadoop is a big thing, and takes (to my mind) quite a lot of effort to get going, and to understand how you can bend it to your will. Par of this learning process has been about finding a comfortable installation pattern for Linux - in particular Ubuntu, and the best help I've found so far has been from Michael Noll. Things that I had to be careful about were getting ssh working, and name resolution exactly right on all nodes that you put in your cluster, as you distribute things like /etc/hadoop/masters and the *-site.xml config files.

The next stage was to find a development pattern that enabled me to avoid Java. The answer to this for me is Hadoop Streaming. This basically allows you to pipe IO in and out of programs written in your favourite language - and in this case Michael does brilliantly again with Python and MapReduce.

As part of some work for the Ministry of Education for LMS -> myPortfolio (Mahara) integration, it became apparent that we needed a Web Services stack. This is not particularly interesting in it's self, but it is something that an interconnected service needs, in order to participate in a Socially Networked world.
Building a WS framework is not a difficult thing, but it is relatively time consuming (anything that takes more than a few weeks is considered expensive here), so the problem was, how to develop an unexciting feature that in itself does not deliver any great new user experience quickly and cheaply. At this point it occurred to me that there might be a solution in what Moodle has achieved with it's Web Services Framework - after all, Mahara is (in a previous life) based on Moodle.

It turned out, that the way that Peta Skoda has developed the Moodle WSF is fundamentally based on Zend data services, and is quite portable.

To this end, I have ported it as an auth plugin which can be downloaded here, and the documentation is here.

This gives the basic features of token, and user based auth, with SOAP, XML-RPC and JSON emitting REST based services. There are a number of other things that I'd like to add to this, the most important being OAuth based authentication, and JSON based import parameter consumption.

Edit: JSON based import parameter consumption, has been done, but I want to add replacing MNet to the list of things to do.

I've looked at a few BI (well SAP, Business Objects, and Pentaho) implementations over the years, and one of the things that I have always found frustrating/off putting is what I consider the huge startup costs for such implementations. This has usually been characterised by expensive infrastructure implementations in both hardware and software coupled with the difficulty that most businesses have in visualising what data they need to have access to, and how it should be most effectively presented.

I've found this dilemma more accute in the Moodle world, as the so many of the customers involved are on a very tight to non-existent budget, yet their requirement to analyse Learning Managment System performance data is still there.

A year ago, I concluded that Pentaho was my first choice, for the twin reasons that it's OpenSource (specifically no license fees), and that it has sufficiently good data modelling tools to enable a suite of reports customised to Moodle to be delivered. While this reduces the cost of delivering a flexible reporting solution for Moodle, it still falls short on a couple of points:

(1) Most people who implement Moodle are not Data Warehousing, or Modelling experts so they are unlikely to be able to sufficiently accurately determine what their requirements are in advance (actually a common business problem, not unique to the Moodle community).
(2) Pentaho, while reasonably straight forward to install, is still another complex piece of software to host - a major barrier to entry for most Moodle implementations.

What I started looking for then, was a set of visualisation tools that could be integrated with Moodles PHP environment - atleast users would then be able to do more complex reporting and analysis. What I found exceeded my expectations, in the form of a Labs project from Google called Fusion Tables.

Fusion Tables is shaping up to be Business Intelligence reporting with the twist of collaborative, and Geo encoding capabilities. The basic mode is that CSV files of data can be uploaded into a flexible storage engine, datasets can be joined and merged, automatically Geo encoded, and then consumed through a good set of graphical presentation tools. Datasets can be shared and collaboratively edited.

As Luck would have it that this service is firstly free, and secondly exposed via an SQL-like API integrated with the standard Google OAuth mechanism. This makes it attractive as a generic data analysis and reporting tool for a low cost operating environment like Moodle and the education sector.

To test out the theory of all this, I've implemented 3 things:
* OAuth integration for Moodle including a site, and secret registry
* A generic Fusion Tables data proxy for Moodle
* A Gradebook export module that enables the export of the standard gradebook data to Fusion Tables

For the curious, this can be found at Gitorious -moodle-local_oauth.

Also, with help from Joachim, I've added a static function sapnwrfc_removefunction(<sysid>, <function name>) to the PHP connector that allows the removing of function definitions from the local cache. this is most useful when developing RFC applications in PHP, as you can modify your RFC definition without having to restart the web server everytime. This is available from version 0.09 here.

For Moodle integration, Erlend Strømsvik of Ny Media AS, developed an authentication plugin, which I've made a number of changes to around configuration options, and Moodle session integration. This has now been documented and added to Moodle Contrib to give it better visibility to the Moodle community at large. Documentation is here and the contrib entry is here.

Additionally, I've had some time to put together a proper cookbook style set of examples in the sapnwrfc-cookbook. These examples, while specifically for Perl, are almost identical for sapnwrfc for Python, Ruby, and PHP too.

It's not easy getting it up and running (need to setup datasources, build Modrian schemas, design MDX queries, and then implement the xaction), but once you are there, there is so much potential. The ability to tune the queries interactively using the Pivot reporting engine, and the drill down features are excellent.