Embedding Perl with Inline - Using Perl to Tame your Daemons

Embedding Perl with Inline - Using Perl to Tame your Daemons Piers Harding

piers@ompa.net

2002 Piers Harding

"Embedding Perl with Inline - Using Perl and Inline to Tame your Daemons"

From the abstract:
This talk shows Cookbook style recipes for embedding Perl interpreters into Daemon programs (or vice versa). Many server type products provide an API for calling out to custom code (e.g. Apache has mod_perl and the Apache API, jabberd and jsm modules, etc), so how do you embed Perl into these products? Furthermore, these products will often contain other APIs for data manipulation (jabberd has an xml manipulation library based on expat) that would also be useful inside your embedded Perl interpreter - how do you do this? The answer is Inline, and the standard Perl API. This talk was born from the learning experience of embedding Perl into Jabber C++ components, and jabberd itself. Introduction: So what's this all about then? What's this all about? Using perl inside server/daemon type applications [mod_perl] or, Intertwining Perl with event driven libraries [Internet protocol clients etc.] "Suffering short term pain, to get maximum gain." Who is this for? This is for developers: who are strongest/most comfortable in Perl have some C/C++ knowledge and want to make use of Perl where ever possible Benefits? Spend time making Perl available in your environment, so that you can: More mileage More speed (usually) CPAN Advocacy Scratch an Itch! ....and have some fun! A few key people Brian Ingerson Neil Watkiss and many more... Contents: What are we going to cover? The Toolset - what you need to know The HOWTO: daemons event driven libraries Deployment Contents: What are we going to cover? - continued Common Issues Maybe three examples: wu-ftpd - role your own mod_perl for ftp Jabber::MIO - The JabberD network Interface Jabber::JAX::Component - the day of the JECL? Resources The Toolset Inline ( or XS ) Inline is used to bind other languages in to your Perl program. Inline is the easiest way to bind other languages to Perl Inline uses XS under the covers to do the binding XS should have a slightly faster startup time, in that Inline will do some of its own checking/processing before the extension is loaded, but after that there should be no difference performance wise. Use Inline to pull in the event driven library Use Inline to encapsulate C functions, and C++ objects within the Perl callbacks As Ingy would say - "Just use Inline; " The Toolset - Continued the standard perlapi Perl has a comprehensive API built into the interpreter Create an interpreter in exactly the same way as the Perl executable does Access to a myriad of functions and macros to define Perl data types, and to coerce in and out of native values (most likely C/C++) The HOWTO Daemons Embed the Perl Interpreter initialise the interpreter load perl module/s create Perl method callbacks Destroy Interpreter Create the container module for Perl callbacks create callback functions to access parent Daemon code encapsulate Daemon functions for callback in Perl (Inline C/C++) The HOWTO - Continued Event driven libraries Library encapsulation Module parameter control for library initialisation ( port, etc. ) Inline code to bind in library functions event loop control/launch insert callback method for accessing Perl subroutine/s Library objects/function encapsulation module encapsulate library objects/functions for access within Perl (Inline) Bless Perl structure (hash) containing pointer into object and pass into Perl The HOWTO - Continued ... again Deployment How to install Perl Module location Perl Module reloading HOWTO Daemons Example is based on the embedding Perl into the WU-FTPD server. Loads the interpreter before the Daemon forks for each request handled Destroys and cleans up the interpreter on shutdown signals Provides Perl callbacks for manipulating the files/names before and after STOR and RETR ftp functions

Fig. 1. Generic Process flow in a Daemon Process HOWTO Daemons - structure Structure of code - where to put things Embed the Perl Interpreter The Perl interpreter is managed in 4 phases.
These 4 phases need to be invoked by the calling daemon process at the appropriate times (to do this you may need to modify the daemon- eg. Wu-FTPD ). Initialise Interpreter load Perl modules Perform callbacks Destroy Interpreter and cleanup resources Note: all of this has been encapsulated in a separate library libmoperl.

Fig. 1. Process flow in WU-FTPD Initialise Interpreter Declare a single global Interpreter instance declare sudo command line arguments for Interpreter - Additional libray paths etc. (not necessary if callback module is installed) allocate/construct Interpreter Parse initial program/arguments passed (NULL/empty program in this case - returns true) Run the program Note: the glue code for loading modules with C extensions, is automatially generated by running: And the result is this: Load Perl Modules Evaluate code fragments to "use" modules, including those that contain methods for later execution Perform callbacks declare local argument stack pointer preserve a marker for the current stack pointer push arguments onto the stack (local copy) give the interpreter the local stack pointer Call the subroutine (using the SV* flavour) retrieve the stack pointer test the result retrieve the return values(make sure that you take copies) give back the original stack pointer free up temporary variables Perform callbacks - continued Coresponding Static Method in Perl Destroy and Cleanup shutdown the interpreter free up any resources Container Module for Perl Callbacks create callback functions to access parent Daemon code encapsulate Daemon functions for callback in Perl (Inline C/C++) Callback functions to access Parent daemon These functions could be built directly into the Inline code instead of being abstracted out, although, this way avoids namespace clashes with Perl declared data, and #includes ( in modperl.c as opposed to ftpd.c for example ). Encapsulate Daemon functions for callback in Perl (Inline) The binding from within a Perl module that enables access to an external function (Daemon) 'DATA', NAME => 'WuFtpd', CCFLAGS => " -I".abs_path((-d "./modperl" ? "./modperl" :".")), MYEXTLIB => abs_path((-d "./modperl" ? "./modperl" :".")). "/libmodperl.a", VERSION => '0.01'; ... __DATA__ __C__ void my_logout( SV* message ) { my_perl_dologout(SvPV( message, PL_na )); } ]]> Wrap Inline function in Perl subroutine.
A further opportunity for parameter validation. Daemon Deployment How to install Need to control build of daemon or external library from Makefile.PL separate Makefile.PL's (directories) for each module that uses Inline. Only for C with Inline so far. Perl module location - either explicitly specified, or must be installed Perl module reloading for Daemons without restart HOWTO - Event Driven Libraries Starting with Perl is completely different to starting with a Daemon. Processing control starts and ends with Perl There is no need to worry about managing the Perl Interpreter Anonymous Subroutines can easily be passed in for the callbacks

Fig. 1. Sample Process flow for Event Driven Libraries HOWTO - Event Driven Libraries - continued Example is based on the mio (managed input/output) library for the 1.5 JabberD server, for managing socket connections/events for the JabberD, or Jabber Components. User has choice to build using select(), or poll(). creates an mio manager creates an event loop provides Perl callbacks at read events provides access to triggering read, write, or close events

Fig. 1. Sample Process flow for Jabber::MIO How the Module is Invoked Based on creating an echo server that reverses the input. 50, 'timeout' => 30 ); $mio->addListener( 'port' => 5555, 'handler' => sub { my ($e) = @_; my $buf = $e->buffer(); $e->end() if $cnt++ >= 5; $buf =~ s/(\r|\n)//g; return join('',reverse split(//,$buf))."\r\n"; }, ); $mio->start(); ]]> Main Library Encapsulation Module parameter control for library initialisation ( port, etc. ), and Perl wrapper methods Inline code to bind in library functions event loop control/launch insert callback method for accessing Perl subroutine/s Parameter control for library initialisation Bless a new Perl object, and parse all the necessary parameters to to create an MIO object (all standard stuff). 50, 'timeout' => 30, @_ }; bless ($self, $class); # instantiate the mio object my_new_mio($self); return $self; } ]]> Call out to Inline code to set up the mio instance ( just a struct ) and tie that to the Perl object. Inline code to bind in library functions - binding for new Access the scalar parameter values passed in for listening port, maximum file descriptiors, and timeout frequency Create the mio object Parameter control for library initialisation - continued Add port listeners '5555', @_ }; $self->{'listener'}->{$parms->{'port'}} = $parms; my_add_listener( $self, $parms ); } ]]> expect a parameter called handler that is an anonymous subroutine ref. eg. Inline code to bind in library functions - binding for addListener Tie in each handler subroutine ref to the associated listener port parms = newRV_inc(SvRV(parms)); jm->self = newRV_inc(SvRV(self)); mio_listen(((mio_t*) SvIV(SvRV(sv_mio))),SvIV(sv_port),NULL,actor,(void*)jm); } ]]> Note: the reference to actor - this is the callback framework of mio. Event Loop Control/Launch Trigger the object to enter into the event loop. Trigger the main event loop that generates the file descriptor events Callback wrapper function - the callback framework mio takes a function pointer (actor), and passes in details about the socket and it's event including the struct containing the Perl subroutine ref. we keep a hash table of file descriptors, and data to write to them. This can be accessed by the Perl callback. READ - If we get some data to read then do the callback, else close the socket. WRITE - If we get write event - check the fd hash to see if we have data to write. self), "fds", 3, FALSE)); ... switch(a) { ... case action_READ: if((len = read(fd,buf,1024)) > 0) { buf[len] = '\0'; if (callback(m, fd, buf, jm)) mio_write(m, fd); } else { mio_close(m, fd); } return 1; /* get more read events */ break; case action_WRITE: memset(h_key,0,sizeof(h_key)); sprintf(h_key,"%d",fd); if (hv_exists(hv_fds,h_key,strlen(h_key))){ sv_write = *hv_fetch(hv_fds, h_key, strlen(h_key), FALSE); //printf("writing to %d\n",fd); write(fd,SvPV(sv_write,SvCUR(sv_write)),SvCUR(sv_write)); hv_delete(hv_fds,h_key,strlen(h_key),G_DISCARD); } return 0; /* no more write events please */ break; ... ]]> Callback wrapper function - Invoking Perl Same as for callback in Daemon program push variables onto the stack call the handler subroutine store the return value into the hash for filedescriptor output self); XPUSHs(jm->parms); XPUSHs(newSViv(fd)); XPUSHs(newSVpvf("%s",buffer)); PUTBACK; result = perl_call_pv("Jabber::MIO::handler", G_ARRAY | G_EVAL ); if(SvTRUE(ERRSV)) fprintf(stderr, "perl call errored: %s", SvPV(ERRSV,PL_na)); SPAGAIN; memset(h_key,0,sizeof(h_key)); sprintf(h_key,"%d",fd); hv_fds = (HV*) SvRV(*hv_fetch(SvRV(jm->self), "fds", 3, FALSE)); if ( result > 0 ){ res = sv_mortalcopy(POPs); if (SvTRUE(res)){ SvREFCNT_inc(res); hv_store(hv_fds,h_key, strlen(h_key),res,0); ... ]]> Bless Perl data structure into an object containing the object/function pointer Bless the mio object reference into a package so that method calls can be performed on it. $fd, 'buffer' => $buffer, 'mio' => $self, 'eventhandler' => $parms }; bless($event, "Jabber::MIO::FDEvent"); my @result = &{$parms->{'handler'}}($event); return @result; } ]]> Library objects/function encapsulating module This is where we get access to call/manipulate data in the external library from within the Perl callback. encapsulate library objects/functions for access within Perl (Inline) Bless Perl structure containing pointer into object and pass into Perl This could be done within a separate Module to the parent Module, but is generally easier if it is all rolled into one. Encapsulate library objects/functions for access in Perl Create wrappers for accessing library functions Encapsulate library objects/functions for access in Perl - continued Create corresponding Perl methods {'mio'}, $fd); } sub write { die "Must have 3 arguments to Jabber::MIO::Manager::wite(self, fd, data)\n" unless scalar @_ == 3; my ($self, $fd, $data) = @_; my_mio_write($self->{'mio'}, $fd, $data); } sub close { die "Must have 2 arguments to Jabber::MIO::Manager::close(self, fd)\n" unless scalar @_ == 2; my ($self, $fd) = @_; return my_mio_close($self->{'mio'}, $fd); } ... ]]> Common Issues accessing functions/objects in the calling daemon Perl Header file definition clashes (farm out to your own library) encapsulating C/C++ objects for access in perl using anonymous perl subroutines Threading - build Perl with -Dusemultiplicity Taintedness, and setuid programs - no -e args to Interpreter Examples wu-ftpd - role your own mod_perl for ftp using JabberDs' mio manager Jabber::JAX::Component - using the multithreaded JECL libraries Example 1: Embedding Perl in wu-ftpd Business case - I have had ocassion to need to trigger a business event based on the receipt of a file via ftp. Place callbacks to enable modification of file names being stored or retrieved, and access to the event of a file being uploaded or downloaded roll your own mod_perl for ftp - Embed a Perl Interpreter into the WU-FTPD server daemon playing with filenames store and retrieve - forcing user logouts, and error messages triggering events in perl on file receipts

Fig. 1. The WU-FTPD Server Example 3: Jabber Components with Jabber::JAX::Component Write highspeed components for Jabber in perl using a threaded C++ library Building a multi-threaded toolkit based on JECL generating perl callbacks in an embedded library encapsulating objects of the embedded library for later access using it to create a Publish & Subscribe Component for Jabber Resources Perl: perlembed perlapi perlcall perlguts perlxs perlxstut Inline: The home of Inline Inline::C-Cookbook Inline::C Inline::CPP Jabber: Jabber.Org Jabber Studio - Jabber related Developer Zone JECL - Jabber C++ Component building library WU-FTPD: Home of WU-FTPD Download from here This Talk: The slides and the source code