=head1 NAME PAR::Tutorial - Cross-Platform Packaging and Deployment with PAR =head1 SYNOPSIS This is a tutorial on PAR, first appeared at the 7th Perl Conference. The HTML version of this tutorial is available online as L =head1 DESCRIPTION =head2 On Deploying Perl Applications % sshnuke.pl 10.2.2.2 -rootpw="Z1ON0101" Perl v5.6.1 required--this is only v5.6.0, stopped at sshnuke.pl line 1. BEGIN failed--compilation aborted at sshnuke.pl line 1. =over 4 =item * Q: "Help! I can't run your program!" =item * A1: Install Perl & C =over 4 =item * How do we know which modules are needed? =item * New versions of CPAN modules may break C =back =item * A2: Install Perl & C =over 4 =item * Possibly overwriting existing modules; not cross-platform at all =back =item * A3: Use the executable generated by C =over 4 =item * Impossible to debug; C usually does not work anyway =back =back =head2 PAR, the Perl Archive Toolkit =over 4 =item * Do what JAR (Java Archive) does for Perl =over 4 =item * Aggregates modules, scripts and other files into a Zip file =item * Easy to generate, update and extract =item * Version consistency: solves forward-compatibility problems =item * Developed by community: C =back =item * PAR files can be packed into self-contained scripts =over 4 =item * Automatically scans perl script for dependencies =item * Bundles all necessary 3rd-party modules with it =item * Requires only core Perl to run on the target machine =item * PAR also comes with C, the Perl Packager: % pp -o sshnuke.exe sshnuke.pl # stand-alone executable! =back =back =head2 Simple Packaging =over 4 =item * PAR files are just Zip files with modules in it =item * Any Zip tools can generate them: % zip foo.par Hello.pm World.pm # pack two modules % zip -r bar.par lib/ # grab all modules in lib/ =item * To load modules from PAR files: use PAR; use lib "foo.par"; # the .par part is optional use Hello; =item * This also works: use PAR "/home/mylibs/*.par"; # put all of them into @INC use Hello; =back =head2 PAR Loaders =over 4 =item * Use C to run files inside a PAR archive: % par.pl foo.par # looks for 'main.pl' by default % par.pl foo.par test.pl # runs script/test.pl in foo.par =item * Same thing, with the stand-alone C or C: % parl foo.par # no perl or PAR.pm needed! % parl foo.par test.pl # ditto =item * The PAR loader can prepend itself to a PAR file: =over 4 =item * C<-b> bundles non-core modules needed by C: % par.pl -b -O./foo.pl foo.par # self-contained script =item * C<-B> bundles core modules in addition to C<-b>: % parl -B -O./foo.exe foo.par # self-contained binary =back =back =head2 Dependency Scanning =over 4 =item * Recursively scan dependencies with C: % scandeps.pl sshnuke.pl # Legend: [C]ore [X]ternal [S]ubmodule [?]NotOnCPAN 'Crypt::SSLeay' => '0', # X # 'Net::HTTP' => '0', # # 'Crypt::SSLeay::X509' => '0', # S # Crypt::SSLeay 'Net::HTTP::Methods' => '0', # S # Net::HTTP 'Compress::Zlib' => '0', # X # Net::HTTP::Methods =item * Scan an one-liner, list all involved files: % scandeps.pl -V -e "use Dynaloader;" ... # auto/DynaLoader/dl_findfile.al [autoload] # auto/DynaLoader/extralibs.ld [autoload] # auto/File/Glob/Glob.bs [data] # auto/File/Glob/Glob.so [shared] ... =back =head2 Perl Packager: C =over 4 =item * Combines scanning, zipping and loader-embedding: % pp -o out.exe src.pl # self-contained .exe % out.exe # runs anywhere on the same OS =item * Bundle additional modules: % pp -o out.exe -M CGI src.pl # pack CGI + its dependencies, too =item * Pack one-liners: % pp -o out.exe -e 'print "Hi!"' # turns one-liner into executable =item * Generate PAR files instead of executables: % pp -p src.pl # makes 'source.par' % pp -B -p src.pl # include core modules =back =head2 How it works =over 4 =item * Command-line options are almost identical to C's =over 4 =item * Also supports C-style long options: % pp --gui --verbose --output=out.exe src.pl =back =item * Small initial overhead; no runtime overhead =item * Dependencies are POD-stripped before packing =item * Loads modules directly into memory on demand =item * Shared libraries (DLLs) are extracted with File::Temp =item * Works on Perl 5.6.0 or above =item * Tested on Win32 (VC++ and MinGW), FreeBSD, NetBSD, Linux, MacOSX, Cygwin, AIX, Solaris, HP-UX, Tru64... =back =head2 Aggregating multiple programs =over 4 =item * A common question: > I have used pp to make several standalone applications which work > great, the only problem is that for each executable that I make, I am > assuming the parl.exe is somehow bundled into the resulting exe. =item * The obvious workaround: You can ship parl.exe by itself, along with .par files built by "pp -p", and run those PAR files by associating them to parl.exe. =item * On platforms that have C, there is a better solution: % pp --output=a.out a.pl b.pl # two scripts in one! % ln a.out b.out # symlink also works % ./a.out # runs a.pl % ./b.out # runs b.pl =back =head2 Cross-platform Packages =over 4 =item * Of course, there is no cross-platform binary format =item * Pure-perl PAR packages are cross-platform by default =over 4 =item * However, XS modules are specific to Perl version and platform =item * Multiple versions of a XS module can co-exist in a PAR file =back =item * Suppose we need C on both Win32 and Finix: C:\> pp --multiarch --output=out.par src.pl ...copy src.pl and out.par to a Finix machine... % pp --multiarch --output=out.par src.pl =item * Now it works on both platforms: % parl out.par # runs src.pl % perl -MPAR=out.par -e '...' # uses modules inside out.par =back =head2 The Anatomy of a PAR file =over 4 =item * Modules can reside in several directories: / # casual packaging only /lib/ # standard location /arch/ # for creating from blib/ /i386-freebsd/ # i.e. $Config{archname} /5.8.0/ # i.e. Perl version number /5.8.0/i386-freebsd/ # combination of the two above =item * Scripts are stored in one of the two locations: / # casual packaging only /script/ # standard location =item * Shared libraries may be architecture- or perl-version-specific: /shlib/(5.8.0/)?(i386-freebsd/)? =item * PAR files may recursively contain other PAR files: /par/(5.8.0/)?(i386-freebsd/)? =back =head2 Special files =over 4 =item * MANIFEST =over 4 =item * Index of all files inside PAR =item * Can be parsed with C =back =item * META.yml =over 4 =item * Dependency, license, runtime options =item * Can be parsed with C =back =item * SIGNATURE =over 4 =item * OpenPGP-signed digital signature =item * Can be parsed and verified with C =back =back =head2 Advantages over perlcc, PerlApp and Perl2exe =over 4 =item * This is not meant to be a flame =over 4 =item * All three maintainers have contributed to PAR directly; I'm grateful =back =item * perlcc =over 4 =item * "The code generated in this way is not guaranteed to work... Use for production purposes is strongly discouraged." (from perldoc perlcc) =item * I is more like it =back =item * PerlApp / Perl2exe =over 4 =item * Expensive: Need to pay for each upgrade =item * Non-portable: Only available for limited platforms =item * Proprietary: Cannot extend its features or fix bugs =item * Obfuscated: Vendor and black-hats can see your code, but you can't =item * Inflexible: Does not work with existing Perl installations =back =back =head2 MANIFEST: Best viewed with Mozilla =over 4 =item * The URL of C inside C: jar:file:///home/autrijus/foo.par!/MANIFEST =item * Open it in a Gecko browser (e.g. Netscape 6+) with Javascript enabled: =item * No needed to unzip anything; just click on files to view them =back =head2 META.yml: Metadata galore =over 4 =item * Static, machine-readable distribution metadata =over 4 =item * Supported by C, C, C =back =item * A typical C-generated C looks like this: build_requires: {} conflicts: {} dist_name: out.par distribution_type: par dynamic_config: 0 generated_by: 'Perl Packager version 0.03' license: unknown par: clean: 0 signature: '' verbatim: 0 version: 0.68 =item * The C settings controls its runtime behavior =back =head2 SIGNATURE: Signing and verifying packages =over 4 =item * OpenPGP clear-signed manifest with SHA1 digests =over 4 =item * Supported by C, C and C =back =item * A typical C looks like this: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 SHA1 8a014cd6d0f6775552a01d1e6354a69eb6826046 AUTHORS ... -----BEGIN PGP SIGNATURE----- ... -----END PGP SIGNATURE----- =item * Use C and C to work with signatures: % pp -s -o foo.par bar.pl # make and sign foo.par from bar.pl % cpansign -s foo.par # sign this PAR file % cpansign -v foo.par # verify this PAR file =back =head2 Perl Servlets with Apache::PAR =over 4 =item * Framework for self-contained Web applications =over 4 =item * Similar to Java's "Web Application Archive" (WAR) files =item * Works with mod_perl 1.x or 2.x =back =item * A complete web application inside a C<.par> file =over 4 =item * Apache configuration, static files, Perl modules... =item * Supports Static, Registry and PerlRun handlers =item * Can also load all PARs under a directory =back =item * One additional special file: C Alias /myapp/cgi-perl/ ##PARFILE##/ Options +ExecCGI SetHandler perl-script PerlHandler Apache::PAR::Registry =back =head2 Hon Dah, A-par-che! =over 4 =item * First, make a C from an one-liner: # use the "web.conf" from the previous slide % pp -p -o hondah.par -e 'print "Hon Dah!\n"' \ --add web.conf % chmod a+x hondah.par =item * Add this to C, then restart apache: PerlModule Apache2 PerlAddVar PARInclude /home/autrijus/hondah.par PerlModule Apache::PAR =item * Test it out: % GET http://localhost/myapp/cgi-perl/main.pl Hon Dah! =item * Instant one-liner web application that works! =back =head2 On-demand library fetching =over 4 =item * With LWP installed, your can use remote PAR files: use PAR; use lib 'http://aut.dyndns.org/par/DBI-latest.par'; use DBI; # always up to date! =item * Modules are now cached under C<$ENV{PAR_GLOBAL_TEMP}> =item * Auto-updates with C =over 4 =item * Download only if modified =item * Safe for offline use after the first time =item * May use C to prevent DNS-spoofing =back =item * Makes large-scale deployment a breeze =over 4 =item * Upgrades from a central location =item * No installers needed =back =back =head2 Code Obfuscation =over 4 =item * Also known as I techniques =over 4 =item * It is I encryption =item * Offered by PerlApp, Perl2Exe, Stunnix... =back =item * Usually easy to defeat =over 4 =item * Take optree dump from memory, feed to C =item * If you just want to stop a casual C, "deflate" already works =back =item * PAR now supports pluggable I with C =over 4 =item * Bundled examples: Bleach, PodStrip and PatchContent =item * True encryption using C =item * Or even _product activation_ over the internet =back =item * Alternatively, just keep core logic in your server and use RPC =back =head2 Accessing packed files =over 4 =item * To get the host archive from a packed program: my $zip = PAR::par_handle($0); # an Archive::Zip object my $content = $zip->contents('MANIFEST'); =item * Same thing, but with C: my $content = PAR::read_file('MANIFEST'); =item * Loaded PAR files are stored in C<%PAR::LibCache>: use PAR '/home/mylibs/*.par'; while (my ($filename, $zip) = each %PAR::LibCache) { print "[$filename - MANIFEST]\n"; print $zip->contents('MANIFEST'); } =back =head2 Packing GUI applications =over 4 =item * GUI toolkits often need to link with shared libraries: # search for libncurses under library paths and pack it % pp -l ncurses curses_app.pl # same for Tk, Wx, Gtk, Qt... =item * Use C on Win32 to eliminate the console window: # pack 'src.pl' into a console-less 'out.exe' (Win32 only) % pp --gui -o out.exe src.pl =item * "Can't locate Foo/Widget/Bar.pm in @INC"? =over 4 =item * Some toolkits (notably Tk) autoloads modules without C or C =item * Hence C and C may fail to detect them =item * Tk problems mostly fixed by now, but other toolkits may still break =item * You can work around it with C or an explicit C =item * Or better, send a short test-case to C so we can fix it =back =back =head2 Precompiled CPAN distributions =over 4 =item * Installing XS extensions from CPAN was difficult =over 4 =item * Some platforms do not come with a compiler (Win32, MacOSX...) =item * Some headers or libraries may be missing =item * PAR.pm itself used to suffer from both problems =back =item * ...but not anymore -- C to the rescue! # same old Makefile.PL, with a few changes use inc::Module::Install; # was "use ExtUtils::MakeMaker;" WriteMakefile( ... ); # same as the original check_nmake(); # make sure the user have nmake par_base('AUTRIJUS'); # your CPAN ID or a URL fetch_par() unless can_cc(); # use precompiled PAR only if necessary =item * Users will not notice anything, except now it works =over 4 =item * Of course, you still need to type C and upload the precompiled package =item * PAR users can also install it directly with C =back =back =head2 Platform-specific Tips =over 4 =item * Win32 and other icon-savvy platforms =over 4 =item * Needs 3rd-party tools to add icons to C-generated executables =item * PE Header manipulation in Perl -- volunteers wanted! =back =item * Linux and other libc-based platforms =over 4 =item * Try to avoid running C on a bleeding-edge version of the OS =item * Older versions with an earlier libc won't work with new ones =back =item * Solaris and other zlib-lacking platforms (but not Win32) =over 4 =item * You need a static-linked C before installing PAR =item * In the future, PAR may depend on C instead =back =item * Any platform with limited bandwidth or disk space =over 4 =item * Use UPX to minimize the executable size =back =back =head2 Thank you! =over 4 =item * Additional resources =over 4 =item * Mailing list: C =item * Subscribe: Send a blank email to C =item * List archive: L =item * PAR::Intro: L =item * Apache::PAR: L =item * Module::Install: L =back =item * Any questions? =back =head2 Bonus Slides: PAR Internals =head2 Overview of PAR.pm's Implementation =over 4 =item * Here begins the scary part =over 4 =item * Grues, Dragons and Jabberwocks abound... =item * You are going to learn weird things about Perl internals =back =item * PAR invokes four areas of Perl arcana: =over 4 =item * @INC code references =item * On-the-fly source filtering =item * Overriding C to handle XS modules =item * Making self-bootstrapping binary executables =back =item * The first two only works on 5.6 or later =over 4 =item * DynaLoader and C<%INC> are there since Perl 5 was born =item * PAR currently needs 5.6, but a 5.005 port is possible =back =back =head2 Code References in @INC =over 4 =item * On 1999-07-19, Ken Fox submitted a patch to P5P =over 4 =item * To _enable using remote modules_ by putting hooks in @INC =item * It's accepted to come in Perl 5.6, but undocumented until 5.8 =item * Type C to read the nitty-gritty details =back =item * Coderefs in @INC may return a fh, or undef to 'pass': push @INC, sub { my ($coderef, $filename) = @_; # $coderef is \&my_sub open my $fh, "wget ftp://example.com/$filename |"; return $fh; # using remote modules, indeed! }; =item * Perl 5.8 let you open a file handle to a string, so we just use that: open my $fh, '<', \($zip->memberNamed($filename)->contents); return $fh; =item * But Perl 5.6 does not have that, and I don't want to use temp files... =back =head2 Source Filtering without Filter::* Modules =over 4 =item * ... Undocumented features to the rescue! =over 4 =item * It turns out that @INC hooks can return B values =item * The first is still the file handle =item * The second is a code reference for line-by-line source filtering! =back =item * This is how C works: # Force all modules used to use strict and warnings open my $fh, "<", $filename or return; my @lines = ("use strict; use warnings;\n", "#line 1 \"$full\"\n"); return ($fh, sub { return 0 unless @lines; push @lines, $_; $_ = shift @lines; return length $_; }); =back =head2 Source Filtering without Filter::* Modules (cont.) =over 4 =item * But we don't really have a filehandle for anything =item * Another undocumented feature saves the day! =item * We can actually omit the first return value altogether: # Return all contents line-by-line from the file inside PAR my @lines = split( /(?<=\n)/, $zip->memberNamed($filename)->contents ); return (sub { $_ = shift(@lines); return length $_; }); =back =head2 Overriding DynaLoader::bootstrap =over 4 =item * XS modules have dynamically loaded libraries =over 4 =item * They cannot be loaded as part of a zip file, so we extract them out =item * Must intercept DynaLoader's library-finding process =back =item * Module names are passed to C for XS loading =over 4 =item * During the process, it calls C to locate the file =item * So we install pre-hooks around both functions =back =item * Our C<_bootstrap> just checks if the library is in PARs =over 4 =item * If yes, extract it to a C temp file =over 4 =item * The file will be automatically cleaned up when the program ends =back =item * It then pass the arguments to the original C =item * Finally, our C intercepts known filenames and return it =back =back =head2 Anatomy of a Self-Contained PAR executable =over 4 =item * The par script ($0) itself =over 4 =item * May be in plain-text or native executable format =back =item * Any number of embedded files =over 4 =item * Typically used to bootstrap PAR's various dependencies =item * Each section begins with the magic string "FILE" =item * Length of filename in pack('N') format and the filename (auto/.../) =item * File length in pack('N') and the file's content (not compressed) =back =item * One PAR file =over 4 =item * Just a regular zip file with the magic string C<"PK\003\004"> =back =item * Ending section =over 4 =item * A pack('N') number of the total length of FILE and PAR sections =item * Finally, there must be a 8-bytes magic string: C<"\012PAR.pm\012"> =back =back =head2 Self-Bootstrapping Tricks =over 4 =item * All we can expect is a working perl interpreter =over 4 =item * The self-contained script *must not* use any modules at all =item * But to process PAR files, we need XS modules like Compress::Zlib =back =item * Answer: bundle all modules + libraries used by PAR.pm =over 4 =item * That's what the C section in the previous slide is for =item * Load modules to memory, and write object files to disk =item * Then use a local C<@INC> hook to load them on demand =back =item * Minimizing the amount of temporary files =over 4 =item * First, try to load PerlIO::scalar and File::Temp =item * Set up an END hook to unlink all temp files up to this point =item * Load other bundled files, and look in the compressed PAR section =item * This can be much easier with a pure-perl C; patches welcome! =back =back =head2 Thank you (again)! =over 4 =item * Any questions, I? =back =cut =head1 SEE ALSO L, L, L, L L, L L, L, L, L =head1 AUTHORS Audrey Tang Ecpan@audreyt.orgE You can write to the mailing list at Epar@perl.orgE, or send an empty mail to Epar-subscribe@perl.orgE to participate in the discussion. Please submit bug reports to Ebug-par@rt.cpan.orgE. =head1 COPYRIGHT Copyright 2003, 2004, 2005, 2006 by Audrey Tang Ecpan@audreyt.orgE. This document is free documentation; you can redistribute it and/or modify it under the same terms as Perl itself. See F. =cut