Creating Debian packages from CPAN.

Introduction

The Perl programming language works on a huge variery of operating systems and computer hardware. A great deal of work has gone into perl to make it cross-platform and its behavior consistent on each platform. Debian, a free, community-developed operating system, uses perl for a variety of tools including essential parts of Debian's base system. Debian has a great deal of perl as its DNA, perhaps that's one reason Debian calls itself the "universal operating system"; (another reason is that it runs on a wide variety of hardware.) Debian also contains a large amount of the CPAN in the form of packages, Debian's way of integrating software into the operating system. This article hopes to provide an overview of the Debian packaging system and how a CPAN module becomes a debian package.

CPAN is upstream from Debian.

Uploading your package to CPAN is just the beginning of its journey to the machines of users, there is a lot to do before someone can run your code. CPAN of course has tools that can install any well-formed perl module onto a machine that can run cpan or cpanplus or has tools to handle tarballs. There are limitations to what CPAN installation tools can do however, namely software dependency resolution, or additionally installing specific software that a perl module relies on, software that may reside beyond the bounds of CPAN. Perl has wrapped so many C libraries and other tools that very often when one installs a module from CPAN, one needs to install a C library with it. CPAN handles perl software dependencies, but it is a huge job tracking down every software dependency and library for every operating system that perl can run on. Enter Debian's package management system which can do just that for debian.

Debian's package management system is a set of tools that work together to build, install, remove, and update packages on Debian or a Debian-based operating system. The set of tools is extensive, from dselect and dpkg to apt and aptitude, with many of these tools written in perl, or prototyped in perl then re-written in C/C++. The most familiar tools are probably apt-get and aptitude, tools that users can use to install and remove software on their system. APT stands for "Advanced Package Tool" and is the tool that does automatic dependency resolution, it goes out and finds whatever library your perl module depends on, gets the appropriate version of that library, and installs it. This task is non-trivial and requires a carefully crafted package to enable these features. It is the package itself that might be considered the atomic unit in debian because it holds the keys to how it gets installed and what additional software it needs to function properly, or what software it conflicts with.

So how does one build a debian package? Well, "there's more than one way to do it", in Debian there are lots of ways to build packages, and plenty of tools to help you do that. The fundamental notion to keep in mind though is that Debian aims to keep the source code of the package pristine. There should be no changes by the package maintainer to the software that is getting packaged. While there are exceptions to this rule, they are only allowed when there are security or other special considerations at hand, like adjusting the resulting binary to the requirements of the File Hierarchy standard. Instructions for how to build and install the package should be in a special directory, called "debian", where all customization is done. To demonstrate package creation I am going to use the module Test::File from CPAN.

Choosing something to package is actually quite important, I'll choose Test::File because I find it useful and have some familiarity with it - two things one needs to generate the interest and motivation required when there are bug reports or new features. Packaging is actually considerable work over time, as a stale package is both a potential security risk and quickly forgotten.

The packaging process

We begin the packaging process with a tool called dh-make-perl. dh-make-perl (the dh stands for Debian Helper) is a wrapper around the cpan tool and a surrogate for dh-make the traditional debian make tool, plus a whole lot more. We call it the same way as we would call cpan, with a module name. It then goes to cpan for the source code of our package, since source code always has to be included in debian, that is one of the basic features of Debian - one can get the source code for nearly any package. Should you have to fix a bug in the source code, try to pass your patch upstream into RT, perl's bug tracker.

Here's an example call to dh-make-perl with a lot of arguments to show some of its features.

Using all of these arguments is not necessary, you can make your call much smaller, but I want to show some of these arguments because they make life a little easier and you may want to use them. Of course the canonical source of dh-make-perl arguments and functions is in its man page, this is good to check on occasion since it has been getting updated recently.[0]

The first argument to dh-make-perl shown above, is the --cpan flag which tells dh-make-perl to go and get the module from cpan as opposed to finding it locally. From the man page; "If neither --cpan nor a directory is given as argument, dh-make-perl tries to create a perl package from the data in ." i.e. the current directory. So if you have a module you want to install locally or for some reason do not want to push up to debian, you can create local debs for your own local machines or mirror, no need to push them downstream as it were.

Next we give the name of our module in the same way we would if we were using cpan, i.e. Example::Name. The --desc switch tells dh-make-perl what to use for debian's short description and the --arch flag is for the architecture. Perl programs are rarely going to be architecture dependent so I am just going to gloss over this issue. It is worth mentioning though that there are two relevant architecture specifications for perl modules in debian; any and all. Using the "--arch any" flag tells the debian autobuilders to build on every supported architecture. Since our perl module is architecture independent, we are going to use "--arch all". This choice has implications as we will see later.

The --version flag provides a way to inform dh-make-perl about the version of the package we are packaging, so in this case it is the current version of Test::File. -e is the email address flag, it wants an email address after it. --dh is a call to specify the version of debhelper you want to use. This is a little tricky because different versions of debhelper create different artefacts, specifically different debian/rules files, and remember that debian/rules is the binary that is a Makefile. So you want to most likely use version 7 for debhelper. To paraphrase the dh-make-perl man page, --dh will set the desired debhelper version. If "ver" is 7, the generated debian/rules file is minimalist, using the auto-mode of debhelper. This minimalist version is what you want, unless you are going to package an XS module or need to do some crazy stuff at build time.

Fortunately we do not have to mess about with our debian/rules file, so I am going to continue discussing the rest of the arguments to dh-make-perl, but I want to say that there is a great deal to discuss regarding debian/rules and you would do well to consider reading about it in the Debian developer's documentation in places like [1] the New Maintainer's Guide. If you are reading this in front of a Debian command line, you can simply do a `aptitude install maint-guide` to get the documentation.

The --requiredeps flag tells dh-make-perl to require perl dependencies, that is to say, if we do not find all the modules needed to build our package, we should fail to build our deb. This is really good because it makes your deb package more portable, all the perl module dependencies will get installed when you install your package on another machine, very convenient. For this call to work you need to have apt-file installed on the machine you are building the package on. apt-file is an excellent tool, written in perl (of course!) it allows you to search for files in Debian packages, even packages that are not installed on your system. This means that apt-file is really the canonical tool to find things in Debian or even Ubuntu packages. A quick example; say we wanted to install libtest-more-perl and we called aptitude to install it thusly,

`aptitude install libtest-more-perl`. Aptitude says;

But we're certain that this fundamental perl module is in debian! Haven't we seen Test::More output in fact? Indeed we have, but this module does not exist on its own. Debian has included it with the package perl-modules because it is such a fundamental tool, and so much else in Debian requires it. Still, when looking for it with `dpkg -L libtest-more-perl` we will see these rather unhelpful results;

But in fact, when we search with `apt-file search Test/More.pm` (which is the format we need to specify since we are looking at the file system) we will find that apt-file finds it for us;

This output tells us that the file Test/More.pm is under /usr/share/perl/5.10.0 and it is in the Debian package perl-modules. This is a handy and reliable way to find if the perl module you are looking for is already packaged in debian. All of these commands were issued on a Debian testing system but should be identical now on debian stable (Lenny).

Finally we pass --build which "builds only a binary package (by calling "fakeroot debian/rules binary") and does not sign the package. It is meant for a quick local install of a package, not for creating a package ready for submission to the Debian archive." So says the man page for dh-make-perl. I like to build the package with dh-make-perl because then certain build problems come to the fore sooner, it is not a requirement to build the package with dh-make-perl however.

Once we have run dh-mke-perl, and we watch all sorts of interesting output fly by, like output from cpan, the test suite of our module, etc. The debhelper build process takes over after cpan has worked its magic and we get a finished two files and a directory when we are done. They are;

File: libtest-file-perl_1.25_all.deb

File: libtest-file-perl_1.25.orig.tar.gz

Dir: Test-File-1.25

The anatomy of a package

At this point doing a `dpkg --contents libtest-file-perl_1.25_all.deb` ought to show this output on our new deb;

While this shows the anatomy of a complete debian package, we will still need to build the deb with the dpkg-buildpackage tool and we need to modify some of the files in the Debian directory. First we'll start by modifying the files in the Debian directory to make sure we have a proper package. The first thing we need to do is to change the name of our directory, Debian has a requirement that says the package name has to be lowercase. So we'll move Test-File to libtest-file-perl-1.25 adding the version of Test-File to the dir name. This format is the standard format for Debian perl packages. While one might say it is not the most beautiful format, it has its strengths. Those strengths are that the format informs the user it is a library package, part of a larger system which might require dependencies. It has the suffix -perl which indicates that it is a perl library. There are a few modules in Debian which are not labeled this way, and there is no absolute law saying you have to call your module this way, but in fact you are doing the user a grave disservice if you don't follow this format, because anyone who is used to Debian or Debian derivatives will search for a module as libfoo-bar-perl and they will not find your module if it is not so labeled.

So once we have moved Test-File-1.25 to libtest-file-perl-1.25 we'll change into that directory and take a look around. We find that it is just like the untarred module from CPAN only with the addition of a Debian directory, we'll take a closer look at the Debian directory now which is at the heart of packaging. According to the New Maintainer's guide[2] "The most important of them are `control', `changelog', `copyright' and 'rules', which are required for all packages. " Let's start by taking a look at the control file; 

1 Source: libtest-file-perl
2 Section: perl
3 Priority: optional 
4 Build-Depends: debhelper (>= 7) 
5 Build-Depends-Indep: perl (>= 5.6.0-12), libtest-manifest-perl (>= 1.14) 
6 Maintainer: Debian Perl Group <pkg-perl-maintainers@lists.alioth.debian.org> 
7 Uploaders: Jeremiah C. Foster <jeremiah@jeremiahfoster.com> 
8 Standards-Version: 3.8.0
9  Homepage: http://search.cpan.org/dist/Test-File/
10  Vcs-Svn: svn://svn.debian.org/pkg-perl/trunk/libtest-file-perl/
11  Vcs-Browser: http://svn.debian.org/viewsvn/pkg-perl/trunk/libtest-file-perl/
12
13  Package: libtest-file-perl
14  Architecture: all
15  Depends: ${perl:Depends}, ${misc:Depends}, libtest-manifest-perl (>= 1.14)
16  Description: Test file attributes with perl.
17   Test::Files provides a collection of test utilities for file attributes.
18   .
19   Some file attributes depend on the owner of the process testing the file in
20   the same way the file test operators do. For instance, root (or super-user or
21   Administrator) may always be able to read files no matter the permissions.
22   .
23   Some attributes don't make sense outside of Unix, either, so some tests
24   automatically skip if they think they won't work on the platform. If you have
25   a way to make these functions work on Windows, for instance, please send me a
26   patch. :)
27   .
28   This description was automagically extracted from the module by dh-make-perl. 

I'll move quickly through the first lines of the control file but I would like to point out lines 4 and 5 where Build-Depends and Build-Depends-Indep are defined. This is where the magic at the core of aptitude lies, and why the entire APT system is so powerful. Here we define the relationships between packages in the operating system and within perl which will be satisfied at build time. These dependencies were calculated by dh-make-perl but there are other mechanisms to do this as well[3], and sometimes we even will need to do this by hand. Looking in the source directory for the package and even the META.yml and Makefile.PL can reveal dependencies that might otherwise be missed. Usually dh-make-perl gets it right however and this is not necessary.

In our Build-Depends line we are saying we depend on debhelper and we will not be able to build our package unless this dependency is satisfied, it is an absolute dependency. The apt system will check automatically for dependencies on your dependencies, so you only specify the dependencies you need for your package, you do not have to rummage around to find out what they depend on. Build-Depends is only for dependencies required to build a binary package on your architecture, if your package needs a special compiler to build for example, then you'll need to specify a "Depends: " field. Since there is so much perl included in debian as essential tools, it is rarely required to create a "Depends: " line.

This is fairly esoteric stuff, and perl largely abstracts the "building" of binaries aware from the perl programmer in the interest of simplicity and ease of use. You can dig into this stuff if you want, there is lots more to learn about building perl both on the perl side and on the Debian side, but since it is a rather large subject area I am going to gloss over the really hairy details and refer you to the Debian policy and your own google prowess to get more info that what I have presented here. You can start with [4].

Most of the other stuff in the debian/control file is pretty self-explanatory; resources for the source code, who was responsible for the package uploading, etc. I would like to direct you to the last line where we see some packaging boilerplate which ought to be removed, i.e. line 28.

If we now turn our attention to debian/copyright we can see the power of Free Software and copyright. The Debian Free Software Guidelines require that a copyright be assigned so that a license can be enforced. Perl is under the Artistic license, a license that has won important legal victories in the United States, and also under the GPL. This dual licensing is effective but only when there is a copyright specified and many perl hackers forget to specify their copyright. I would like to encourage you to document your copyright, even if it you received the copyright by default when you author new code, this makes it easier to package your software. Here is what our copyright file looks like;

     1  Format-Specification:
     2      http://wiki.debian.org/Proposals/CopyrightFormat?action=recall&rev=196
     3  Upstream-Maintainer: brian d foy 
     4  Upstream-Source: http://search.cpan.org/dist/Test-File/
     5  Upstream-Name: Test-File
     6  Disclaimer: This copyright info was automatically extracted
     7      from the perl module. It may not be accurate, so you better
     8      check the module sources in order to ensure the module for its
     9      inclusion in Debian or for general legal information. Please,
    10      if licensing information is incorrectly generated, file a bug
    11      on dh-make-perl.
    12
    13  Files: *
    14  Copyright: brian d foy 
    15  License-Alias: Perl
    16  License: Artistic | GPL-1+
    17
    18  Files: debian/*
    19  Copyright: 2009, Jeremiah C. Foster 
    20  License: Artistic | GPL-1+
    21
    22  License: Artistic
    23      This program is free software; you can redistribute it and/or modify
    24      it under the terms of the Artistic License, which comes with Perl.
    25      On Debian GNU/Linux systems, the complete text of the Artistic License
    26      can be found in `/usr/share/common-licenses/Artistic'
    27
    28  License: GPL-1+
    29      This program is free software; you can redistribute it and/or modify
    30      it under the terms of the GNU General Public License as published by
    31      the Free Software Foundation; either version 1, or (at your option)
    32      any later version.
    33      On Debian GNU/Linux systems, the complete text of the GNU General
    34      Public License can be found in `/usr/share/common-licenses/GPL'

This file is pretty straight-forward. We will remove the boilerplate from lines 6 through 11 and then fill in the exact date of the copyright for the software, in this case we'll have to go to CPAN and find out that it is 2008, but after that we are done with the copyright file.

The compat and watch files play minor roles in our package building drama. The watch file is a tool to check and see if there have been \ any new releases upstream, it gets used by a tool called uscan which allow one to update a new CPAN module into an existing Debian package quickly. The compat file merely is a "compatibility" number for some of the other Debian tools, I'll leave that up to you to explore. 

Building the package with dpkg-buildpackage

Now its time to look at the main build tool for building perl debs, dpkg-buildpackage. There are plenty of build tools in Debian and there seems to be a new one every month. For example there is now one called git-buildpackage now, I like dpkg-buildpackage so that is what I am going to tell you about.

Like every build tool there many potential arguments to pass to dpkg-buildpackage, I call dpkg-buildpackage like this:

What we have right after the call is the flag -r with the word fakeroot right after it, that is the command used to gain root. The -D is for checking conflicts and dependencies which I highly recommend although you can do it without checking dependencies but that would most likely not be portable. Finally, -k and my email address is the key I use to sign the package. 

dpkg-buildpackage is written in perl and if you look at the source you'll see the name Ian Jackson in the copyright section. Ian Jackson is the guy who started debian, he is in fact the Ian part of Debian with his wife Debra being the deb part. You can also see that this file is not very well documented, no pod for example, which is a shame. There are other modules as well being pulled into this one, modules like Dpkg and Dpkg::Version which is useful for checking version numbers of packages. Why won't you find these packages on CPAN? Good question. It is one of my long term goals to expose all these tools to CPAN and get the public to examine them and help with development and documentation. The developers in Debian seem to feel these tools are only relatively interesting to a Debian developer or packager, which may be true, but I suspect it is valuable to have tools that work on such a fundamental level with Debian packages since Debian is so wide-spread. Then people can either use the tools themselves or even devise tools on top of them that might be useful, like the cpan2dist tool in CPANPLUS. I can also see these tools as potentially being useful for a distribution agnostic linux packaging program. In any case, I think Debian should follow the best practices of the perl community either way and make the tools available and I intend to do that work if someone does not beat me to it.

In the meantime, what happened when we built our package? Since we passed -D to check build time dependencies, dpkg-buildpackage called dpkg-checkbuilddeps and found that we cannot build our package because we are missing a dependency; Test::Manifest. You can run dpkg-checkbuilddeps separately and this is the output;

The above line tells us that the Perl modules Test::Manifest needs to be included in our operating system for us to build our package and that Test::Manifest already exists in Debian as the package libtest-manifest-perl. Marvel at the power off the apt system! It saved us a journey to dependency hell. We simply install libtest-manifest-perl and try to build again . . .

This time, success! dpkg-buildpackage will ask me for my key passphrase, which I give it, and it signs the package for me. Now if we look in our dir we have;

We have our deb, signed and sealed. You can install it now with dpkg -i libtest-file-perl_1.25-1_all.deb but before we pass it out far and wide, let's take one final step and build it in a "clean room" or a minimal Debian install. This we can use as a baseline and assume that if it builds and installs in the clean room, it can build and install anywhere. To do this we are going to use pbuilder which is a "personal package builder". It creates a chroot, downloads a minimal Debian install, adds your package and any dependencies and builds a deb for you. If that works, you can be reasonably sure it will work out in the greater wide world of the Debian installed base. 

Here is the call; 

I will go through an arbitrary selection of pbuilder's output;

The base tarball gets unpackaged to create the build environment.

Here pbuilder updates the base Debian install with the latest diffs of packages so your clean room is up-to-date. You can update it manually as well and change with distribution you want to use, I prefer to use testing but you might want to use stable.

pbuilder pulls in our source for the package.

dpkg-buildpackage takes over and does its stuff.

Aha! I missed a useful tool. Since Test::Pod gets called while running tests, I should add it to Build-Depends-Indep in the debian/control file to get these tests to run. Of course it build without it, but it is better to run all our tests like the original developer envisioned. Once I add that module and the module Test::Pod::Coverage which is also used in tests to the debian/control file, all the tests pass and the package gets built. This is a pretty good indication that this package will build on someone else's debian machine.

To confirm that we are in accordance with debian policy we ought to run the package through lintian, the Debian policy checker. I run it with the -i and -I flags which provides much more verbose output, it has a --pedantic switch as well.  We might run it against our deb like this;

And get output like this;

These warnings are good to have, were you to submit your package for inclusion in Debian the expectation is that your package is "lintian clean" which means without warnings from lintian. Now we can submit this to Debian or put it in our own personal deb repo with confidence.

The package goes through some automatic building on a variety of architectures, sits in a queue for about ten days in normal circumstances, then gets put into the Debian "testing" distro. Anyone who has Debian testing sources in their /etc/apt/sources.list will now be able to install it just by calling aptitude and the name of the package. Now your package or software is available to millions of users. Congratulations.

0. http://git.debian.org/?p=pkg-perl/apps/dh-make-perl.git;a=summary dh-make-perl git repo

1. http://www.debian.org/doc/maint-guide/ch-dreq.en.html#s-rules debian's rules file

2. http://www.debian.org/doc/maint-guide/ch-dreq.en.html New Maintainer's guide

3. See apt-cache search depends and rdepends

4. http://www.debian.org/doc/debian-policy/ch-relationships.html

5. http://www.debian.org/doc/debian-policy/ch-binary.html#s3.5

http://www.debian.org/doc/packaging-manuals/perl-policy/ debian's perl policy

http://pkg-perl.alioth.debian.org/ Debian perl team