Managing and distributing packages

Software is distributed on Linux distributions in packages managed by tools appropriately called package managers. Once upon a time those package managers were also used to distribute the packages (ask an old-timer about urpmi some day and enjoy the rant), but now we have tools to manage distributing packages rather than the packages themselves sometimes called updaters. For example, the Red Hat package manager (RPM) is a free software package manager used by Red Hat Enterprise, Fedora Core, SuSE, and Mandrake distributions, among others. A variety of updaters are available for packages produced by RPM (.rpm files) including yum (command line), pirut (GUI) and MaintainingDistributionsWithPackageKit GUI.

Package managers and updaters play a critical role in CollaborativeToolBuilding. For end-users, these tools automate the process of letting you know when updates of the software you use are available, installing those updates, etc. For developers, these tools serve a much more important role; they help us expose and track the relations between our work, refining and refactoring those as we go.

What's in a package?

Package contents vary a lot by Linux distribution, type of package, and best practices of the community to which the developers belong (not to mention the degree to which the developers adhere to those best practices!) A package includes (in a perfect world) everything needed to install, remove, and upgrade the software in question on the distribution in question. More specifically, any one package may include:

• software modules in one of a dizzying array of languages;
• header files;
• configuration files;
• other resource files;
• a list of packages on which this package depends (called dependencies);
• a variety of scripts run at specific times (triggers) that automate the install, upgrade, uninstall processes.

You can use the package manager used to create a package to inquire into a package at varying levels of detail. For .rpm files, rpm --filesbypackage will tell you what files will be installed by a package, but not what scripts are included in a package.

Maintaining distributions with Yum

[edit]

Yum (yellowdog updater modified) is the update layer for distributions using RPM packages (see also PackageManagers for a general discussion of package managers vs. updaters.)

The main configuration file for yum (on fedora) is /etc/yum.conf. To add something to the excludes list (or to start one), see this example YumDotConf. Updates should be run regularly (whether nightly or weekly), although probably not via a cron job.

In general, use yum in upgrade mode rather than update mode, so as not to leave older versions of packages lying about, as this causes real problems. You can also use package-cleanup (see below.)

In addition to yum itself, there are the yum-utils (separate package), and a variety of useful plugins. Make sure you enable plugins in /etc/yum.conf.

• yum-utils: includes package-cleanup, e.g., run package-cleanup --orphans to find 'orphaned' packages. An orphaned package is a package installed from an rpm that no repository currently in your repository list has knowledge of. Usually these are older, redundant versions of packages, but not always. Not all orphans are problems, because not all rpms are in repositories.
• yum-fastestmirror: sorts the mirror list for the fastest mirrors.
• yum-merge-conf: provides --merge-conf command line option,
• yum-remove-with-leaves: removes any unused dependencies brought in by an install 'but not normally removed' (I guess someone was sure that you would want them?) Keeps your system cleaner!
• yum-skip-broken: provides --skip-broken command line option.
• yum-upgrade-helper: "allows yum to erase specific packages on install/update based on an additional metadata file in repositories. It is used to simplify distribution upgrade hangups."
• yum-versionlock: Lets you specify packages as 'locked' - protected from upgrade. Can be used to help yum play well with other package managers.

[back to top]

Maintaining distributions with PackageKit

[edit]

PackageKit is a cross-distribution, cross-architecture updater. There's a yum interface to PackageKit and a command line tool called pkcon so you don't have to dork around learning yet another slow, transient GUI tool. At the date of this update (-- HilaryHolz - 29 Mar 2009), although the situation had improved over the last six months, PackageKit remains too new (version 0.3.14) and sketchy to be worth the work in all:

• but the most polyglot environments
• by the most intrepid administrators

and getting to the actual meaningful documentation (PackageKit Reference Manual) took a significant amount of work which really concerns me for the future viability of the project. Buy a clue, people, transparency, otherwise known as the ability to look under the hood, is make or break for FOSS projects.

I really do like the design, and I will say that I saw a distinct improvement in transparency.

[back to top]

Upgrading Fedora With Yum

[edit]

First, a quick comment on why? Why do a major (core) upgrade of your workstations and/or servers with yum (or analog), rather than starting with a clean install each time?

Gee, I would have thought that was obvious, in all honesty. We've got our servers configured and tuned perfectly for our needs. A major upgrade comes out. That upgrade is, of necessity, one-size-fits-all. Telling us to start all over is, in essence, telling us that someone (or some team) who has never met us, never seen our situation, knows literally nothing about our particular needs knows better than us. To get back to where we were will take some amount of looking through what has changed. I would much, much rather assume that we know what we need, and phase in the new configuration, than assume that someone else knows what we need and load a whole bunch of what is almost surely dancing bear ware on our servers. I am more than happy to serve as a beta deployment platform for certain select packages, but not for all and sundry, including gnuchess (yes, that's GNU Chess). So I'm conservative. I install the base package and the packages we know we need, and leave out the rest until they come to my attention.

Some general comments:

• Read the release notes! Yeah, yeah, yeah, I know, I just took those cheap shots at the folks who developed the release, so why start with this? Listen, this is lovely stuff, and you should always approach new software by trying to understand what the folks who wrote it had in mind. You are installing it on your precious boxes, yes? Trusting your precious work to it, yes? Still think that taking a few minutes to skim through a document that distills the major changes and highlights, nifty new stuff and deprecated (on the chopping block and in danger of disappearing in the future if people don't speak up for it) features is that unreasonable? Glad we had this talk
• use yum upgrade whenever possible, not yum update, so that you don't accumulate cruft
• These instructions assume that you are at least at FC5 (fedora core 5). If you aren't at FC5, you won't have package-cleanup (bummer). See Brandon Hutchinson's Notes to claw your way to FC5, then follow these instructions
• Read RemovingPackages before resorting to rpm --nodeps to remove any packages!

Step by step:

1. Read about common problems.
   1. The Fedora Project provides a comprehensive list of version specific notes
   2. For the upgrade from Fedora 8 to Fedora 9, make sure to add the fedora-updates-newkey.repo file to /etc/yum.repos.d
   3. In addition, there's an additional bug in dircolors in Fedora 9 not fixed as of August 5, 2008, which can be fixed by commenting out the line in /etc/DIR_COLORS that starts with CAPABILITY
   4. please add any additional notes to this list!
   5. if you are relying on 3rd party repositories (livna, adobe, etc.), make sure that they are ready for the upgrade, or the process may fail.
2. What about backups?
   Manual backups are a fairy tale told to excuse one of the most inexcusable oversights in design history; namely that all filesystems don't come with versioning and mirroring built in. Only do live upgrades on systems with mirrored, versioned file systems or development systems with disposable data.
3. What about PreUpgrade?
   Good question. Who knows? Our info is oriented towards servers on which:
   1. we do not run any GUI packages, and in an environment in which
   2. we need to be very cost-sensitive, so we tend not to use GUI tools for systems administration (duh.) PreUpgrade has what seems to be a purely gratuitous GUI interface (it appears to piggy-back in some way on anaconda, so it may have seemed natural at the time, but our rule is to avoid GUI whenever possible for sys admin, and always offer a text alternative. So PreUpgrade is thus far irrelevant. Too bad, as it looked useful.
4. Prep your system.
   1. Review, consolidate information, and remove all .rpmsave and .rpmnew files before and after upgrading. This step is a major part of the benefit you get from doing this process, as it gets you to look at systems you might otherwise not spend time with. Modify the following script for your use to review the changes captured by the .rpmsave and .rpmnew files, then delete them (you may want to look in places other than /etc and /var, for example.)

      for a in $(find /etc /var -name '*.rpm?*'); do diff -u $a ${a%.rpm?*}; done

   2. Take a few moments (I do this until I can't stand it anymore...) to review all the packages that have accumulated on your system and work on RemovingPackages
   3. Clean out the local cache that yum uses:

      yum clean all

5. Switch repositories:
   1. install the packages (rpms) that tell fedora which release you want. These packages are architecture indepedent, so you can get both the release and the release notes by using a wildcard, for example, for Fedora 9:

      rpm -Uhv "ftp://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/Packages/fedora-release-*.noarch.rpm"

   2. switch any 3rd party repositories, as well
6. Use runlevel 3
   If you are not already working at runlevel 3 (and all servers should live at runlevel 3, see the Center for Internet Security Benchmark for one set of reasons why, and DesignPhilosophy for another), now is the time to switch to runlevel 3. At runlevel 3, networking and user accounts are fully enabled, but x windows (and hence all GUI stuff) is not.

   If you are running a graphical desktop environment, log out of it.
   Switch to a text console	ctrl + alt + F1
   log in as root
   Go to runlevel 3	telinit 3

7. Do the upgrade: yum, kernel & Base
   You could do the entire upgrade all at once, but that's very dangerous, as you will trash your system if it dies in the middle. I do it in stages, as follows:

   Upgrade yum and all its bits 'n pieces	yum upgrade "yum*"	This is also a good time to review and install any nifty new yum plugins, such as the yum-upgrade-helper
   Upgrade the kernel	yum upgrade kernel	if you get any 'missing dependencies' errors, read this excellent post so that you really understand how dependencies work and what these errors mean. Then you'll not only be able to confidently untangle and rebuild the relevant parts of your system, but you'll develop your understanding of package managers, which is a very important free software concept.
   Use the Base group to design the base of your system	yum groupupdate Base yum upgrade <base-packages-you-want> yum install <new-base-packages-you-want>	Unfortunately, fedora throws all sorts of stuff into the base that many of us neither want nor need. Review the packages in the Base group to see if you want them, otherwise you'll just be introducing a huge headache.

8. Do the upgrade: chunks
   Now upgrade the rest of the system in bite-sized chunks. Groups are one good way to do that if you have defined your own, upgrading by wildcard by letter is another.

   yum grouplist	will get you a list of installed and available groups. Just be careful how many you upgrade at a time. A couple hundred packages at a time is ok, but 800 is playing without a safety net. Questions: 1. I received errors during installation of some package(s) during a groupupdate. Is this a big deal? Discussion 2. When you do yum groupupdate GROUPNAME on a group that is listed as installed via grouplist and you see only a couple or few packages installed, shouldn't you install those packages by hand to avoid package installation bloat (i.e. installing more packages than we really need, thus causing a mess again?) Discussion
   yum groupinfo GROUPNAME	Provides description and list of mandatory packages, default packages, optional packages, and conditional packages.
   yum groupupdate GROUPNAME	Upgrade a group
   yum groupremove GROUPNAME	groupremove will remove an entire group of packages and any dependencies. See also RemovingPackages.

9. Do the upgrade: package stragglers
   Finally, we upgrade any leftover packages. By stragglers we mean any packages that weren't upgraded by the groupupdate procedure above. Upgrade your stragglers as follows:

   Let's take a look at what stragglers are left	yum check-update	this will show you which packages you have installed that can be updated/upgraded. These should be packages not part of a group, since you just upgraded the groups in the previous steps. You may be surprised to see how many packages are not upgraded by the groupupdate procedure discussed above.
   Get info on a package	yum info PACKAGENAME	If that doesn't give you enough info, follow the link to the package homepage.
   Upgrade a package	yum upgrade PACKAGENAME
   Removing packages you don't want with yum	yum remove PACKAGENAME or yum erase PACKAGENAME	remove will remove a package and any dependencies. Take a very careful look at the dependencies listed. See also RemovingPackages.

10. Clean up
    At some point, you need to research and dispose of orphans. RemovingPackages discusses orphans and what to do about them.
11. Restarting the server.
    Wait! Did you create any new files, logs, etc. during the upgrade procedure that you would like to backup? When you are ready, restart the server

    shutdown -r now

    If your server starts up fine, pat yourself on the back and take a bow! If your server doesn't start up correctly, DontPanic. Grab your favorite towel, take a deep breath, and take a look at DontPanic.ServerRecovery.

Remember where we started - you are trusting your work, your time, your precious code/homework/recipes/finances/etc to this box. All of us who know these systems well got comfortable with them not by being born with the knowledge, nor on the job, nor (unfortunately) by going to university classes, but by learning in stages with and from each other, when we had some need.

See also the Fedora Live Upgrade Special Interest Group, the Yum project wiki and our own YuM topic in the new Free / Open Source Software Web.

[back to top]

Removing Extraneous Packages Will Keep Your Server Happy and Healthy!

[edit]

In the natural course of things, linux servers accumulate both orphans (packages installed from rpms that are not associated with any current repository) and cruft (packages that are not relevant to the mission of the server.) Periodically reviewing and, in most cases, removing, such packages is a good thing (tm). Seriously, though, a review will:

• improve security and decrease vulnerability (see also the Center for Internet Security Benchmarks)
• improve system performance and response times (e.g., removes unneeded daemons, conserves system resources, ...)
• save disk space
• result in fewer and faster updates
• improve system stability (problems with cruft won't affect you)

Orphans

What is an Orphan?

Technically, an orphan (yum calls these extras) is a package installed from an rpm that is not associated with any current repository.

What should I do with Orphans?

That depends on the type of orphan:

1. Packages initially installed from a repository (look at the rpm and yum installation logs, usually located in /var/log). These packages have effectively become orphaned, hence the name. You should research what has happened with the project that is maintaining the package.
   • Sometimes there's a different repository from which you can now get the rpm. If so, add the new repository to your yum configuration. If the package is central to your mission, however, you will probably benefit from learning more about what is going on with the project.
   • Sometimes the package has become deprecated in favor of another package, or included in another package. Continuing to run your production environment using the orphaned package is probably fine for the moment, however, migrating to the newer package should become your top priority.
2. Packages installed from an rpm which did not come from a repository. These packages are effectively extras, hence the name used by yum. Some projects have gotten to the point of providing an rpm but not to the point of providing a repository, and the rpm is not available from any major repository yet (did you check livna?) a It's worth taking a few minutes to check to see if the project has started to provide a repository, or if the rpm has gotten picked up by one of the major repository servers. That way, you'll get any updates as they are scheduled. a Failing that, you might consider providing your own internal repository for your lab and any collaborators. It will greatly ease working with VirtualizatioN, etc. a As a last resort, make sure that any true extras are well documented, as well as the results of the previous steps, so that the next time any members of your team conduct this review, they can pick up where you left off.

Finding Orphans

Depending on which version of which Linux distribution you are using, you may have one or more of the following tools available.

yum list extras	list the packages installed on the system that are not available in any configured yum repository.
package-cleanup --orphans	(package-cleanup is in package yum-utils which you may need to install.) Sometimes, especially when using multiple repositories, a package will get stuck and won't upgrade. Then you'll need to delete the version from the old core and install the version from the new core. You can set showdupesfromrepos to 1 in your yum.conf file temporarily to help with this, especially as just because a package has an old core number in its name does not mean it is not part of the current core (argh).
yum list installed "*fc#*"	especially when doing live upgrades. Replace fc# with the correct value for the old core, e.g., fc8 when upgrading to core 9.) Conversely, not every orphaned package is detected by package-cleanup, so you need both commands we list to catch all orphans.

Cruft

Cruft, like orphans, comes in several flavors. Many kinds of cruft can actually be prevented from accumulating by careful configuration of yum.

Preventing Cruft

Configuring yum to prevent cruft (would love to have similar info contributed for apt-get, etc...)

• obsoletes=1 will prevent obsolete packages from accumulating on your system. If you need to hunt down existing obsolete packages, yum list obsoletes will find them.
• multilib_policy=best will install only the best option for 64 bit machines.
• plugins=1 turns on a variety of plugins that we also use to tune how yum behaves
• we've tried working with the exclude list, but have found that module authors need the flexibility with the dependencies, and it generally works better to let things install and then pull them back out, rather than prevent the installation at all.

Leaves

Leaves are packages installed from rpms that are not depended upon by any other rpms. While being a leaf does not make a package cruft, leaf packages are likely candidates to be cruft, particularly if those leaves are libraries. package-cleanup, in yum-utils, is particularly helpful in identifying leaves that are cruft.

Dancing Bear Ware

Review all the packages that have accumulated on your system:

yum list installed

Research the unfamiliar ones. Do you really want those? Do you need them? Do you have a clue what they do? De-cruftify your OS a bit. Snuggle up to it and get to know it, take loving care of it. Don't let anyone tell you that you are unqualified to know what all that stuff is - it's your computer, not theirs.

yum info < packagename >

Provides a minimal set of information about a package, which is often enough to help you decide. If that doesn't give you enough info, there's a link in the provided info to the package homepage. Follow the link and read more. You can also try running yum erase < packagename > and see if the results are terrifying.

Understanding Dependencies

The dependency system is at the heart of the collaborative development of FOSS. PackageManagers provide a relatively easy way for package developers to specify which package(s) their package depends on. The package manager generates dependency chains from this dependency information, resolving dependency issues as it progresses.

Pay attention to dependency information while removing packages, or you may very easily end up with a dead or malfunctioning server. You may have to override dependency info from time to time (see Circular dependencies), so here are some general guidelines:

• start by running package-cleanup --problems to identify any existing dependency problems in your local rpm database and fixing any problems encountered.
• never, ever override dependencies related to the package manager system itself (rpm, yum, and python, in which the system is implemented.) The exception would be the rare occasion when there's a known problem and a published work-around that is carefully documented as a step-by-step process. You'll know when this situation has occurred because the information will come from official sources.
• be very, very wary of overriding any dependencies related to core system functionality such as mount (mounts filesystems!), etc. Again, if it seems like that is what is needed, spend some time googling to see if you are looking at a known issue. If not, you have probably done something wrong earlier in the process and should consult an expert.

Circular dependencies

Circular dependencies occur when package A depends on package B, but package B depends on package A. You can have a longer loop as well (package A depends on Package B depends on package C which depends on package A). You can remove circular dependencies with rpm --erase --nodeps < packagename >, but you must remove all the packages in the circle.

Removing Packages with Yum

yum erase < whateveryoufound >

(don't use yum -y here!). If yum wants to delete all sorts of stuff for dependencies, look carefully. It doesn't hurt to google the package and any error messages you may be getting with yum erase or yum remove before continuing. Sometimes, it's ok. For example, you may be removing orphans, and what it wants to delete may also be orphans. If not, you may want to let yum delete the packages and then re-install, or you may want to fall back to rpm

Removing Packages with RPM

You should only attempt to use rpm to erase packages if yum can not remove the package due to various errors. Do NOT use this command lightly!

rpm --erase --nodeps < whateveryoufound >

This will erase a package while ignoring dependencies. It is NOT recommended to use rpm in this manner, unless you have checked for dependencies (at least with yum) and there are NO dependencies listed or you have adequately researched the dependencies. You are on your own if you get some wild ideas to just yank packages out with no regard for dependencies.

mea culpa...

Yes, I'm guilty of yanking out packages with no regard for dependencies. It's true. But the internet told me I could do it!?! There might be situations where using rpm --erase --nodeps < packagename > might be warranted, but beware. I used it wholesale once for an upgrade on various packages that didn't seem related to the core install for a server. Needless to say, the server didn't restart after reboot.

What if I've removed too much?

• If you haven't restarted the server yet, you should be able to just reinstall the packages you removed. Please let us know if you went through this experience, and the server did not start.
• If you have restarted your server, and it doesn't boot up correctly, you will probably need to boot off of a CD or recovery disk, and then try to reinstall whatever it is you removed. See Server Recovery - Help! My server won't boot! for more info.

[back to top]

Perl's package managers: cpan and cpanp

[edit]

Perl has its own package managers (see BootstrappingPerl), which should be used in preference to rpm or yum. There are two interfaces (modules) in use right now - a lower layer of abstraction, the CPAN module (MCPAN) and a higher layer of abstraction, the CPANPLUS module (MCPANPLUS) to which the community is migrating, slowly. It's part of a larger migration to 'pure perl' based on a new module abstraction layer (Module::Build). In any case, you should use cpanp for most things, however, for some things you will still need to fall back to perl -MCPAN -e shell.

Some hints about using cpanp.

• use 's reconfigure' to configure cpanp to your taste. I like setting the interface to 'classic', which makes it behave a lot like MCPAN, which is what I am used to.
• you will need to use 'l Whatever::Module' to check if a module is installed, as 'm Whatever::Module' does not have the cool = marker to show whether a module is installed or not like the listing on MCPAN.
• if you are having trouble installing a package, it may be that header files needed to compile the packages are missing from the system. Some linux distributions (redhat/fedora does this) distinguish between production and development versions of packages. If there's an error message that says can't find foobar.h, or no such file foobar.h, install the -devel package that provides foobar.h

[back to top]

[see also ManagingUnix]

---