Managing Unix

Installing Fedora
- Installing Fedora 10 via Live CD
Managing and distributing packages
Managing Security
Death By Networking
- DHCP and pocket research lab servers
- static networking setup

having problems? see DontPanic
need to RtFM? We can help

The Ahat lab, which hosts this wiki, uses a rich set of platforms. Our professional work, however, tends towards a blend of Unixes. The big servers are all on Fedora, mostly on core 9. We tend towards bleeding edge webserver/language/interaction research and development rather than Unix kernel development itself, so walking the knife-edge of being on the most recent truly stable Fedora core (i.e., lagging one "production" core behind) has given us the best trade-off of bleeding edge agility vs. bleeding edge instability. We've played with Debian (and some others), but find ourselves back with Fedora. It's not perfect, but it's home...

Personally, many of us use Darwin, Apple's BSD-style Unix OS, as Mac laptops are the laptop of choice (for those with the funding to afford them.) Folks with windows laptops tend to start by installing Ubuntu on their laptops. As a starter Linux, Ubuntu is the clear current winner, and it's an easy knowledge migration from Ubuntu to Fedora. When we need to test things on Windows, we tend to rely on VirtualizatioN. It's a digital inclusion thing; we find that the payoff from putting our limited resources into FOSS has a better return.

Fedora is the free software project companion to Red Hat Linux, Enterprise edition. The information here tends towards fedora, largely due to the tools available for managing servers for scientific programming. We would welcome a companion set of materials about how to achieve the same or similar effects with other platforms.

Installing Fedora

[edit]

There are a rich variety of ways to install Fedora (as well as many other distributions of Linux.) Essentially they all break down to the following steps:

Get the files needed in a bootable configuration
Prepare your system for installation
Boot your computer (see step 1) and install
Reboot your computer and configure.

Fedora's New Users Chapter of the install guide does a pretty good job of discussing the options and when you would use which. We discuss installing from a live CD in detail, simply because we captured lots of notes on the topic since we started keeping this primer. As we progress we promise to expand our set of pocket lab-oriented info.

Installing Fedora 10 via Live CD

[edit]

One of the ways to install some Linux distributions is via a live CD. A live CD is a bootable CD from which you can install the distribution. Why install from a live CD? It comes with less crud. But it is harder to install. See Fedora install guide for f10.

Download the Fedora 10 Live CD from the fedora website. Burned the iso file to a CD. Check the sha1.
Pop in the CD. When you are presented a message similar to "Loading in 9 seconds" and the numbers count down, press any key and select Verify and Boot to verify the CD and then boot from it.
Let it boot up. When you are in X windows, double-click on the icon Install to Hard Drive to start the Fedora Installer.

partitioning the hard drives & installing fedora:

select remove all partitions on selected drives and create default layout, and click at the bottom to choose Review and modify partitioning layout for a custom partition set up. The UI is hard to work with, so we actually had to do this twice.
why should you partition by hand? there are many good reasons, including increased security, building a more robust system, and a system that can handle new fedora installations without losing or needing to reinstall /home or /var directories, for example.
look in the CIS documents to see recommendations for partitions and their respective sizes

the CIS recommendations may be overkill for some. On the installation we ran during this write-up we choose to set partitions for:

`boot`	100MB	not part of the volume group
`/`	16512MB
swap	2048MB	this one actually isnt' called swp, it's the one without the mount point
`/usr`	10240MB
`/var`	4096MB
`/home`	4096MB
`/tmp`	1024MB

swap partition size? RHEL recommends...
- 2x the size of your memory (if total memory is <2GB)
- 1x the size of your memory for memory above 2GB
- example: if your total is 2GB, recommendation is 4GB swap; if your total memory is 3GB, recommendation is 5GB swap.
- note: depending on what you are running on your server, these are mainly guidelines. You may want more or less.
when you are ready, click on edit, and shrink down the boot to 100MB.
click on VolGroup00 and click on edit, and a new window will pop up
- edit the big one ... take off the final digit to make room for the new partitions
- then add and adjust the numbers as you need them for your system setup
- when ready you will be able to begin the formatting process, and fedora will reflect formating on each partition.
when the formatting is complete, you will see a window with...
- a checked checkbox "Install boot loader on /dev/sda"
- an unchecked checkbox "Use boot loader password"
- a menu "Boot Loader operating system list" and on entry that is checked "Fedora /dev/VolGroup00/LogVol00". This may be a little confusing because you haven't installed the OS yet, but this is the default, and if you try to delete it, the installer will inform you. We left this menu as we have described it here, and clicked next.
- the fedora installer will begin to install the OS. This can take a while. Don't panic if the screen saver goes on and you want to go back in X and windows look like they've frozen. Just give it some time.
- you will be prompted when it is done, and you will need to reboot or shutdown. You can then eject the CD.

post installation:

turn the computer back on if you shut it down
after booting up, fedora will make you go through some setup screens
once you are in, open a terminal
- use su to setup sudo

[back to top]

Managing and distributing packages

[edit]

Software is distributed on Linux distributions in packages managed by tools appropriately called package managers. Once upon a time those package managers were also used to distribute the packages (ask an old-timer about urpmi some day and enjoy the rant), but now we have tools to manage distributing packages rather than the packages themselves sometimes called updaters. For example, the Red Hat package manager (RPM) is a free software package manager used by Red Hat Enterprise, Fedora Core, SuSE, and Mandrake distributions, among others. A variety of updaters are available for packages produced by RPM (.rpm files) including yum (command line), pirut (GUI) and MaintainingDistributionsWithPackageKit GUI.

Package managers and updaters play a critical role in CollaborativeToolBuilding. For end-users, these tools automate the process of letting you know when updates of the software you use are available, installing those updates, etc. For developers, these tools serve a much more important role; they help us expose and track the relations between our work, refining and refactoring those as we go.

What's in a package?

Package contents vary a lot by Linux distribution, type of package, and best practices of the community to which the developers belong (not to mention the degree to which the developers adhere to those best practices!) A package includes (in a perfect world) everything needed to install, remove, and upgrade the software in question on the distribution in question. More specifically, any one package may include:

software modules in one of a dizzying array of languages;
header files;
configuration files;
other resource files;
a list of packages on which this package depends (called dependencies);
a variety of scripts run at specific times (triggers) that automate the install, upgrade, uninstall processes.

You can use the package manager used to create a package to inquire into a package at varying levels of detail. For .rpm files, rpm --filesbypackage will tell you what files will be installed by a package, but not what scripts are included in a package.

Maintaining distributions with Yum

[edit]

Yum (yellowdog updater modified) is the update layer for distributions using RPM packages (see also PackageManagers for a general discussion of package managers vs. updaters.)

The main configuration file for yum (on fedora) is /etc/yum.conf. To add something to the excludes list (or to start one), see this example YumDotConf. Updates should be run regularly (whether nightly or weekly), although probably not via a cron job.

In general, use yum in upgrade mode rather than update mode, so as not to leave older versions of packages lying about, as this causes real problems. You can also use package-cleanup (see below.)

In addition to yum itself, there are the yum-utils (separate package), and a variety of useful plugins. Make sure you enable plugins in /etc/yum.conf.

yum-utils: includes package-cleanup, e.g., run package-cleanup --orphans to find 'orphaned' packages. An orphaned package is a package installed from an rpm that no repository currently in your repository list has knowledge of. Usually these are older, redundant versions of packages, but not always. Not all orphans are problems, because not all rpms are in repositories.
yum-fastestmirror: sorts the mirror list for the fastest mirrors.
yum-merge-conf: provides --merge-conf command line option,
yum-remove-with-leaves: removes any unused dependencies brought in by an install 'but not normally removed' (I guess someone was sure that you would want them?) Keeps your system cleaner!
yum-skip-broken: provides --skip-broken command line option.
yum-upgrade-helper: "allows yum to erase specific packages on install/update based on an additional metadata file in repositories. It is used to simplify distribution upgrade hangups."
yum-versionlock: Lets you specify packages as 'locked' - protected from upgrade. Can be used to help yum play well with other package managers.

[back to top]

Maintaining distributions with PackageKit

[edit]

PackageKit is a cross-distribution, cross-architecture updater. There's a yum interface to PackageKit and a command line tool called pkcon so you don't have to dork around learning yet another slow, transient GUI tool. At the date of this update (-- HilaryHolz - 29 Mar 2009), although the situation had improved over the last six months, PackageKit remains too new (version 0.3.14) and sketchy to be worth the work in all:

but the most polyglot environments
by the most intrepid administrators

and getting to the actual meaningful documentation (PackageKit Reference Manual) took a significant amount of work which really concerns me for the future viability of the project. Buy a clue, people, transparency, otherwise known as the ability to look under the hood, is make or break for FOSS projects.

I really do like the design, and I will say that I saw a distinct improvement in transparency.

[back to top]

Upgrading Fedora With Yum

[edit]

First, a quick comment on why? Why do a major (core) upgrade of your workstations and/or servers with yum (or analog), rather than starting with a clean install each time?

Gee, I would have thought that was obvious, in all honesty. We've got our servers configured and tuned perfectly for our needs. A major upgrade comes out. That upgrade is, of necessity, one-size-fits-all. Telling us to start all over is, in essence, telling us that someone (or some team) who has never met us, never seen our situation, knows literally nothing about our particular needs knows better than us. To get back to where we were will take some amount of looking through what has changed. I would much, much rather assume that we know what we need, and phase in the new configuration, than assume that someone else knows what we need and load a whole bunch of what is almost surely dancing bear ware on our servers. I am more than happy to serve as a beta deployment platform for certain select packages, but not for all and sundry, including gnuchess (yes, that's GNU Chess). So I'm conservative. I install the base package and the packages we know we need, and leave out the rest until they come to my attention.

Some general comments:

Read the release notes! Yeah, yeah, yeah, I know, I just took those cheap shots at the folks who developed the release, so why start with this? Listen, this is lovely stuff, and you should always approach new software by trying to understand what the folks who wrote it had in mind. You are installing it on your precious boxes, yes? Trusting your precious work to it, yes? Still think that taking a few minutes to skim through a document that distills the major changes and highlights, nifty new stuff and deprecated (on the chopping block and in danger of disappearing in the future if people don't speak up for it) features is that unreasonable? Glad we had this talk
use yum upgrade whenever possible, not yum update, so that you don't accumulate cruft
These instructions assume that you are at least at FC5 (fedora core 5). If you aren't at FC5, you won't have package-cleanup (bummer). See Brandon Hutchinson's Notes to claw your way to FC5, then follow these instructions
Read RemovingPackages before resorting to rpm --nodeps to remove any packages!

Step by step:

Read about common problems.
1. The Fedora Project provides a comprehensive list of version specific notes
2. For the upgrade from Fedora 8 to Fedora 9, make sure to add the fedora-updates-newkey.repo file to /etc/yum.repos.d
3. In addition, there's an additional bug in dircolors in Fedora 9 not fixed as of August 5, 2008, which can be fixed by commenting out the line in /etc/DIR_COLORS that starts with CAPABILITY
4. please add any additional notes to this list!
5. if you are relying on 3rd party repositories (livna, adobe, etc.), make sure that they are ready for the upgrade, or the process may fail.
What about backups?
Manual backups are a fairy tale told to excuse one of the most inexcusable oversights in design history; namely that all filesystems don't come with versioning and mirroring built in. Only do live upgrades on systems with mirrored, versioned file systems or development systems with disposable data.
What about PreUpgrade?
Good question. Who knows? Our info is oriented towards servers on which:
1. we do not run any GUI packages, and in an environment in which
2. we need to be very cost-sensitive, so we tend not to use GUI tools for systems administration (duh.) PreUpgrade has what seems to be a purely gratuitous GUI interface (it appears to piggy-back in some way on anaconda, so it may have seemed natural at the time, but our rule is to avoid GUI whenever possible for sys admin, and always offer a text alternative. So PreUpgrade is thus far irrelevant. Too bad, as it looked useful.
Prep your system.
1. Review, consolidate information, and remove all .rpmsave and .rpmnew files before and after upgrading. This step is a major part of the benefit you get from doing this process, as it gets you to look at systems you might otherwise not spend time with. Modify the following script for your use to review the changes captured by the .rpmsave and .rpmnew files, then delete them (you may want to look in places other than /etc and /var, for example.)
  
  for a in $(find /etc /var -name '*.rpm?*'); do diff -u $a ${a%.rpm?*}; done
2. Take a few moments (I do this until I can't stand it anymore...) to review all the packages that have accumulated on your system and work on RemovingPackages
3. Clean out the local cache that yum uses:
  
  yum clean all
Switch repositories:
1. install the packages (rpms) that tell fedora which release you want. These packages are architecture indepedent, so you can get both the release and the release notes by using a wildcard, for example, for Fedora 9:
  
  rpm -Uhv "ftp://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/Packages/fedora-release-*.noarch.rpm"
2. switch any 3rd party repositories, as well

Use runlevel 3
If you are not already working at runlevel 3 (and all servers should live at runlevel 3, see the Center for Internet Security Benchmark for one set of reasons why, and DesignPhilosophy for another), now is the time to switch to runlevel 3. At runlevel 3, networking and user accounts are fully enabled, but x windows (and hence all GUI stuff) is not.

If you are running a graphical desktop environment, log out of it.
Switch to a text console	`ctrl + alt + F1`
log in as root
Go to runlevel 3	`telinit 3`

Do the upgrade: yum, kernel & Base
You could do the entire upgrade all at once, but that's very dangerous, as you will trash your system if it dies in the middle. I do it in stages, as follows:

Upgrade `yum` and all its bits 'n pieces	`yum upgrade "yum*"`	This is also a good time to review and install any nifty new yum plugins, such as the `yum-upgrade-helper`
Upgrade the kernel	`yum upgrade kernel`	if you get any 'missing dependencies' errors, read this excellent post so that you really understand how dependencies work and what these errors mean. Then you'll not only be able to confidently untangle and rebuild the relevant parts of your system, but you'll develop your understanding of package managers, which is a very important free software concept.
Use the `Base` group to design the base of your system	`yum groupupdate Base` `yum upgrade <base-packages-you-want>` `yum install <new-base-packages-you-want>`	Unfortunately, fedora throws all sorts of stuff into the base that many of us neither want nor need. Review the packages in the `Base` group to see if you want them, otherwise you'll just be introducing a huge headache.

Do the upgrade: chunks
Now upgrade the rest of the system in bite-sized chunks. Groups are one good way to do that if you have defined your own, upgrading by wildcard by letter is another.

`yum grouplist`	will get you a list of installed and available groups. Just be careful how many you upgrade at a time. A couple hundred packages at a time is ok, but 800 is playing without a safety net. Questions: 1. I received errors during installation of some package(s) during a groupupdate. Is this a big deal? Discussion 2. When you do `yum groupupdate GROUPNAME` on a group that is listed as installed via grouplist and you see only a couple or few packages installed, shouldn't you install those packages by hand to avoid package installation bloat (i.e. installing more packages than we really need, thus causing a mess again?) Discussion
`yum groupinfo GROUPNAME`	Provides description and list of mandatory packages, default packages, optional packages, and conditional packages.
`yum groupupdate GROUPNAME`	Upgrade a group
`yum groupremove GROUPNAME`	`groupremove` will remove an entire group of packages and any dependencies. See also RemovingPackages.

Do the upgrade: package stragglers
Finally, we upgrade any leftover packages. By stragglers we mean any packages that weren't upgraded by the groupupdate procedure above. Upgrade your stragglers as follows:

Let's take a look at what stragglers are left	`yum check-update`	this will show you which packages you have installed that can be updated/upgraded. These should be packages not part of a group, since you just upgraded the groups in the previous steps. You may be surprised to see how many packages are not upgraded by the groupupdate procedure discussed above.
Get info on a package	`yum info PACKAGENAME`	If that doesn't give you enough info, follow the link to the package homepage.
Upgrade a package	`yum upgrade PACKAGENAME`
Removing packages you don't want with yum	`yum remove PACKAGENAME` or `yum erase PACKAGENAME`	`remove` will remove a package and any dependencies. Take a very careful look at the dependencies listed. See also RemovingPackages.

Clean up
At some point, you need to research and dispose of orphans. RemovingPackages discusses orphans and what to do about them.
Restarting the server.
Wait! Did you create any new files, logs, etc. during the upgrade procedure that you would like to backup? When you are ready, restart the server

shutdown -r now

If your server starts up fine, pat yourself on the back and take a bow! If your server doesn't start up correctly, DontPanic. Grab your favorite towel, take a deep breath, and take a look at DontPanic.ServerRecovery.

Remember where we started - you are trusting your work, your time, your precious code/homework/recipes/finances/etc to this box. All of us who know these systems well got comfortable with them not by being born with the knowledge, nor on the job, nor (unfortunately) by going to university classes, but by learning in stages with and from each other, when we had some need.

See also the Fedora Live Upgrade Special Interest Group, the Yum project wiki and our own YuM topic in the new Free / Open Source Software Web.

[back to top]

Removing Extraneous Packages Will Keep Your Server Happy and Healthy!

[edit]

In the natural course of things, linux servers accumulate both orphans (packages installed from rpms that are not associated with any current repository) and cruft (packages that are not relevant to the mission of the server.) Periodically reviewing and, in most cases, removing, such packages is a good thing (tm). Seriously, though, a review will:

improve security and decrease vulnerability (see also the Center for Internet Security Benchmarks)
improve system performance and response times (e.g., removes unneeded daemons, conserves system resources, ...)
save disk space
result in fewer and faster updates
improve system stability (problems with cruft won't affect you)

Orphans

What is an Orphan?

Technically, an orphan (yum calls these extras) is a package installed from an rpm that is not associated with any current repository.

What should I do with Orphans?

That depends on the type of orphan:

Packages initially installed from a repository (look at the rpm and yum installation logs, usually located in /var/log). These packages have effectively become orphaned, hence the name. You should research what has happened with the project that is maintaining the package.
- Sometimes there's a different repository from which you can now get the rpm. If so, add the new repository to your yum configuration. If the package is central to your mission, however, you will probably benefit from learning more about what is going on with the project.
- Sometimes the package has become deprecated in favor of another package, or included in another package. Continuing to run your production environment using the orphaned package is probably fine for the moment, however, migrating to the newer package should become your top priority.
Packages installed from an rpm which did not come from a repository. These packages are effectively extras, hence the name used by yum. Some projects have gotten to the point of providing an rpm but not to the point of providing a repository, and the rpm is not available from any major repository yet (did you check livna?) a It's worth taking a few minutes to check to see if the project has started to provide a repository, or if the rpm has gotten picked up by one of the major repository servers. That way, you'll get any updates as they are scheduled. a Failing that, you might consider providing your own internal repository for your lab and any collaborators. It will greatly ease working with VirtualizatioN, etc. a As a last resort, make sure that any true extras are well documented, as well as the results of the previous steps, so that the next time any members of your team conduct this review, they can pick up where you left off.

Finding Orphans

Depending on which version of which Linux distribution you are using, you may have one or more of the following tools available.

`yum list extras`	list the packages installed on the system that are not available in any configured yum repository.
`package-cleanup --orphans`	(`package-cleanup` is in package `yum-utils` which you may need to install.) Sometimes, especially when using multiple repositories, a package will get stuck and won't upgrade. Then you'll need to delete the version from the old core and install the version from the new core. You can set `showdupesfromrepos` to `1` in your `yum.conf` file temporarily to help with this, especially as just because a package has an old core number in its name does not mean it is not part of the current core (argh).
`yum list installed "fc#"`	especially when doing live upgrades. Replace `fc#` with the correct value for the old core, e.g., fc8 when upgrading to core 9.) Conversely, not every orphaned package is detected by `package-cleanup`, so you need both commands we list to catch all orphans.

Cruft

Cruft, like orphans, comes in several flavors. Many kinds of cruft can actually be prevented from accumulating by careful configuration of yum.

Preventing Cruft

Configuring yum to prevent cruft (would love to have similar info contributed for apt-get, etc...)

obsoletes=1 will prevent obsolete packages from accumulating on your system. If you need to hunt down existing obsolete packages, yum list obsoletes will find them.
multilib_policy=best will install only the best option for 64 bit machines.
plugins=1 turns on a variety of plugins that we also use to tune how yum behaves
we've tried working with the exclude list, but have found that module authors need the flexibility with the dependencies, and it generally works better to let things install and then pull them back out, rather than prevent the installation at all.

Leaves

Leaves are packages installed from rpms that are not depended upon by any other rpms. While being a leaf does not make a package cruft, leaf packages are likely candidates to be cruft, particularly if those leaves are libraries. package-cleanup, in yum-utils, is particularly helpful in identifying leaves that are cruft.

Dancing Bear Ware

Review all the packages that have accumulated on your system:

yum list installed

Research the unfamiliar ones. Do you really want those? Do you need them? Do you have a clue what they do? De-cruftify your OS a bit. Snuggle up to it and get to know it, take loving care of it. Don't let anyone tell you that you are unqualified to know what all that stuff is - it's your computer, not theirs.

yum info < packagename >

Provides a minimal set of information about a package, which is often enough to help you decide. If that doesn't give you enough info, there's a link in the provided info to the package homepage. Follow the link and read more. You can also try running yum erase < packagename > and see if the results are terrifying.

Understanding Dependencies

The dependency system is at the heart of the collaborative development of FOSS. PackageManagers provide a relatively easy way for package developers to specify which package(s) their package depends on. The package manager generates dependency chains from this dependency information, resolving dependency issues as it progresses.

Pay attention to dependency information while removing packages, or you may very easily end up with a dead or malfunctioning server. You may have to override dependency info from time to time (see Circular dependencies), so here are some general guidelines:

start by running package-cleanup --problems to identify any existing dependency problems in your local rpm database and fixing any problems encountered.
never, ever override dependencies related to the package manager system itself (rpm, yum, and python, in which the system is implemented.) The exception would be the rare occasion when there's a known problem and a published work-around that is carefully documented as a step-by-step process. You'll know when this situation has occurred because the information will come from official sources.
be very, very wary of overriding any dependencies related to core system functionality such as mount (mounts filesystems!), etc. Again, if it seems like that is what is needed, spend some time googling to see if you are looking at a known issue. If not, you have probably done something wrong earlier in the process and should consult an expert.

Circular dependencies

Circular dependencies occur when package A depends on package B, but package B depends on package A. You can have a longer loop as well (package A depends on Package B depends on package C which depends on package A). You can remove circular dependencies with rpm --erase --nodeps < packagename >, but you must remove all the packages in the circle.

Removing Packages with Yum

yum erase < whateveryoufound >

(don't use yum -y here!). If yum wants to delete all sorts of stuff for dependencies, look carefully. It doesn't hurt to google the package and any error messages you may be getting with yum erase or yum remove before continuing. Sometimes, it's ok. For example, you may be removing orphans, and what it wants to delete may also be orphans. If not, you may want to let yum delete the packages and then re-install, or you may want to fall back to rpm

Removing Packages with RPM

You should only attempt to use rpm to erase packages if yum can not remove the package due to various errors. Do NOT use this command lightly!

rpm --erase --nodeps < whateveryoufound >

This will erase a package while ignoring dependencies. It is NOT recommended to use rpm in this manner, unless you have checked for dependencies (at least with yum) and there are NO dependencies listed or you have adequately researched the dependencies.

You are on your own if you get some wild ideas to just yank packages out with no regard for dependencies.

mea culpa... Yes, I'm guilty of yanking out packages with no regard for dependencies. It's true.

But the internet told me I could do it!?!

There might be situations where using rpm --erase --nodeps < packagename > might be warranted, but beware. I used it wholesale once for an upgrade on various packages that didn't seem related to the core install for a server. Needless to say, the server didn't restart after reboot.

What if I've removed too much?

If you haven't restarted the server yet, you should be able to just reinstall the packages you removed. Please let us know if you went through this experience, and the server did not start.
If you have restarted your server, and it doesn't boot up correctly, you will probably need to boot off of a CD or recovery disk, and then try to reinstall whatever it is you removed. See Server Recovery - Help! My server won't boot! for more info.

[back to top]

Perl's package managers: cpan and cpanp

[edit]

Perl has its own package managers (see BootstrappingPerl), which should be used in preference to rpm or yum. There are two interfaces (modules) in use right now - a lower layer of abstraction, the CPAN module (MCPAN) and a higher layer of abstraction, the CPANPLUS module (MCPANPLUS) to which the community is migrating, slowly. It's part of a larger migration to 'pure perl' based on a new module abstraction layer (Module::Build). In any case, you should use cpanp for most things, however, for some things you will still need to fall back to perl -MCPAN -e shell.

Some hints about using cpanp.

use 's reconfigure' to configure cpanp to your taste. I like setting the interface to 'classic', which makes it behave a lot like MCPAN, which is what I am used to.
you will need to use 'l Whatever::Module' to check if a module is installed, as 'm Whatever::Module' does not have the cool = marker to show whether a module is installed or not like the listing on MCPAN.
if you are having trouble installing a package, it may be that header files needed to compile the packages are missing from the system. Some linux distributions (redhat/fedora does this) distinguish between production and development versions of packages. If there's an error message that says can't find foobar.h, or no such file foobar.h, install the -devel package that provides foobar.h

[back to top]

Managing Security

[edit]

Security is a huge subject. The information we provide here is based on our experience, and you follow it at your own risk, we make no promises, warranties, etc., implied or otherwise. The Center for Internet Security publishes best practice guidelines tailored to all the major platforms as well as many important services: apache, various SQL database servers, etc. The Internet Storm Center collects and disseminates information about internet security threats on an ongoing basis, as well as providing useful tools for combating a variety of threats.

Practical, usable security

Security isn't secure if you won't use it! The initial configuration sequence for our minimally reasonable, responsible, and usable configuration is given here. Note that all the instructions are command-line, as GUI interfaces are much slower to use, and not generally appropriate for servers.

On initial install, your distribution will ask you to create a user account that you will use to administer the machine.
- Each administrator needs their own account, as they will customize it to their own working style, and as the security system will log their activity. The user account is important because you should really never log in to your machine as root unless doing disaster recovery or for some other specialized purpose. So go ahead and create the account immediately. Let's call this account rosta (for sentimental reasons.)
You need to add rosta to group wheel.
- Get your spiffy new linux box installed and configured per the distribution's instructions, and boot it into the OS. The configuration instructions may already have had you add rosta to wheel, but if not, here goes:
  1. Log in to root (this is one of those times.)
  2. add rosta to group wheel
```
usermod -a -G wheel rosta
```
    (don't forget to take a minute or two now to read the usermod man page, as per RtFM)
follow the instructions under UseSudo to setup sudo
Using sudo
1. follow the instructions under SshD to setup sshd on your server and ssh clients for any workstations that will access the server.
2. install DenyHosts
3. take a first pass through the relevant CIS (Center for Internet Security) benchmark, so that you are at least aware of all the issues involved.

Use Sudo!

[edit]

sudo is a system that (1) provides you with a log of everything you do as superuser, and (2) allows finer grained control over who can do what than simply putting trusted users into group wheel. Really, the only time you should ever use the su command to act as root is to setup sudo.

Getting started with sudo

We provide a crash course in setting up sudo , however, don't forget to RtFM as well, as it is your security that is on the line, not ours!!

The configuration file for sudo is /etc/sudoers. You have to use a special command, visudo, to edit /etc/sudoers . You can make visudo use your favorite editor (rather than vi ) by setting the environment variable EDITOR in your shell before invoking visudo , but that's properly the subject of another topic.

Log in to your privileged administrator account (we're calling it rosta for sentimental reasons.) Enter
```
su
```
at the prompt, followed by the root password to become the superuser. This is marginally better than logging in to the root account to work on your server...
type visudo at the prompt. An editor will open with the default configuration of /etc/sudoers. Work your way down to the line below the comment that says "Allows people in group wheel to run all commands" (don't choose the one that says "Same thing without a password" - you can do that later, if you decide to configure SshD to disable password logins and opt for certificate or public-key logins instead.)
make a copy of the existing line (much easier IMNSHO when scanning diff reports of configuration files later), then remove the comment # and space at the beginning of the line. Save your work. Log out and back in and check to be sure that you did things correctly by using sudo to acquire superuser privileges:
```
sudo -s
```
Most of the time, you'll want to just prefix each command with sudo, as asking for superuser privileges is a dangerous habit.

Better use of sudo

It's really better not to just put folks in wheel and let them have full privileges. One of the wonderful things about sudo is it's ability to confer heightened privileges within a limited scope. So you can give privileges to mount and unmount media, but not other things, or to restart certain servers, for example.

back to top

denyhosts

[edit]

denyhosts is a python daemon that detects and thwarts brute-force ssh attacks. The configuration script, by the way, needs to be /etc/denyhosts.conf, but seems to be installed as /etc/denyhosts.cfg in some versions. Just read and follow the comments in the configuration file.

back to top

Ports & iptables

[edit]

On any new server, you'll need to ask to have any ports you want accessible from off campus explicitly entered into the university's firewall setup. Send email to the designated contact (this should really be a link to the appropriate entry in the IT directory, you know..., currently, for us, that's richard.uhler@csueastbayHYRUQAPZ.edu) with the request. You'll need to explain what ports you want open, and why.

One way to learn what we usually have open is to look at a sample IpTables configuration file with lots of comments and figure out what services apply to your server. Always ask for ports 50000 - 50500, as those are the Math and Computer Science research ports - ports on which we are free to take services up and down at will, but are not preallocated to anything in particular. If you reserve a particular port on a particular server for some long term use, you should add that info to the relevant entry in the LabData topic.

In combination with good security practices set out by the CIS guidelines (as well as those discussed in this twiki) such as using DenyHosts, ModSecurity, SpamAssassin, ..., having some externally open ports, even the research ports, has not proven to be a major issue.

Speaking of iptables (a firewall that runs on linux servers), we run iptables on all our machines. Don't open any ports unless you know what/why, even if a program wants you to do so. Ask for help. /etc/services lists all the port numbers with an incredibly terse description of the related service, but it will give you a key word to Google for.

back to top

sshd, the secure shell daemon

[edit]

The ISC guidelines cover most of the important ssh configuration information. We provide some more detail to help you get going, as well as explain how to prevent remote connections from getting dropped all the time!

Configuring ssh

Password authentication is inherently unsafe, not to mention a pain-in-the-you-know-what. Other forms of authentication, such as public key or certificate authentication may be a much better fit, depending on your lab setup.

The configuration we show here is for public key authentication (no passwords), with PAM (pluggable authentication modules) account and session checks. We show only the sshd_config settings we change from the default settings, and group them a bit differently than the order they appear in the file to make it easier to explain.

# HJH, duh!
PermitRootLogin no  

# HJH, per CIS_RHEL_Benchmark
IgnoreUserKnownHosts yes

# HJH, no password authentication
PasswordAuthentication no

# HJH, disable s/key passwords
ChallengeResponseAuthentication no

# HJH, enable PAM session processing
UsePAM yes

# HJH, poll client every 30 secs, so router doesn't
# drop connection. Settings give 25 minutes of inactivity.
ClientAliveInterval 30
ClientAliveCountMax 50

# HJH, as per CIS_RHEL5_Benchmark
Banner /etc/issue.net

Be sure to review the pam related packages to be sure that you have the appropriate ones installed on your system. We'd love to give you a canonical list, but they get re-factored all the time.

Configuring public-key authentication

The specifics vary somewhat according to the specific protocol, but the general outline is:

generate the private/public key pair on the laptop/workstation
upload the public key to the account on the server, and put it in the target account's .ssh directory with the appropriate name

back to top

Death By Networking

[edit]

Pocket labs just don't fit in to the networking scheme of larger IT. With Cyberinfrastructure all the rage (there's a vaporware term, goes right along with Computational Thinking, another of my favorites), we have a very real, very growing need to come up with better systems administration turnkey solutions to minimize our impact on our host institutions. That's a focus area for me, and one of the reasons I'm leaving academia for industry (because it seems to be a necessary step for me to be able to work on the problem....) In any case, for the moment, here's our older manual setup, with more automated info to follow the moment we have some breathing space to document what we're using:

DHCP and pocket research lab servers

[edit]

DHCP (dynamic host configuration protocol) is a system that helps with efficient and secure management of a network of computers. The networking information is kept on a central server (the DHCP server), and the individual computers run DHC client programs which contact the DHCP servers on startup. This plan works fine when either (a) all the machines are administered by some central organization (the case for computers in classroom laboratories, for example, which are administered by university IT), or (b) the DHCP server and the client computers are fully decentralized (the case with faculty and student laptops, for example.) The system breaks down, however, for pocket research lab servers, which are, by necessity, neither fully integrated into the UIT structure, nor always easily accessible by research staff. For example, in an unplanned university-wide power outage, pocket research lab servers reboot before the DNS servers, end up with incorrect network information, and become unreachable, unless each one has its own UPS. Hardware that provides the ability to remotely power cycle the servers is available, but costs even more than a UPS. A much cheaper answer is to collaborate with UIT on an initial configuration for each server via DHCP, and then convert the server to a static configuration, documenting the relationship to the dynamic configuration meticulously in the relevant configuration files on the server itself. That way, desired changes to the networking setup can be coordinated with UIT, the servers are robust with respect to the vagaries of life in the big city, the local control necessary for research and learning is retained, and we save a bunch of $$$$. A perfect example of a situation in which the fix becomes obvious when you view the mediating artifact as an integrated whole, rather than focusing exclusively on the programmable components.

Speaking of focusing on the programmable components, the situation is further complicated by the fact that most Unix distributions are just sure that you want to run DHCP, and will revert you back to that default configuration at every major upgrade, unless you take explicit action to prevent that from happening. So be sure to add dhclient and *dhcp* to the excludes list in YumDotConf once you have converted to a static networking setup. In time we'll get more surgical about just what needs to be excluded.

back to top

static networking setup

[edit]

(If you have not read DHCPornotDHCP, read that first.) In addition to erasing the dhclient and various dhcp packages, you should also erase NetworkManager. Manually configure:

/etc/sysconfig/network-scripts/ifcfg-eth0 (for the first ethernet card, possibly additional scripts for additional cards)
/etc/sysconfig/network
/etc/resolv.conf
/etc/hosts

see examples in the serverfiles subversion project.

back to top

. . .

You mentioned Kickstart the other day... RedHat's automated installation method. When i have time I'd like to explore that further. Seems like it could be a really nice time saver once you have it set up for future server updates. some useful links...

Kickstart Installation:
http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Installation_Guide-en-US/ch-kickstart2.html
RedHat customization guide:
http://www.redhat.com/docs/manuals/linux/RHL-7.3-Manual/custom-guide/index.html -- RobertSajan - 28 Nov 2008

Great write up Hilary. Thx! -- RobertSajan - 28 Nov 2008