Fixing the Reckless Debian Security Update to Samba 4.2

If you’re like me, and were wanting to get the Samba security fixes in place quickly for Debian Jessie version of SAMBA — you’ll maybe have learned that they decided not to backport the fixes for man-in-the-middle, but instead upgrade the whole thing to SAMBA 4.2.

That’s all well and good — but the security update broke SAMBA on my Debian boxes in the test lab (which run Jessie).

The fix was pretty simple after some fairly serious wasted time Googling for it. By default, it SAMBA will now require the winbind package to be installed, otherwise it will silently fail to start.

The symptom I had was that internal DNS resolution for SAMBA stopped working.  And nothing was bound to port 53.

Of course, that was because Samba wasn’t starting at all.

I hope that helps someone — just make sure winbind is installed and running.

They really should have tested this better, even with such a bad flaw in the Microsoft and Samba stuff.

Pipe to Perl for the Quick and Dirty Big Jobs

I always find myself using Perl for systems administration tasks, particularly when you must do things on many systems at once. It is in Perl’s DNA to be down in the trenches of the operating system.

For example, last night I had an NTP configuration file I needed to get out to a few dozen machines. I had a list of them from a display of managed machines, and they were tabulated like this:

62 comp1.dom.com 8 Workstation 6 Company, Inc.
61 comp2.dom.com 8 Workstation 6 Company, Inc.
63 comp3.dom.com 8 Workstation 6 Company, Inc.

And I need to get an NTP configuration file to each. Sure, you can write a shell script using sed to do it just as easy, but with Perl you do get a little more flexibility to do the strange things that must be done from time to time.

#!/usr/bin/env perl

foreach (<STDIN>) {
    my($id, $host, $ignore) = split;
    print `scp -v ntp.conf root\@$host:/etc`;
}

Of course, if you like to live dangerously, you can always put in full ssh commands. You can make the command you want to run be an argument. You can use any of the numerous Perl modules available to handle specific things or concerns.

The command for this is something like:

list_hosts | my_perl_program

Perl just lets you read from STDIN as a normal pipe, with nothing extra you have to do. Columnar/tabular data – it’s so easy. Calling a shell command and sending the output anywhere? Right there.

Edit that to reload your NTP daemon on each. Or make the command an argument when you call the program. Or an option, using Getopt::Long, or whatever you like.

Anyway, my suggestion here, start using some Perl. For admin purposes, it’s like the shell, on multi-dimensional steroids that don’t cause cancer.

Creating a Debian GNU/Linux Server – a Minimal Install

When you’re creating servers you usually don’t want the overhead of a GUI. This is particularly true when you’re creating a virtual server. You only want the stuff you need. And the basic Debian, of course, to hold it.

Grab an Install Image

Debian has several install images available. I usually prefer the network install image (netinst). It’s minimal in size, giving you what  you need to install, and then downloads only what you need.

I usually use the 64-bit one as well, even though you can save a little memory using a 32-bit one. There is a “Live” one, but I’ve not had good luck with it in unusual circumstances, which is often the case when I’m making servers or little devices.

If you won’t have a working network connection, get one of the full images instead. All Debian images are updated from time to time, as updated releases happen, so check occasionally.

Install Media

If you’re installing into a virtual machine, you’re already set with this ISO image. If you’re installing onto hardware, I find a USB thumb drive is most useful.

The Debian ISO images have long been set up to create proper bootable disks by just dumping the ISO data directly to the USB.

To create the USB stick in Linux, it’s just a matter of using dd to dump it. In Windows you’ll need a program such as Unetbootin to create the bootable USB stick from that ISO image. Unetbootin also runs in Linux, and it often packaged in Debian already.

But the quick and easy way is just use a command prompt. Find your USB drive’s device name – that’s the hardest part. In Gnome you can look at disk utility, or tail the syslog and watch while you plug it in. 😉

$ wget http://cdimage.debian.org/debian-cd/7.6.0/amd64/iso-cd/debian-7.6.0-amd64-netinst.iso

$ dd if=debian-7.6.0-amd64-netinst.iso of=/dev/usbdrivename bs=4M ; sync

Please be very careful with that dd command. You can easily destroy your system or data if you put in the wrong of= parameter. But if you don’t destroy your system or data, we’re ready to go.

Make sure you set your BIOS or UEFI to boot from the USB drive, or if you’re installing into a virtual machine, don’t forget to attach it as a CDROM.

The Install

The first screen you’ll come to when booting will let you pick between a normal install and a graphical install (along with some advanced options, which we’ll hopefully cover later).

I usually pick the normal install because you don’t have to worry about moving a mouse around as much, nor do you have to worry about there being any strangeness with your graphics cards and this particular kernel (which is rare any more on a base install).

The next screen lets you pick the language, which defaults to English, your location, your keyboard layout (all English and American by default).

After just those few very basic questions, the install will start loading some kernel modules it might need, including those needed to activate your network interface.

The Network Interface

The Debian installer will now try to set up any network interfaces it finds. It will try to configure both IPv6 and IPv4 with DHCP. You can, of course, specify the configuration manually if it doesn’t detect network configurations.

If you’ve got a good hostname, it will fill that in from DNS automatically, or you can specify it yourself. Same with domain name.

If no hardware interfaces are detected, you can always go back to the main menu and try loading kernel modules manually, if there are any, or just weep. It seems that just about everything is recognized any more.

User Account Setup

Next you’ll be prompted to enter a password for the root user. Debian enabled root logins by default. If you want to disallow them, you’ll need to install “sudo” and  add a normal user to the “sudo” group, and disable the password of root (passwd –lock root). If you do that, you can still log in as root with SSH keys, if you have them.

After you set root’s password, you’ll be prompted to add a new, normal system user. I hate that they added this bit. But just enter something I suppose.

Final Little Miscellaneous Config

You’ll need to enter in your timezone. I know! So hard.

Disk Partitioning

Gods help you. There are so many ways you can do it. If you have one drive and want everything just on that drive, select “Guided – use entire disk”.

I never do that, though, unless it’s a virtual server. I like to set disks up with LVM, and RAID multiple disks together. You can do all that nice and easily in the disk partitioner if you like. It’s very well laid out. But more complex disk partitioning schemes is beyond the scope of this quickie walk-though.

You can, of course, read through an intro to various Logical Volume Manager disk stuff if you like. The Debian installer automates a lot of that.

Another nice feature is that you can set up full disk or partition encryption here, too, which is convenient for your secret purposes. You can encrypt a partition or a RAID device or LVM volumes themselves. Many people argue over which is best to do. I have my moods, and like variety.

I rarely choose anything but to place all files in one partition. This is because I would rather muck about with the underlying volumes and and devices. However, there are some very good reasons to separate your mountpoint concerns in some situations, particularly with read-only and highly space-constrained systems.

After answer the partitioner’s few questions, you’ll be presented with its final understanding of what you wanted, which you should verify. Let’s look at one I just did:

Debian Partitioner

Here you see I chose to put everything in one partition using LVM underneath it. Which, of course, the installer lies about, and really creates a boot partition for you as well.  And even a UEFI one if your system’s got that. It’s good practice to keep a separate boot partition.

The Debian installer organizes things from the most abstract up top, to the most low-level stuff below. You’ll see the LVM volume group and logical volumes it wants to create on top, and the actual disks that contain those down below.

Generally, when you want to make changes, start at the top and work your way down, if you really need to. I’m going to skip the LVM configurator and the RAID configurator, because we could really just go on forever with that.

The point is, the Debian installer gives you incredibly good access to setting up your storage in all kinds of ways. They were the first to support so many different things at boot, and although it can bring complexity, you don’t have to deal with that complexity any more if you just follow their defaults (all in one place).

So… after it looks right to you, or doesn’t and you’re just trusting, you select “Finish partitioning and write changes to disk” and you’re good to go.

Oh, you can move the cursor to any of those top mount points and change the filesystems you want to use, or what gets mounted where, all that stuff. If you like, or have the need. Even mount options that will get automatically passed along to your system’s fstab

You should know that the base Debian system is very small indeed. The 8Gb you see here will leave plenty of room for many, many things that servers do. The end user data stuff, probably not. But you can always tack on a new disk for that, or partition it in a different place.

Installing the Debian System

After your partitioning and formatting are successful, the installer will install the base Debian system, which is, well, the base system. All you need to boot up and get going, adding whatever packages you like.

After it does that, it will want to set up updates for you, and will ask where you are so that it can find a Debian archive mirror that is close to you.

Pick one! Then it will ask you if you use an HTTP proxy to get out to the net. If you do, enter the info here. And as an aside, setting up a proxy for Debian packages is really, really nice when you do a lot of virtual machines. It saves you having to download the packages to every machine, which can save a great deal of time, over time. apt-cacher-ng is my personal favorite.

It will then download archive package indexes, install and update any packages that may need doing, and then it wants to know just what type of Debian system do you want here?

Software Selection

If you’re doing a server, uncheck the Debian desktop environment. It’s too fat. Odds are you don’t want a print server, either. Why is that checked by default? Whose version of madness is allowed to permeates us all so?

Standard system utilities is all you want. And SSH server.

But, maybe you’ll want to run this as a DNS server, whether master or slave? You can check that and it will set up Bind9 for you.

File server will give you Samba 3 and some utilities. Mail server gives you Exim 4 – which I finally, after years of clinging to Sendmail, surrendered to myself. Web server gives you Apache 2

But all those others can be easily added with apt-get later. Standard system and SSH is the way to go. Check that, and away you go.

Boot Loader

After the install finishes, the installer will want to place a boot loader on the drives. The boot loader used by default is Grub. It’s pretty good at detecting other installed operating systems, which hopefully you haven’t overwritten by telling the installer to use an entire hard drive. Well, unless you want that.

The installer seems to do a very good job of working with UEFI as well. However, I have noticed that from time to time, Windows 8 will not like to boot after installing a new boot loader. Be warned if you’re trying to dual boot with Windows 8. Sometimes it works, and sometimes it doesn’t, and the only thing I can think of is different motherboard manufacturers implementing UEFI differently.

Let the boot loader install, and your new minimal Debian system should reboot up just fine, ready for you to begin work on making that new server.

The Debian developers have done a really great job of creating a very minimalist and simple installation system. And for popping out one virtual machine after another, it really can’t be beat.

Add a Simple Samba File Server as a Domain Member

ssambaIf you already have an Active Directory Domain Controller in place, diligently servicing all your needs and making itself indispensable, hopefully you’ve chosen Linux and Samba 4 to fulfill this.

If you haven’t used this free and open version of Active Directory and the domain controller, perhaps you’d like to? For the latest and greatest Samba 4 version that you compile yourself, you can follow the steps outlined for Debian Wheezy (and possibly Ubuntu).

Remember, too, that Debian backports now has a much more recent version of Samba 4 in its archives, and it seems to be working great right now.

But suppose you have that beast in place, and you want to add, say, a network file server — some storage that can be accessed by all domain users and whose rights and permissions are determined by your Active Directory domain controller.

It’s easy to just create a share on your Samba 4 Active Directory Domain Controller, and serve that out, with all the permission “goodness”.  But maybe you don’t have tons of storage in that box, or in that VM, and you don’t want to, either.

Or, you could find some filthy user’s computer in the domain that has a lot of hard drive space in an array, and map it out to the others. (I mean lovely user – it’s a joke!#$@!). But we know that’s a terribly unsettling idea.

So why not build out or spin up a new Samba 4 server that’s pretty much dedicated to housing user data, whether that’s shared between the masses for apps or data, or just for daily backups.

I’ll tell you why. Because you’re frightened. How can you get one Samba server to listen to another Samba server and believe all its tales about the users and their permissions? How will the UIDs and GIDs match up for filesystem stuff? Sure, you know it’s possible. You’ve looked around. You’ve seen all kinds of insanity for linking Samba 3 into Windows AD/DC, mapping local users and huge cut-‘n-paste swaths of wild configuration blocks.

But if you go Samba 4 again, will you find any documentation that is currently to the point where Samba 4 development is? Or to the way that Debian has mutated it?

The answer is yes! All over the place, including the Samba site itself, and it’s all whacko and incomplete, if all you want to do is create a simple file server – you just want a simple Samba 4 file server that is a domain member file server. You don’t care if users can ever log into the Linux box. You just want them to have access to Windows file shares served with Windows users permissions and rights honored.

And you may think… I’ll just make a big RAID disk on some server, and serve out to my One Box To Rule Them All Samba 4 AD/DC – and then it has big drives and plenty of space! Well, it would if NFS or CIFS handled extended filesystem attributes… Alas! We are thwarted.

So here’s what I did:

Prerequisite

You must a Samba 4 Active Directory Domain Controller running just fine already. Or a normal Windows-y one if you must run a Windows one for some nonsense reason.

You must have a Debian (or possibly Ubuntu) server ready to go with only the minimal stuff installed – like the SSH server. That’s because it’s proper. And you will be proper. Another distribution is fine of course, just don’t complain to me.

Don’t use Debian Wheezy’s version of Samba 4. It’s not ripe yet. And don’t use Samba 3. It’s overripe. (Simply too many notes) Either roll your own Samba 4 from source or enable the Debian backports repository and go with that version of Samba 4, there.

If you follow the instructions to roll your own, skip the samba-tool domain-provision step and the Cold, Cruel Kerberos sections!It makes your new Samba 4 server have delusions of grandeur and it won’t want to listen to your already-existing AD/DC.

If you go with Debian backports, this is what you’ll need:

# apt-get -t wheezy-backports install samba samba-doc samba-testsuite winbind libnss-winbind
# apt-get install acl

Setting Up the Samba 4 Domain Member

The smb.conf [global] section

If you’re using the Debian version of the /etc/samba/smb.conf from backports, throw away everything in it, because it’s garbage Samba 3 stuff and they haven’t bothered tidying anything up.

For a simple Samba file server, you just need your [global] section and your share definition. I’ll highlight some of the stuff in it, after this example listing (that works just great). Be certain that your filesystem that serves out the stuff is mounted to support xattr and acl’s. (in your /etc/fstab put the mount options “user_xattr,acl” in place of “defaults”).

Anyway, here’s that /etc/samba/smb.conf file:

[global]
  netbios name = <servername>
  workgroup = <win domain>
  security = ADS
  realm = <kerberos realm>
  encrypt passwords = yes

  idmap config *:backend = tdb
  idmap config *:range = 70001-80000
  idmap config <win domain>:backend = ad
  idmap config <win domain>:schema_mode = rfc2307
  idmap config <win domain>:range = 3000000-4000000

  winbind nss info = rfc2307
  winbind trusted domains only = no
  winbind use default domain = yes
  winbind enum users = yes
  winbind enum groups = yes

  vfs objects = acl_xattr
  map acl inherit = Yes
  store dos attributes = Yes

That’s it, really. I love Samba 4 for this. Consider me gushing and oozing again, all over the Samba development team again for doing such a damn fine job pulling so much together. Very well done.

Here’s what some of those things are:

  • netbios name – anything you like – I like to use the hostname portion from DNS
  • workgroup – that’s your Windows “domain”. Make it the same as your main Samba 4 server’s.
  • realm – your Kerberos realm – again, make it the same as what your main Samba 4 server’s is.

Now, the idmap stuff should be fine for you to use, if your AD/DC is standard Samba 4. The range values are UID values that map to local users (the asterisks) and your AD users (the <win domain> ones).

By default, at least right now, Samba 4 is by default using the UID range of 3000000 – 4000000. That can be changed when you set up your Samba 4 AD/DC. A Windows-based AD/DC is probably different. Find it out, and put it there.

The local mapping range just shouldn’t overlap your domain range ones. And I kept mine well above any UIDs I’d really be using locally on this machine (in the 70000’s… 😉 )

I hate winbind. I just copy and pasted that bit from some Samba site documentation dealing with domain member servers. It seems necessary, especially the nss bit for letting your system know about domain users on the AD/DC.

The last 3 lines are critical if you want permissions to be right on the filesystem here.

The smb.conf Share Definition

Behold the glory of letting a fascist AD/DC handle all your decisions for you!

[storage]
   path = /srv/storage
   read only = no
   admin users = "@MYDOMAIN\Domain Admins"

Whatever path you want there, Sparky. And the MYDOMAIN bit is your domain of course, just like above. And oddly, this is the tricky one that caused me quite the headache.

After I got all this set up, I couldn’t set permissions on the share as a domain administrator. Permission denied. You have to add the “admin users” parameter even for domain administrators. I like this because it gives your local root the final authority still. But nobody mentioned this and it took forever to figure out. Here, I have the admin users be anyone @MYDOMAIN who is part of the Domain Admins group on the AD/DC.

But I digress.

The Final Steps

Of course, reload or restart this Samba daemon and its associated daemon cohorts of nmb and winbind.

# service samba restart
# service winbind restart

I hope you didn’t “provision” this Samba 4 server with samba-tool…

Also, remember to edit your /etc/resolv.conf file to make sure the nameserver you’re using is your Active Directory Domain Controller. If you don’t, you won’t be able to join the domain, since it relies on that absurd DNS injection flotsam.

The next step is to join your domain as a domain member.

# net ads join -Uadministrator

No, it’s not magic. It just gets your domain info from you smb.conf file.

If you got an error, you might want to make sure that you’re using your main AD/DC server as your DNS server in /etc/resolv.conf – that is, make sure your ‘nameserver’ is set to the IP address of your primary AD/DC. If it’s not, you won’t get the revolting Microsoft DNS injection they made necessary, and you’ll fail.

Remember the NIS/YP stuff? No? Well, then just do what I say:

# edit /etc/nsswitch.conf

Then add “winbind” to the end of the passwd and group lines, while leaving the rest of the file alone:

passwd:         compat winbind
group:          compat winbind

Final Thoughts

You might want to reboot. Make sure your mountpoint for the share you’re offering has those attributes specified in /etc/fstab.

You’ll probably want to grant your domain admins the ability to control files and permissions on your shares as well:

# net rpc rights grant 'MYDOMAIN\Domain Admins' SeDiskOperatorPrivilege -Uadministrator

But maybe not! I think it’s pretty well handled by specify the admin on the share level of the smb.conf file. But I did both, because I was flailing with that problem I mentioned earlier, and will not reiterate out shame.

Anyway, that’s about it. Except for, of course, always be sure you install ntp since time is so critical for this implementation with Kerberos. Just a few steps really, but lots of words and background info on why. Hope it helps.

Change Default SMTP Relay Port in Debian’s Exim4

It seems fairly common for someone to have a private range of IP addresses behind a dynamic IP address assigned by an ISP. If this is your situation, you may get your SMTP port blocked by your ISP.

For those of us with SMTP relays in a central place on the Internet, having our SMTP port blocked by our “conscientious” local ISP is troublesome. But, they usually excel at troublesome.

So when your local machines or servers need to send mail, like, say, to report that a hard drive in the array has failed… they’re out of luck unless you send it through your ISP’s relay.

Unless… your ISP allows at least some mail ports though (or you set yours to listen on bizarre ports, which is commendable when necessary).

So we know that residential Comcast blocks SMTP port 25 which keeps us from relaying our valid email from local machines. But they don’t block port 587, which they consider “secured” for some reason. Why? I don’t know. You can, and should, and most smart people do encrypt on port 25. And you don’t have to encrypt on port 587 if you don’t want to. And relays can be open on port 587 as easily as port 25. So… not sure why they think port 587 is “secured” while port 25 “unsecured”.  I think they just enjoy being fascists all-around. (please don’t smite me Comcast, I’m just a poor thing trying at humor)

Anyway, mail servers typically aren’t configured to relay on ports other than 25. It’s pretty easy to get them to listen and relay on the other ports, though. This post isn’t about listening, though. It’s about sending. And to send mail, relaying on port 587 (submission port) instead of port 25:

# edit /etc/exim4/update-exim4.conf.conf

Then just change your SMTP smarthost (mail server that relays mail on  your behalf to its destination) line:

dc_smarthost='mymxserver.mydomain.com::587'

You just append 2 colons and the port number. Of course, your mail server actually has to be listening on that port as well. Debian’s (and by extension Ubuntu’s) mail server Exim4 automatically deals with protocol and encryption negotiation.

Remember, any time you change your update-exim4.conf.conf file you need to run:

# update-exim4.conf
# service exim4 reload

That lets Debian generate all it’s Exim4 configuration magic that vexxes so the Exim4 developers. But believe me, it’s nicer than having to worry about doing it all by hand in the pure Exim4 way.

By the way, you can also just reconfigure Exim4 using the standard Debian dpkg scripts, and for your “smarthost” question, answer with those extra 2 colons and the port number as well as the FQDN of your mail relay.

# dpkg-reconfigure exim4-config

That script stuff will also restart the exim daemon for you.

Do that, and your boxes can now happily relay to your central SMTP mail server on port 587 instead of port 25 – or whatever other port your preferences or necessities might take you.

Override DHCP assigned DNS Server and Domain Search in dhclient

Sometimes, particularly if you are on a residential broadband Internet connection, your Linux box might need to get its external IP addressed assigned by your ISP.

In this case, and in the case of Debian, you define your network interface with the dhcp flag in /etc/network/interfaces

auto eth0
iface eth0 inet dhcp

When you do this, by default, your /etc/resolv.conf file will be overwritten by whatever your ISP wants to assign you for your DNS servers and your search domain as well. It’s not always desirable that your /etc/resolv.conf file gets overwritten.

In Debian, if you are not using Network-Manager, that is, if you have a nice, minimal system, like for a router, it is the “dhclient” program that is handling the task of getting your IP address and network configuration information from your ISP.

You can, if you like, alter, override or ignore whatever your ISP is assigning you by editing the dhclient configuration file in Debian:

edit /etc/dhcp/dhclient.conf

In here, at the bottom, if you want to ignore your ISP’s name server assignment and use your own machine as the DNS server, you can “supersede” what the ISP’s name server gives you – that is, completely ignore the stupid thing:

supersede domain-name-servers 127.0.0.1;

That gives you just your local machine as your DNS server. So please do make sure you have one. But perhaps you still want to use your ISP’s name servers, but want to use yours as well, say, for example, reverse DNS entries in the ARIN black holes that you might locally use for your subnet. In that case, you can just “prepend” or “append” your own entry to what the ISP will assign:

prepend domain-name-servers 127.0.0.1;

That way, you get yours first, then whatever they want to give you.

Of course, you’ll probably want to assign your own search domain too, so you don’t have to go typing FQDN’s all the time and give yourself carpel tunnel’s syndrome. So here’s some preventative care:

supersede domain-search "orbislumen.net";
supersede domain-name "orbislumen.net";

If you do these things, then your /etc/resolv.conf file will be just how you like it, even with that presumptuous dhclient trying to make your machines believe everything it hears from your ISP.

Of course, you’ll need to bring the interfaces down and up to see the changes happen – just use the

# ifdown eth0
# ifup eth0

I would think that is self-evident, but I’ve been nagged at before for not saying such things. And I’m delicate.

Hope this helps!

Fix mdadm SparesMissing event detected

Sometimes when you create a RAID array using mdadm and do not specifically specify there are no spare drives available for the array, the mdadm monitoring daemon will report to you in daily emails that:

A SparesMissing event had been detected on md device /dev/md*

If you check your /etc/mdadm/mdadm.conf file, you’ll most likely find the ARRAY assembly line specifies an “spares” parameter that is non-zero.

To stop this daily nagging email message, just change the “spares” parameter to 0.

ARRAY /dev/md0 metadata=1.2 spares=0 name=<mymachine>:0 UUID=<myUUID>

The reload the monitoring daemon. In Debian:

service mdadm reload

Currently, that command will actually stop and start the daemon, rather than reloading it, but hey, who knows what the future modern world will bring?

Also, this is all assuming that you really do have no spares in your array, and the mdadm system just couldn’t believe such audacity, and concluded that you must have a least one that you just didn’t tell it about.

LVM Basics – A Quick Intro and Overview by Example

Storage technologies and methods can be a confusing subject when you’re first starting out, and depending on which route you choose, and how you organize them, these storage methods can continue to be confusing even when you know lots – unless you plan and organize well, at the beginning.

Take the time to think about what you want to have before losing yourself in the details of any given technology. And keep this in mind as you create.

Following is very generalized overview of some practical LVM use, presented in simple, no-frills terms. LVM tends to scare people away, seeming complex at first. And it can be, but it doesn’t have to be. This is the easy way, to get you started with LVM if you’re interested. You can make it harder (and perhaps better) later.

What is LVM? Why use it?

GNU/Linux gives you many options. LVM, the Logical Volume Manager is one of them. LVM lets you combine disks and partitions and even arrays of disks together into single filesystems.

LVM is very flexible. None of your storage media or arrays need to match up – you can combine a high performance PCI-e RAID10 array with an external USB 3.0 4TB hard drive if you like. Not always the best idea performance-wise, but often performance isn’t the most important thing.

LVM performs very well, too. Is your RAID0 SATA array running out of space? Add 2 more drives in another RAID0 and combine their capacities together. Or just throw in one new drive. It’s a messy way to do it, but you can. And it would still be fast.

Or perhaps you’re more sober, and want to create a SAN for your network, starting off with a 4-disk RAID10 with smaller drives. But you’re worried the space might fill up, and you can’t have the files span multiple filesystems. If you use LVM, you can just put in a second array at a later date, and expand your logical volume seamlessly.

 LVM Details

From a designer and administrator standpoint, you can think of LVM as having 3 main components:

  1. Physical volumes (devices or arrays).
  2. Physical volumes you bundle into made-up groups.
  3. Logical disks created and carved out of those groups you made.

This is where your planning comes into play (or not if you’re bad).

I’ve gotten into the habit of always using LVM to create the volumes which I will then format into filesystems. This gives me the flexibility to add more capacity, or take some away, at any point in the future.

The Debian distribution has long supported creating and installing itself into LVM volumes. Most of the major distributions now seem to support this as well.

So, using LVM, it’s a simple matter to take your 6TB RAID array and make a nice little root drive of 10GB for Debian, and a 5TB home directory, that you could then share with another 10GB partition you might make later for, say, Fedora.

Of course, you could partition your RAID array with extended partitions to achieve much of the same result, but you would not have nearly the flexibility later.

The Nitty Gritty

Working with LVM requires that you have the LVM tools. In Debian you can get these installed for you automatically if you choose to install onto an LVM volume, or you can get them later with

apt-get install lvm2

Instead of “normal” devices for hard drive volumes,  you’ll get logical devices that are create by the device mapper. In Debian, at least, they live under /dev/mapper and you also get a prettier version of the device names you create under /dev/<volume_group> — and you get to choose the <volume_group>

Creating Physical Volumes

The first thing you need to do is decide which devices you’d like to take over as LVM-controlled beasties. You can also specify individual partitions of disks if you like. The point being, you don’t have to even partition a disk first – you can specify the whole device if you like. Or partition it if you prefer. People will always argue over what’s best, and there is no clear winner. Which means, do what you like!

Let’s say you have 2 SATA3 drives, /dev/sdb and /dev/sdc. And you want to use them for LVM. The first thing to do is claim those physical volumes for LVM:

pvcreate /dev/sdb /dev/sdc

And lets suppose later you decide you also want to use your RAID 1 drive (/dev/md0) for LVM as well. You can designate any physical volume to be used with LVM, at any time.

pvcreate /dev/md0

Oh, I also have a USB 3.0 drive I might want to do something with:

pvcreate /dev/usb_drive_0

Do be careful though – consider any data on these destroyed.

Creating Volume Groups

Once you have some physical volumes in your system designated for LVM use, you can start grouping them up together. Creating volume groups doesn’t give you anything you can format into a filesystem, it just groups together whichever physical volumes you want to use into a named group that you can reference them by as one.

Think of it as the first abstraction layer — which of these physical devices do I want to group together and use as a big pool of space, from which I can make my littler hard drives.

You might want to consider how fast the drives perform when grouping. For example, you probably don’t want to group your super fast RAID array in with your USB drive. But maybe you’re fine grouping your RAID array in with your other SATA drives.

Yes, yes indeed, we’re fine with grouping our RAID in with our SATA drives. It will be fast. And the USB will be slow. How about we name these groups to reflect the fast and slowness of each. Maybe I can use the slow for backups?

vgcreate fast /dev/md0 /dev/sdb /dev/sc

That’s it. Easy to create a volume group. Just pick a name, like “fast” as we did here, and then list the volumes you want to use in it — ones that you created with pvcreate. And if you’re wondering, yes you can skip the pvcreate part above — but don’t — until you’re very sure that all this structure stuff is going to stick in your head.

Now, even though we only have one device in our USB drives, we may later add more nonetheless. So why not put it into an LVM group now – then later if we need to expand our backup area with more space, we can add a second USB drive, and just add it to this same group for instant gratification.

We’ll call this one slow:

vgcreate slow /dev/usb_drive_0

So we now have two spaces we can work with – the “fast” space, and the “slow” space. These are the volume groups. And we no longer have to care about which devices make them up. Well, until something goes wrong.

I suppose it’s worth noting that LVM provides no redundancy or backups, by default, although there are some great mechanisms built into it for doing so later, in some… interesting ways. So it’s really best to make sure your physical volumes have the level of redundancy and protection you want, before making those physical volumes into logical ones. I almost never make an LVM volume out of anything but a RAID array, unless I just don’t care that much about losing data, because I’m so awesome about backups (mm hmm).

Creating Logical Volumes

Now you can create logical volumes in your volume groups, and these logical volumes you can format just like hard drive partitions.

When you create your groups, LVM will allocate all the space possible on those devices, and make it available to you to create your logical volumes in. If you ever want to see how much space you have:

vgdisplay fast

You’ll get to see how much space you have, and also how much space has already been allocated to logical volumes.

Now, lets say I want to have a 100GB “disk” for my database files. Fast disk access is important for database work, so we’ll put that on the “fast” volume group:

lvcreate -n database -L100G fast

This will create a logical device of 100GB size called /dev/fast/database (at least in Debian). A more pedantic device name is also created called /dev/mapper/fast-database but I prefer using the first naming convention.

You can then use this newly-created device just like you would any hard drive. You can partition it if you like, or you can just create a filesystem on it:

mkfs.ext4 /dev/fast/database

Now, if you decide you just must partition these, they can be a little tricky to mount. Using a tool called partx you can extract that partition table out into devices you can work with, but we won’t go into that just now, here.

But if you skip the partitioning bit, these logical volumes mount just like normal filesystems, except really, they can be coming off of any number of drives in your volume group.

mount -t ext4 /dev/fast/database /var/mydatabase

Easy as pie! (when you don’t have to make the crust)

The same holds true for your slow USB drive:

lvcreate -n backups -L 2T slow
mkfs.ext4 /dev/slow/backups
mount -t ext4 /dev/slow/backups /mnt/backups

Here we created a “drive” of 2TB called “backups”, formatted it, and mounted it at /mnt/backups. This would be our USB drive.

Adding More Drive Space to an LVM Logical Volume

Now that we’re demystified, and have been using our system just fine for weeks, we ended up filling up our /dev/slow/backups with backups. Let’s say our USB drive was 3TB. We only allocated 2TB to our logical volume for backups, so really we have 1TB left free in the “slow” volume group. We could see this with

vgdisplay slow

So, if we wanted, we could just increase the size of our logical volume /dev/slow/backups to eat up that extra 1TB that exists in that volume group, and so give us the full capacity of that USB drive:

lvextend -L+1T /dev/slow/backups

Here we give a relative size in the -L parameter, saying we want to add 1TB to the already-allocated size of 2TB, making a total of 3TB. Of course, this doesn’t make the filesystem bigger, just the “disk” the filesystem is on. But ext4 is easy to resize:

resize2fs /dev/slow/backups

By not specifying a specific size, resize2fs will just make the filesystem as large as it can on that “drive”. And you don’t have to take the disk offline either — this is an online resize, and I’ve never had it fail. But of course you should do it offline. But I never will.

You can also shrink filesystems and volumes, but I’m not going to go into any of that here because shrinking filesystems is unnatural.

But suppose even 3TB is not enough for your backups. Well, it’s easy to add a new device to a volume group, and make it available for logical volume to eat up in its gluttony.

You buy another USB drive! So now we need to add it to the “slow” volume group. You got a good deal on 4TB drive this time. So you have your original 3TB one, and now a 4TB one you’re going to use to expand your storage. Assuming it’s assigned /dev/usb_drive_1

First, create the physical volume, as usual:

pvcreate /dev/usb_drive_1

Then add it to your “slow” volume group:

vgextend slow /dev/usb_drive_1

After you do that, if you look with

vgdisplay slow

You’ll see that you now have a whopping 7TB volume group called “slow”.

Then you just do the same as above to extend your logical volume, or “drive”, for your backups:

lvextend -L+3T /dev/slow/backups
resize2fs /dev/slow/backups

Et voila! You now have a 6TB backup “drive”, because you didn’t use the full 4TB off the new one. You left 1TB free in the “slow” volume group, because you’re not always a complete hog, and you may like to use that 1TB for some other volume in the future, like:

lvcreate -n pron -L1T slow
mkfs.ext4 /dev/slow/pron
mount -t ext4 /mnt/relax /dev/slow/pron

Because, well, if not a hog, then perhaps a pig. Oink.  And that uses the last of the drives.

Conclusion

You can, of course, remove devices from LVM groups as well. You just need to make sure you have enough space in the volume group to do so. For example, if you have 2 USB drives, one 3TB and one 4TB, making a total of 7TB — if you’re using up 6TB in logical volumes, you won’t be able to take away either of those USB drives, unless you can shrink down your logical volumes first (after shrinking your filesystem first, unless you like agony).

The point being, LVM is quite flexible.

One of my other favorite features is the ability to take a live snapshot. This is great for backing up whole disk images of a live-running system. You create an LVM “snapshot”, and then you can dump that image anywhere you like, and the filesystem will be in a consistent state, even with the system running. It’s wonderful for virtual machine images especially.

But I’m not going into that either here, just  yet.

Hopefully this might have helped you a little. I remember when I was first looking at LVM it was a confusing mish-mash of all these different options, and nothing spelled it out simply. Hopefully, this has managed to do so, if you’re looking to get started with it, for play or production.

And as always, check out the man pages. There are lots of other places out there too with more detailed and specific use cases and feature examples.

Honestly, using LVM has changed everything for me. It’s certainly worth looking at. Best to you!

Fix for MRTG Generating SNMP_Session Error in Debian Wheezy (and possibly Ubuntu)

Lately, after an upgrade from Debian Squeeze to Debian Wheezy, MRTG is sending emails every few minutes when it runs from the crontab. This is quickly filling up my Inbox. There error message is as follows:

Subroutine SNMP_Session::pack_sockaddr_in6 redefined at /usr/share/perl/5.14/Exporter.pm line 67.
 at /usr/share/perl5/SNMP_Session.pm line 149

After waiting for a while to see if a fix came from Debian, I decided to look around on my own. It seems there is a patch that works, but it has not been propagated out to the repositories. Luckily, this error problem is easy to fix.

You can apply the patch at the link above, or you can just edit a file and make 2 quick changes: Edit the file

/usr/share/perl5/SNMP_Session.pm

Change line #149:

old: import Socket6;
new: Socket6->import(qw(inet_pton getaddrinfo));

Then change line #609:

old: import Socket6;
new: Socket6->import(qw(inet_pton getaddrinfo));

That seems to fix the problem quite well. Hopefully the Debian maintainer will get that change in sooner that later so others don’t have to bother!

Note: Someone commented that the line numbers listed were a bit off from version to version of Debian. Not entirely unexpected. It’s the change to calling the class directly that counts.

How To Deal With Udev Children Overwhelming Linux Servers With Large Memory

A few months ago a client purchased a new server and asked me to set them up with Linux-based virtual machines to consolidate their disparate hardware. It was a big server, with 128 Gigs of memory. I chose to use Debian Wheezy and QEMU KVM for the virtualization. We’d be running a couple Windows Server instances, and a few Debian GNU/Linux instances on it.

Unfortunately, we ended up encountering some strange problems during heavy disk IO. The disk subsystem is a very fast SAS array, also on board the server, which is a Dell system. The VM’s disks are created as logical volumes on the array using LVM.

Each night, in addition to the normal backups they do within the Windows servers, they also wanted a snapshot of the disk volumes sent off to a backup server. No problem with LVM, even with the VM’s running, when you use the LVM snapshot feature. This does tend to create a lot of disk IO, though.

What ended up happening was that occasionally, every week or two, the larger of the Windows server instances would grind slowly to a halt, and eventually lock up. The system logs on the real server would begin filling with timeouts from Udev – about it not being able to kill children. This would, in turn, effect the whole system – making a reboot of the whole server necessary. Very, very ugly, and very, very embarrassing.

I tried a couple off-the-cuff fixes that were shots in the dark, hoping for an easy fix. But the problem didn’t go away. So I had to dig in and research the problem.

It turns out that Udisk (which is part of Udev) decides how many children it will allow based upon the amount of memory the server is running. In our case, 128G – which is quite a lot. This number of allowed children was a simple one-to-one ratio, based upon memory. However, with this much memory, that many children seemed to be overloading the threaded IO capacity of this monster server, causing blocks, during live LVM snapshots being copied.

What I ended up doing was manually specifying that the maximum number of allowed children for Udev would be 32 instead of the obscene number the inadequate calculation in the Udev code came up with. Since doing this, the server has run perfectly, without a hitch, for a good, long time.

So this is for anyone who may have run into a similar problem. I could find no information about this on the Internet at the time. But I did manage to find how to effect the number of children Udev allows. You can do it while the system is running (which will un-happen once the server is rebooted) or you can put in a kernel boot parameter to effect it, until the Udev developers fix their code to provide a sane value the maximum number of children allowed in systems with a large amount of memory.

At the command line, this is how. I used 32. You might like something different, of course.

udevadm control --children-max=32

And, as a permanent thang, the Linux kernel boot parameter is “udev.children-max=”.

Hopefully this will save some of you some of my headache.