Archive for the ‘System Management’ Category

FreeNAS iSCSI and VMWare ESXi

Friday, September 3rd, 2010

For a long time I’ve been using NFS to serve the diskspace my FreeNas server has, to my ESXi machine, and ‘ve been wondering if iSCSI would be a better solution. NFS has more overhead than iSCSI, so iSCSI should outperform NFS on higher workloads.
I’ve never really taken the time to get to know iSCSI and it alwasy looked very messy and difficult to set up and use. But today, that idea changed!

We’ve been running ESXi at the office for a couple of weeks too in a test environment, and are investigating the necessities for a production installation at one of our clients. They’ll have 2 servers (1 master, 1 ‘spare’) and a shared NAS/SAN solution to store the virtual disks. As performance is necessary there (I didn’t really care for my own setup at home, as long as it is reliable), I was looking for a simple howto description to use iSCSI on the ESXi side, and tried that with my own FreeNAS storage machine.

Following this guide I found a detailed description of the configuration needed in FreeNAS to get it up and running in 10 minutes. It really did it!
An iSCSI datastore is connected now onto my ESXi machine, for testing. I only took 50GB to test with, as I’m thinking of redesigning my FreeNAS box with another motherboard, raid controller and disk setup, as the current Dual AMD Opteron with 12GB RAM is a bit overkill, and too power hungry… As I don’t have a decent motherboard that supports the Raid controllers I currently use (PCI-X based), I need to rethink the setup and get me a decent PCI-e x4/x8 controller capable of holding 8 to 12 disks (2 controller setup is just fine, even 3 controllers might do if it costs less that way.) When I finish that migration and get all the data copied over, I’ll setup the FreeNAS-ESXi connection on a purely iSCSI based setup…

We’ll see…

Posted in Server Management, System Management, Virtualisation, hardware, linux | No Comments »

Linux Virtualization – OpenVZ/Proxmox

Wednesday, August 11th, 2010

In times of economical troubles, a small company needs to look for money-saving ideas and use them to take their infrastructure to a less costly setup with the same (or at least nearly) power and possibilities.
For that reason, over the last 2 years we were looking into both energy efficient hardware and virtualization solutions to get the most out of our machines in the datacenter. Every single Ampère costs us a lot of money every month. To save energy, we fitted our backupservers with Western Digital Green Power drives instead of the standard Raid Editions we had before, as we did have to expand the Raid volume anyway.

Another way of reducing the power needs is virtualization some of the separate machines, so the total of our servers could be scaled down a little.
We had a couple of servers used for web- and ftp traffic, which weren’t utilized 100%, not even 25%…
After an inside testing setup with OpenVZ, a container based Linux virtualization solution, we looked into expanding it onto our datacenter installations. The big problem here was the VLAN setup we use in the datacenter, to separate all client networks. Apparently OpenVZ doesn’t really cope well with VLAN’s, or at least there is not that much info about the usage and configuration of setting up containers in separate VLAN’s.
Looking into GUI-possibilities, web based management options for OpenVZ, we stumbled upon Proxmox VE which is an all-in-one distribution, supporting OpenVZ, KVM etc as virtualization options, and which has a nice and full-featured web based management console.

Setting up machines in Proxmox is really easy. You can choose which virtualization environment you want to use (KVM or OpenVZ containers). KVM allows you to run different OS’es next to each other, like Windows servers, BSD etc, while OpenVZ uses the same basic kernel, but allows the containers to be run independently, giving a very low overhead, and maximum performance.
OpenVZ containers are ‘template based’, and Proxmox has a list of already defined and downloadable templates like Debian 4, 5 and 6, Ubuntu 8.04, their own Proxmox email security suite (I think you need a license for this), but also predefined WordPress, Drupal, Zenoss monitoring, Joomla, Mediawiki, …
You can download the ones you need or want to deploy, and a couple of minutes later, the template is ready to be used (depending on your internet connection speed, off course)
Once you have one or more templates, defining a container’ed machine is easy as hell: Create a new ‘machine’, choose OpenVZ as the type, select the template, define a root password for accessing the machine, and select network, disk space etc… When clicking the create button, Proxmox will set up your environment, which will be ready for use in minutes.

VLAN integration with the Proxmox web console is really easy: all you need to do is define extra interfaces as ethx.yyyy where x is the ethernet interface number, and yyyy is your VLAN ID.
Once those were all set, we could define a container, and assign a vmbr bridged network connection to it, which holds the VLAN tag info.

We migrated 4 ftp and webservers in a matter of hours instead of days. Most of the time went into syncing the data between the old and the new machines.

An extra option in Proxmox is defining separate storage possibilities, for example local storage, iSCSI, NFS, … for (automated) backup and snapshot storage…

Proxmox hosts can be ‘joined’ into a cluster, so you can migrate containers between hosts, without downtime, interruption, big troubles or manual intervention. That way you can move machines needing more memory, or migrate to heavier hardware, perform hardware maintenance and have the most important machine run on a backup host server, without the long-time interruptions, and stress accompanied with such a migration or maintenance.

Proxmox is, for Linux virtualization, my favorite now, far above any other solution we have been using in the past.

Posted in IT-news, Server Management, System Management, Virtualisation, linux, ubuntu | No Comments »

Nagios3 on Ubuntu

Tuesday, March 30th, 2010

I reinstalled a low power Via C3 machine to monitor my network, now almost 90% of my machines run on ESXi… Sure I didn’t want to monitor a virtual host from one of the virtual guest it runs… when the host goes down, what good will do monitoring from within?

The machine was installed using Ubuntu Server x86 9.10, the most recent stable one, and after basic installation, fixing the IP, setting a hostname and apt-get update, apt-get upgrade and apt-get dist-upgrade, I installed nagios3 and all its extras from aptitude.

Configuration was quite easy, I just had to take over the config I already had running on an older box which will be shut down later this week to save energy (it’s an old power-hungry Pentium 4 machine with too many single-disk storage blocks). After logging in, I noticed a lot of the services were ‘muted’… Notification was disabled on all HTTP checks etc…

When trying to enable the notification for a service check on one of the machines, I got the error Error: Could not stat() command file '/var/lib/nagios3/rw/nagios.cmd'!
I checked the rights on the file, owner and group ownership, but nothing wrong was to be seen there.
Googling around, I stumbled upon a short post on Yes! That Works Fine! and tried the 4 command lines.
I had to change it a bit so the original code:


stop nagios2
/etc/nagios2# dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios2/rw
/etc/nagios2# dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios2
start nagios2

became:

/etc/init.d/nagios3 stop
/etc/nagios3# dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
/etc/nagios3# dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
/etc/init.d/nagios3 start

After retrying to enable the notification for the HTTP check that caused the message above, it worked without any error, and the notification is working again.

Posted in Server Management, System Management, linux, ubuntu | 2 Comments »

Rescanning sata/scsi devices on non-hotswap capable controllers

Tuesday, January 12th, 2010

I wanted to use the hotswap tray I recently installed in my computer case, and easily plug in the 1.5TB disk to make a backup of the backup server, before reinstalling it in a couple of days.
When plugging in the disk, the Ubuntu system didn’t see it appearing, or at least, I didn’t notice or know where to look for it.

Looking around (googling) for a ‘sata rescan’ I found a thread describing how to rescan the scsi/sata bus (as sata is seen by the kernel as scsi).
It quite simple, once you know which bus holds which disk…

It works as follows:

Become root:
sudo su
go to the directory for the host/controller in /sys/class/scsi_host/hostN where N is the number of the host

and issue this command:
root@ubuntu:/sys/class/scsi_host/host0# echo - - - > scan

in dmesg you will see the new devices found on this controller/host appearing, and the appropriate /dev/sdX device directives are created automatically.

Easy, no?

Posted in System Management, hardware, linux, ubuntu | No Comments »

And then there was Zimbra!

Wednesday, December 9th, 2009

Zimbra, the collaboration suite, gives you a complete MS Exchange server replacement, as far as I can see.
Email, Contacts, Calendar, Documents, Tasks, … everything is in there!

There is an Open Source edition, where you can get the basics for free, without support that is, and can install it on a number of systems. According to Wikipedia and the FAQ it consists of a number of Open Source packages such as postfix, clamav, and such (full list is on the wikipedia article), combined to make a solid Groupware suite.

I recently re-started looking to replace the old Exchange 2003 system I’m still running in a VM, but was always confrontend with either a do-it-yourself approach (combining an mta, antivirus and antispam solution, some ical stuff,… myself), or a you’ll-have-to-pay-a-fee for a complete basic solution.

I knew about Zimbra, and had played a bit with it a couple of years ago. In the beginning of this year, I started looking at the Zimbra Desktop as an email client with integrated calendar and task management modules, because of the platform support. It has an installer routine for Windows, Mac OS X and Linux, and could easily replace Outlook and other proprietary email/groupware clients at some of the customers of the firm I work for.

The server side installation was done in a VM’ed Ubuntu Server 8.04 LTS 32 bit running under VMWare Server 2.0. Installation is quite easy for the Open Source edition (and I presume that the installation routine is the same for all editions…?). You run the big installer package you downloaded, and answer a couple of questions regarding desired modules. It came to tell me a couple of dependencies weren’t met, which were easily installed through apt-get.
After the installer finished, it proposed me a list of settings. In this list, the installer required me to set an admin password, which wasn’t all that clear at first. I asked to quit, because I didn’t see what the installer needed. Seeing, after quitting, that it didn’t start the server daemons, I knew I missed something. I quickly did a reinstall of the package, and when the list was proposed again, I noticed the *** before the password line.
After setting the password, I could Activate the settings, which wasn’t possible before.
Once activated, the install routine ended with a start of all the services.

I redirected a browser to http://serverIP:7107 and was prompted for a username and password. I logged in with the details I entered in the installer, and got a nice and complete configuration module.
Setting up a user for myself, and fiddling around with the settings, the first installation was completed.

I changed my MX records to the newly installed Zimbra server, and noticed mail coming in… It was working.

Next up:
-setup another user to check calender sharing and such
-zpush configuration for Activesync based push email and synchronization
-export-import Exchange mailbox into the Zimbra server mailbox
-shutdown Exchange server VM at last ;)

I’ll keep you posted.

Posted in Server Management, System Management, linux, ubuntu | 2 Comments »

MythTV 0.22 on Ubuntu 9.04

Thursday, December 3rd, 2009

I had a major issue at home with my MythTV backend server… and off course, right before I was implementing a decent backup strategy.

I wanted to move to a Raid5 solution to increase speed and disk capacity, but wanted to do an Ubuntu 9.04 reinstall, move all data and config over from the old single-disk install, and keep running on settings and recordings already in the database. At the same time I wanted to upgrade the backend to Ubuntu 9.10, and use the newer MythTV 0.22 packages.

My install went pretty fine, using software Raid 5 on 3 disks I still had lying around (WD Raid edition SATA2), formatting the 500GB usable space as an ext4 volume, and installation of MythTV and other related packages was done without any issues. I started copying from the old single-disk setup to the new soft-Raid volume, and was almost done migrating. The configuration of MythTV through mythtv-setup couldn’t find my Technotrend S-1500 DVB-S card I used for months now, without any issues on Ubuntu 9.04… so I needed to have a look at that… but I even didn’t get to that point!
All of a sudden, the session over ssh was having some big issues, not finding commands, giving errors etc… Returning to the console, a lot of raid error messages were displayed, so a reboot and recover/rebuild was needed… the machine didn’t take any commands anymore… even a ‘halt’ or ‘reboot’ didn’t work…
When powering down and back up, the machine hung at the kernel loading, giving me a serious error not retrieving md0 anymore… Panic!!!

I disconnected my original disk, booted a Live session of the Ubuntu Desktop 9.10 cd (ext4 is better supported in 9.10 than it is in 9.04), and tried almost everything I know and could find about retrieving data from a faulty software raid volume… to no result…
I decided to do a fresh install on ext3 instead of ext4 with the Ubuntu 9.04 distro, to resolve the card-support issues I was having with Ubuntu 9.10, which worked out pretty fine again.
Card was detected, and scanning gave the results I expected.

I hooked up the original single-disk install SATA drive, and wanted to start copying the database, recording and settings files, but I wasn’t able to mount any of the original partitions on the disk. After some looking around on the disk, I found all partitions vanished! The disk was empty! Even more panic!!!!

Testdisk, an open source partition and data recovering program, had me regain access and information of the recordings partition, and I immediately made a copy of the data to my backup-server, but still no /etc or /var/lib/mysql to be retrieved to get access to the original database and settings… So I need to start from scratch, and convert the old recordings to standard mpg or avi files, so I can at least play them without info etc… That’s not the end of the world!

Now, the MythTV 0.22 upgrade part was going to be the hardest…, or so I thought.
In Ubuntu 9.04, the 0.22 MythTV packages aren’t available through the normal repositories…
I searched in google, and eventually found a page on the MythBuntu site that describes the necessary steps and packages needed to upgrade a 9.04 (or even a 8.10) Ubuntu distro with the MythTV 0.22 release…
The MythTV backend server was easily upgraded through the use of the MythBuntu Repositories package
After installing this deb file through dpkg -i, going into aptitude and doing an update/upgrade, the MythTV packages were upgraded to 0.22 automatically.

I tried connecting with my 0.21 based frontend machines (which run Ubuntu 8.04 LTS), but found that the database structure was changed in such a way the 0.21 cannot contact and use the 0.22 database structure… An upgrade to 0.22 on these machines is needed as well.

I installed the same MythBuntu Repositories package package on these 2 machines, and found it could only install the already installed 0.21 version. I needed to upgrade the distro to 8.10 or higher… I decided to go all the way up to 9.10…

The standard AMD based pc which resides in my living room and is hooked onto the beamer, is the easiest… if anything goes wrong, I can just reinstall it with 9.10 from scratch…, so I started with this one before messing up the MythBuntu 8.04 installation on the AppleTV in the bedroom (which is a lot more work to do a complete (re)installation).
Though not supported, I changed over the /etc/apt/sources.list hardy listings to karmic, and did an apt-get update followed by an apt-get upgrade and an apt-get dist-upgrade… I met a lot of errors installing or configuring packages, and needed to uninstall a lot of packages through aptitude, but eventually using apt-get dselect-upgrade and another apt-get upgrade and dist-upgrade, the machine eventually ended up booting fine with Ubuntu 9.10.
I installed the MythTV 0.22 packages through aptitude and ended up configuring the connection to the newly installed database on the backend. Two minutes , and some screen adjustments later, I was looking at LiveTV through the new 0.22 MythTV release!

For the AppleTV, I will follow the same things, but I’m making a full rsync of the complete filesystem first. If anything goes wrong, I can just rsync it back over it, and have the machine working again…

Now I just need to convert all recordings to a viewable format (mpg or avi) and change all the filenames so I know what recording is in what file… that will take another weekend to do so, I think.

First impressions of 0.22: faster, slicker, … but I didn’t see that many adjustments or changes… (except from the new theme stuff, which was the biggest change announced off course)

Next on the list: BACKUPS!!!! I have a backupserver giving me 2,3TB through NFS of which still 1,2TB is free, why wouldn’t I use it??? :p

More to follow later…
At least now, I’m up to date again…

Posted in AV/TV, MythTV, Server Management, System Management, hardware, linux, ubuntu | 6 Comments »

Arkeia free Enterprise Backup solution for Ubuntu

Monday, November 16th, 2009

According to this page found through this article Arkeia is providing a free license for their Enterprise grade Backup solution Arkeia Network Backup version 8.

The license includes 2 agent licenses for Windows, Mac OSX/BSD and Linux machines and the server software is included in the Ubuntu 8.04 repositories.

I’ll install it on my server for sure!!

Posted in IT-news, Server Management, System Management, linux, ubuntu | No Comments »

Virtual Desktop

Thursday, September 17th, 2009

I have been playing with the idea for a long time now: No more noisy, power-hungry desktop on my desk at home… just a simple, noisless, thin client of some kind.

At one of our clients, we use Igel thinclients that connect to an Ubuntu Desktop -server, over XDMCP.
XDMCP is rather easy to configure and use, and allows full X over networks, so all processing is done on a central machine.
I have a VMWare Server 2.0 and a Citrix XenServer 5.5 machine running for other virtualized testing machines, so why not integrate the ‘Desktop server’ in the virtual setup, to even cut more power costs?

I used an old Wyse 9450XL I bought at the computer fair over half a year ago, formatted the seated 512MB IDE flash drive and ran Damn Small Linux on it, which can work both locally and through an XDMCP session on my Desktop server.

I’ll install FreeNX on the server side too, and compare the speed differences, but as far as I know and remember, DSL does not have the needed recent versions of several libs, so that might be an issue…
Well, we’ll see. For now I’m set.
Desktop ‘computer’ setup only takes about 20W instead of about 200W before with a fullblown PC.
Advantage of the FreeNX way is that remote sessions are also possible… Goodbye to slow VNC connections!

I’ll post some configs, like the gdm.conf on the server side, later.

Posted in System Management, hardware, linux, ubuntu | 2 Comments »

FreeNAS v0.69

Wednesday, March 25th, 2009

I decided to give my NAS box another go, reinstalling the machine (after a backup to a 500GB USB disk) with the latest v0.69, as the previously installation didn’t recognise the onboard nvidia gigabit ethernet card.
This one does, and brings a complete new interface and a couple of newly added features.

The backups are running again, and I’m trying out the FreeNAS in NFS and iSCSI mode as a storage backend for a Citrix XenServer 5 installation which I’m testing right now…

The Citrix part I will cover later.

FreeNAS, and I cannot say this enough, is just wonderful!
NFS, FTP, SMB/Cifs and lots more, and now even uPNP and iTunes/DAAP support!
And for free!

Posted in Server Management, Virtualisation, hardware | 2 Comments »

FreeNAS as a backup server

Sunday, December 7th, 2008

I recently had a big issue with my backup machine, on which I had a couple of disks mounted, and where I copied some important data to, once in a while, as a sort-of-backup strategy.

Off course, the one disk holding the most important data died, and I didn’t notice fast enough… And when the shit hits the fan, Murphy decided to crash my MS Exchange setup and database on my testing machine, which held _all_ my mails from the past 8 years… (very important things, config backups, license keys, important addresses, passwords, …) I realised: now is _the_time_ to get a decent backup strategy set up.

I already had a small 2U rackmountable box which I got from a former colleague and that could hold 5 normal 3,5″ disks. I had 3 Western Digital Raid Edition 250GB SATA-II disks in the old ‘backup server’ that could be wiped after some copying, and an Athlon XP 2600+ without any SATA on board. I plugged in 2 SiS PCI 2 port SATA cards and ordered a fourth WD RE 250GB disk at work.

Looking for a decent setup, easy to use, easy to backup and stable enough to count on, I started checking google, and almost instantly got the FreeNAS website as one of the best results. Decent background, FreeBSD based, good hardware support, easy to manage Software Raid setups and loooots of protocols for file transfer. I downloaded a LiveCD and gave it a try. At the first sight, I was overwelmed… NFS, SMB/CIFS, FTP, SSH, AFP, Rsync, and even iSCSI Target were among the list of supported services. Software Raid was easy and fast, and the hardware support was very good.
I started building a system around the old AMD Athlon XP 2600+ I still had lying around, and as AMD’s are pretty power-friendly, I thought this was the best solution to start with. I hooked up an IDE-CF adapter, put in an old 128MB CF Kingston card and installed the live CD onto this card. I started defining a Raid 5 volume on UFS, giving me a total of 715426MB of storage, around 700GB which was more than I need as a backup server. When finalizing the Raid setup, I was asked the option to Initialize And Format the Raid ‘device’, and I said ‘hell yeah!, go for it. make it even faster to start’

The initialization started, but for some reason, it always stopped working after a couple of minutes/hours, and the system just froze up. Nothing to check, nothing to see. Keyboard and screen just locked, No info… Giving the machine a simple Reset didn’t do anything: the Raid volume began Checking and Verification, and after about half an hour, freeze was there again.

At a local computer fair, I acquired a new Nvidia nForce-with-geforce-onboard mbo, and a new LE2xxx series AMD cpu, the total package costing me no more than €60. The board has the needed 4 SATA-II ports, so this one looked to be the perfect match.

Once the board was set in to the case, the CPU installed, heatsink clicked on, Ram and disks installed in place, I got the system installing into this new hardware setup. No network card was found, so I had to put in a PCI Realtek 8139 based card to enable the network connection. A restart later I was able to finish the Disk and Raid setup.
After clicking the ‘Initialize Raid Volume’ again, same things happened: system froze up after a while, nothing to be done… No access at all. A simple Reset didn’t help either. It just took 20 minutes and the system locked up again.

I was panicing now. I had the system up and running on a single disk before, no problems. I checked all four disks, no issues. I checked the power supply (450W should be enough, PSU was brand new), nothing there either. I checked the CF card, and replaced it twice. No change… What now?

I completely started over, as there is still no data on it I cannot miss. A full reinstall, reconfig, although this time, I didn’t set the mark for the automatic ‘Create and Initialize RAID’. I let the Raid array build itself, verify and waited until I got the Completed message (which took about a night… 700GB is a lot of diskspace to be verified and initialised). After that, I did the last steps: mounting and exporting the Raid volume through NFS and SMB/Cifs. It looked as if it worked without any error.

The machine had been running for two days since, and no lock ups, no errors.
I started putting in backup schedules (an rsync batch job scheduled through cron from 2 servers holding important data, and an ntbackup job backing up the recovered Exchange database and Active Directory info needed to restore all email stuff if Murphy would strike me again…) and let it run for a few nights, and that too looked pretty good.

As the newly bought mbo/cpu could come in more handy for another machine, I decided to replace this combo with the originally chosen hardware. I labelled the disks and SATA connection cables 1-4, and set them in what I thought would be the right order on the PCI SATA cards. The disk id’s chowed ad4-ad6-ad8-ad10 instead of what used to be (on the new nForce mbo with onboard SATA-II) ad4-ad5-ad6-ad7 and I noticed I switched disk 3 and 4 because of the cable for number 4 wasn’t long enough to reach the card. However, the Raid volume came back up after completing the boot, and even better: all testing data was still there and available. I guess FreeNAS built in some disk-id recognition into its Software Raid setup and config. The system was up and running in 5 minutes, without the need of a complete reconfiguration. How sweet!!!

Now, let’s see if this hardware keeps up, so I can use the newly bought AMD board for some other stuff.
Backups are set, in place and running. Let’s hope I don’t need them very soon!!!

Posted in IT-news, Server Management, System Management, hardware, linux | 2 Comments »