Hacking for fun and profit

Things have been moving in the Feedjack front. There are new planets using it, like Debian Perú, Planet SCM and Planet Python@TW, but also apps that use use feedjack on the background like Ian Holman’s Economy Chat and VC Chat.

Also in the news: Feedjack just went into the FreeBSD ports, and I hope to see more Django apps get in there soon.

About the new home, Feedjack is now hosted in www.feedjack.org, and there are some new services too (thanks to Google):

• Subversion repository: http://feedjack.googlecode.com/svn/ (instructions)
• Issue tracker: http://code.google.com/p/feedjack/issues/list
• Mailing lists: one for discussion and another one for ticket updates and SVN commits

I’m planning to work again on Feedjack very soon, in the meantime don’t forget to report bugs and send your suggestions to the issue tracker.

Months after I discovered FreeBSD’s excellent performance, and because my debian unstable desktop became, well, unstable, I decided to give FreeBSD as a desktop a try.

My desktop needs are pretty basic, I just need:

• GNOME: Originally I migrated from fluxbox to Gnome to “try linux as a real final user”, now I just can’t live without it. It is a solid, simple and elegant desktop. I need an enviroment with excellent Gnome support.
• Mplayer/xmms: I like to watch movies while I work. It helps me to relax. When I need to focus a little more (when I am programming), I need xmms for some good old Ludwig Van.

That is pretty much it. Both Firefox and xterm are everywhere, and I take them for granted.

FreeBSD covers this pretty well. The FreeBSD Gnome project provides a great enviroment and I was very happy with my desktop. I could see ALL my videos with mplayer, and I didn’t have flash installed in firefox (I HATE flash).

But after months of bliss, problems arised:

First, when I tried to sync my palm wigh Gnome, it didn’t work. I remember this was damn easy in Debian. I just forgot about it and never used my palm again. I really didn’t care too much.

But then, something REALLY annoyed me.

When the Snakes and Rubies videos (Django and Ruby on Rails) were released, I couldn’t watch them in FreeBSD. I had to go to use a Windows box. I really tried everything I could but the codecs just didn’t work in FreeBSD. I gave up.

After that, an email in the django-users mailing list mentioned that the snake and rubies videos were in Google Video. The problem is that Google Video uses flash. I hate flash. I hate flash sites. Flash is awful.

But google video and youtube (also flash) are great sites, and have great stuff. I was missing all that.

This weekend I decided to install flash in FreeBSD.

I did. I installed the most recent “supported” version of flash in FreeBSD: Flash6. The problem is, both Google Video and youtube need Flash7 to work. And flash7 support in FreeBSD is pretty bad (there is no flash player for FreeBSD so it must run in linux-emulation mode).

Now, Installing Flash in FreeBSD is not as simple as it is in other OSs. For instance, the port that handles linux plugins for firefox is bugged, and wasn’t creating a plugins directory. I had to read Makefiles to see what was wrong and create that directory by hand.

Also, to install the Flash7 port, you must apply a patch in your core userland source code and recompile it. Can you believe that? Recompile the core of your operating system just to run flash? Anyway I did that. The docs said there was the risk in some systems that nothing would work.

And it didn’t work.

That was my last night with FreeBSD. I decided I needed another desktop. My list of candidates was short:

• Mandriva: Maybe, but it’s too KDE centric, I need a good Gnome desktop
• OpenSuse: Suse is good if you just install it and never touch it again. But it is by far the worst OS to upgrade. It breaks. It commits suicide. You have to fix things. It will waste your time. Avoid it like fire.
• Fedora: I used FC4 for a short period, a very solid desktop, but I still have nightmares with RPM.
• Debian: Now we are talking. The quintessential linux distro. My favorite linux for a server, but for a desktop, why should I use unstable when now I can use…
• Ubuntu: Ubuntu is what I was looking for. They are completly Gnome and Python centric. And best of all, it has everything good about debian (the packages) without what is not good about debian (the politics and the endless flamewars).

So I downloaded the latest beta of Ubuntu Dapper (beta4) and installed it this Monday. As usual with the new d-i I had no problems. But when I wanted to install
the extra (non-free) stuff, like the nvidia drivers, mplayer and the codecs, realplayer, acrobat reader, the flash player, java and all that, I just used easyubuntu and after a few clicks I had everything I wanted installed and working. In minutes. I couldn’t believe that. Last sunday I wasted THREE hours of my life trying (and failing) to make Flash work in FreeBSD, and now after a some minutes and a few clicks, I had even more software I ever had in my FreeBSD desktop.

Probably I should tell you about what a PITA is the installation of java in FreeBSD.

It is a big, very big, PITA.

In ubuntu you can have it installed with a couple of mouse clicks.

Now, I know there are lots of elitist kids around believing that because they use a “complicated” desktop (*BSD, gentoo, debian?, slackware, linux from scratch) they are technically superior. That attitude is stupid.

I have lots of stuff to do, and most of it is a lot more complicated than setting up a desktop. Believe me, I have better things to do than configuring mplayer or making my webcam work. I just want to plug in things or click buttons and everything should work. Ubuntu and OSX do that. FreeBSD and gentoo don’t. Ubuntu and OSX win. The ubuntu motto is “linux for human beings”, it should be “linux for professionals without time to waste”. FreeBSD is for servers. Ubuntu for my desktop.

And I will not, ever, waste three hours of my time configuring flash again. I can design, develop and test a web app in Django in three hours. There is no time to waste.

The Apache HTTP server is by far the most used web server in the world. It is an excellent, feature packed and standards compliant web server. Extremely configurable, with an endless amount of modules, superb documentation and, due to its license, is being used commercially by companies like Oracle. There is only a small problem with Apache: It’s not the fastest server around. This is because historically, Apache’s priorities have been correctness and configurability, not performance. Correctness and configurability are the reasons why Apache powers almost 70% of the web today, but still, Apache has a big, fat ass (we are talking about system resources here).

(more…)

It has been more than two months since I blogged for the last time. There has been a lot of things going on. The big thing is that now I am working full time in my own business: Aureal Systems, a consulting company in Peru. I’m working here with Breno, Benji, Fernando, Nestor and Nicolas (and Javier and Jackson, but they don’t have websites AFAIK). We have a month in our new (bigger and nicer) headquarters and now I have my own office. It is a mess of course (our OpenBSD router, Forlorn, is in my closet) but at least I can close the door and listen to Ludwig Van.

Aureal Systems specializes in secure computing. Everything we do is with security in mind, from (*NIX/Linux/BSD) server deployment to development. And since we specialize in security, we are listed in OpenBSD’s Support and Consulting site.

Business is going very well, a lot better than we expected actually, and that means lots of work, which I love of course :) I miss my son every time I have to stay in one of those (very frequent) all night long hack-a-thons, but it is for the best. As the title of this post say, Aureal is “A New Hope”. We are growing fast, sometimes too fast, but our customers are all very satisfied with our work

You can expect more activity in this blog for now on.

Debian is having serious problems with its core security infraestructure. The main issue being that there is just one (overworked) Martin “Joey” Schulze in charge of the security updates, and the rest of the security team is, well, busy. It is even said that one of the security team members is working in Ubuntu, which is perfectly ok since he is getting paid.

This is a very complex problem. First we have the Debian Organization. It has a large amount of developers and maintainers, and the process to be accepted as a new developer is both long and demanding. Now this has an advantage and a disadvantage: it is good because the process to become a developer somehow guarantees that the new maintainer will be both skilled and with knowledge about what debian, the policy and free software are. The problem is that this acceptance process is, sometimes, incredibly long. Take Ian Murdock as an example. He is the creator of Debian, even the name Debian means Deb (Ian’s wife) and Ian. Ian Murdock applied as a new maintainer more than a year ago, and he is still waiting for aproval. And he will probably wait for another year. This is somewhat discouraging for potential new developers (it was discouraging for me at least when I flirted with the idea of applying) and will become more problematic now that important core debian developers have been hired by Canonical to work on Ubuntu.

(more…)

I’m sorry for this post in spanish, it’s about a silly peruvian distro-flamewar and a friend just blogged about a PRIVATE conversation we had. If you can’t read spanish, just ignore this. (and if you DO know spanish, maybe you’ll want to ignore this anyway). Thanks.

Walter Cuestas:

Ya que te gusta ventilar conversaciones privadas en tu blog, por qué no pones el log para que se entienda lo del ejemplo de win32?

Como a tí mismo te gusta decir de los demás, “te vas por las ramas”. No te culpo por no entender el ejemplo en mención: no eres programador. Y de la misma manera no eres especialista en muchas cosas de las que hablas, mas sí en otras. Como ya te dije, nadie es experto en todo. Tienes que aprender a escuchar críticas, y ser mas caballero cuando te hacen ver tus incorrecciones, que en el caso de tu post anterior (el que originó todo esto), eran muchas.

(more…)

There has been some adoption of free software lately at my current job. I’m not talking about the servers running apache, postgresql, samba, postfix, squid, etc. I’m talking about free software in everybody’s computer. Note that everybody (but me) is still using Windows, but hopefully once they all use free software a migration to Linux will be painless, since people use applications and not operating systems.

The first free software application used around was the most obvious: Firefox. It started by word of mouth. It is faster, safer, smaller, it blocks popups. People love it. Not everybody is using it but still, it’s an spontaneous move, nobody is forcing them to do it (there was a squid rule for a couple of users with lots of spyware problems that forced them to use firefox, but it’s gone now).

Now the second move was an official one: OpenOffice.org. This is the BIG money saver. Microsoft Office costs A LOT of money, and most people just use Outlook and the most basic features of Word. Excel is used mostly to store lists of stuff, not as a spreadsheet.
Management gave me the freedom to prepare an OpenOffice migration. And it has been a success. Right now almost half of the people around here is using OpenOffice instead of Microsoft Office. Note that I installed the 2.0 beta. It has an excelent quality, is VERY stable, and the users love it (once they get used to the fact that it takes longer than MS Office to load). Disabling Java and tweaking the memory settings helps A LOT.
In the first days I gave a “migration talk”, I prepared some case uses and everybody noted that the way oo.org works is very similar to MS office. What I didn’t expect was a general concern: How to setup keyboard shorcut keys. I was in trouble, I didn’t know how to do that. So I was honest and I told them the truth, that I didn’t know how to do that but we can use the search feature in the help system and we immediatly found what we were looking. Props to the OpenOffice.org documentation project for this, it saved my day.
Real non-geek-final-users are using oo.org here. It is an excelent product.

But there is something that Microsoft Office has that OpenOffice.org doesn’t: Outlook. Final users love that buggy piece of crap. Sysadmins and helpdesk people hate it. I hate it. It’s buggy, sometimes it refuses to work and it has a lousy security record. I want it (and Internet “explorer”) outside my nework. But I just can’t say “don’t use outlook anymore”, I must come up with an alternative. At the time I was using Evolution to test the “linux-final-user experience” (I am a disciple of the mutt order). But evo has a serious problem: it doesn’t run on windows (at least not yet). So I had two choices, both from mozilla: Thunderbird and the Mozilla suite (codename seamonkey). I tested both in my debian workstation and allthough I really liked thunderbird, I fell in love (again) with the Mozilla suite. I loved the integration of his parts: a (mature) web browser, a (mature) mail/news client, an excelent html editor, and a calendar application. It is every sysadmin’s dream for his users. But I chose Thunderbird instead because:

• Mozilla is devoting all it’s efforts in the “birds-suite” : firefox, thunderbird, sunbird (firefox used to be a bird: phoenix/firebird). And now it is official, the mozilla suite won’t have a new release. The birds suite is the way to go.
• Outlook stores all it’s junk in .pst files. Both Thunderbird and Mozilla Mail can import .pst files, but only Thunderbird could import the attachments of the messages. We have a winner.

And of course, Thunderbird is the tool of choice if you have spam problems (and who doesn’t). It has a built-in bayesian filter that WORKS(tm).

On the server side, FreeBSD is taking over the server room. But this belongs to another post…

Today I’ll talk about this very important PostgreSQL server. It was running RedHat Linux 7.x on a dual P3, and a process that sometimes runs there usually took like 17 hours.

There was a spare Dual Xeon laying around, so I decided to replace that old server. Obviously my first choice for a sensitive database server was Debian GNU/Linux, so I installed Debian Sarge to test it: kernel 2.6, PostgreSQL 7.4.6 and scsi drives in software raid-0 (striping) in XFS. Everything was perfect, and the process that took 17 hours in the other box took only 5:30 hours in the new box. A big improvement.

But I am a curious person.

Since there was no hurry to replace that old box (other than the performance since besides the 17 hours process it serves a high-traffic website), I installed FreeBSD 5.3 in that same Dual Xeon.

It was hell.

First, I had some experience with FreeBSD since Aureal System’s server is running it. I have administrator access to the server but I had no experience:
- Installing FreeBSD (this was a piece of cake)
- Making software RAID work (this was HELL)

Now, I have read Michael Lucas’ Absolute BSD and Greg Lehey’s Complete FreeBSD. Lehey’s book has nice chapter about Vinum, FreeBSD’s Raid volume manager (Lehey is vinum’s author after all), but there is a problem with vinum, freebsd5 and SMP (I was installing it in a Dual Xeon remember?). I tried 5.2.1, same problems. Finally it worked with an obscure and undocumented version of vinum called gvinum. The g stands for geom-aware. It worked.

But this was just the beginning.

The beginning of the good news, for me at least. After installing everything and running the previously mentioned process, there was something very hard to believe. In the same box, with the exact same hardware configuration:

Debian+linux 2.6.9 + raid tools + XFS + PostgreSQL 7.4.6 = 5 hours 30 minutes
FreeBSD 5.3 + gvinum + UFS2+S + PostgreSQL 7.4.6 = 44 minutes

Clearly there was something very wrong here.

There wasn’t.

I ran the process again and again and the results were similar. I don’t know exactly if this extraordinary performance gain in this particular process is because of the filesystem or the raid handler, probably it’s because of both. I’ll have to perform some serious benchmarks here, and publish the results of course.

The end of this story is:
- The FreeBSD server is now in production, it runs like a charm and the web pages that access the database now run almost 6 times faster than with the old dual P3.
- I’m installing more FreeBSD boxes.
- I’m reading McKusick’s The Design and Implementation of the FreeBSD Operating System (thanks Breno) and Absolute OpenBSD.
- I am still running debian (unstable) on my desktop boxes, and probably this will not change.

Oh, and I’ll use that SCSI powered dual P3 for my benchmarks, to be published here in a near future.

I just can’t believe I’ve been avoiding BSD so many years.