I always try to do the Right Thing ™, specially when the thing in question is job related. That is why I can’t understand how some people can take their work so lightly.
I’ve had a very, very bad time because a postgresql database collapsed, the reason being that the SCSI drive where the data resides had 0 bytes free left. I must admit that I didn’t know the internals of the custom-made-tomcat+postgresql-based-system in that server, so my approach was rather cautious. On closer examination of the box and after reading the “documentation” of the system in mention and the shell scripts written by the same author (those systems were deployed long before my arrival), I could notice that:
- The database had no VACUUMs in 2 years (broken scripts)
- The “cleaning up” of old unused data didn’t work (broken DELETEs)
- The backup procedure was idiotic: backup everything to a file, then copy that file to tape
- Since the database was HUGE (70GB), even a compressed backup would be a lot larger than the 2GB ext3 file size limit for files (see previous point). In fact, I could see the previous backups stored in /backup, and all were of the same size (2147483647 bytes). All the backups were broken and useless.
- Worst of everything, the design of the tables in the database was completly childish. There was one HUGE table with everything stored in there. There was no normalization (there should be death penalty for people that do that in a production server). Because of this, the application is very SLOW.
I also noticed that the custom-system is very poorly written. For example, after a Java procedure to insert data to the database, the user must READ THE LOG FILES and search for Java exceptions to MANUALLY insert the missing data in another table. This guy didn’t even know how to catch exceptions. I’ve seen his code in C, PHP, Java and shell, and it amazes me how somebody can be a bad coder in so many languages. I’m not talking about style here, but doing the Wrong Thing all the time.
At least the system in mention looks pretty, and that’s what managers will look at anyway. That guy, as a programmer, is an excelent graphic designer.
Oh, and the server is up and running again, I rewrote the backup procedures and I’ll replace that crappy system as soon as I can.