After many years as a professional sysadmin, my best tip for those just starting out is: take notes. Our job is basically about solving hard problems. (If they were easy, users would have solved them already.) This being so, keeping a detailed notebook of everything you do is like having an external brain pack:

You don’t have to remember or rediscover how you solved that bizarre DNS problem back at your last gig; it’s in the book

You don’t have to reread the man page for half an hour to remind yourself how to self-sign an SSL certificate; it’s in the book

And you don’t have to waste time wondering where you got to on the current project at the end of last week; it’s in the book

How to keep a notebook

The sysadmin notebook is one place technology won’t help you. A simple text file is best; you can check it into version control, and access it everywhere. It’s searchable, and if you organise your entries by date, you can find stuff that way as well.

I don’t recommend using a paper notebook, as one of the most useful functions of my notebook is to copy and paste information into it directly from the console session, or from web pages, and then to copy that information out again to save retyping. If you need your notebook with you when you’re on the move, try keeping the text file in your Dropbox and accessing it from your smartphone that way. Alternatively, if you keep it as a Simplenote document, you can sync it with Simplenote on your mobile device.

What to put in it

Working scientists keep a lab notebook in which they write down everything they did, and what the results were. Sysadmin work is also about careful, scientific, detailed experimentation, especially when troubleshooting. When there are many combinations of things to try, you need to document which ones you’ve already tried, and what happened. You can copy and paste console output directly into your notebook for later reference.

One handy tip is to record your console sessions with the script command, for later reference. For example:

$ script session.txt
Script started, output file is session.txt
bash-3.2$ echo Hello, world
Hello, world
bash-3.2$ exit
^F
Script done, output file is session.txt

$ cat session.txt
Script started on Mon Sep 27 17:30:40 2010
bash-3.2$ echo Hello, world
Hello, world
bash-3.2$ exit
^F
Script done on Mon Sep 27 17:30:52 2010

Once the job is completed, you can use the session file as a reference for writing up your notes, including copying and pasting commands and output where necessary.

A record of what you did

If you’re a freelancer, it’s important to be able to justify the time you’ve spent on a client project, and to be able to account for what you did and why. If you’re on salary, it’s even more important to be able to communicate regularly and accurately with the rest of the organisation, explaining what you do, and letting management know how valuable you are. The notebook is ideal for ‘weekly achievements’ meetings and emails, and a goldmine when it comes to annual review time.

A valuable reference

The nature of our work tends to involve complicated, lengthy processes requiring accurate command-line input and configuration settings, which take time to get right. This is where the notebook really scores: you need never rely on memory again. Having gone through the process of trial and error to get a system working, you can go back through your notes and distil them into a ‘how to do it’ document which will help to share knowledge with your co-workers, and form part of your essential procedure documentation.

A project management tool

You can also use it to store ‘things to do’ tasks and other reminders (keeping those at the bottom ensures that you see them every time you write a new entry). When you meet with clients or management to discuss project status, your notebook is an essential reference for what you’ve done, problems and roadblocks you’ve encountered, questions that need answering, and tasks that still remain to be done.

A communication tool

Wherever two or more sysadmins are gathered, confusion and problems can result from them all working on the same system - either at the same time, from remote locations, or consecutively, in a shift system. A common document where everybody details what they did and when, can be very useful. When something goes wrong it’s essential to be able to trace through all the changes that were made recently, and find out who or what broke the system.

Anyone who works in a team (and we all do, whether we realise it or not) needs a way to share information with their colleagues about what’s happening, and a team notebook is an excellent way to do this.

Examples

Here’s a couple of illustrative entries from my own notebook:

# 2009-10-09 Friday

Got Cribbage Corner site migrated from HostMonster and ran upgrade
script OK, but now having some trouble delegating the DNS from HM to
DynDNS.

Need to accept mail on peterman for help@cribbagecorner.com so trying
out sendmail virtusertable as detailed here:

https://www.rushworth.us/lisa/?p=1526

This works fine so email to CC now works - everything is ready to
delegate the domain, but HostMonster doesn't accept the DynDNS
nameservers. Possibly it just hasn't been loaded yet on DynDNS so will
try again later.

Now fixed - the right sequence was:

Set up the domain on DynDNS and 'pre-activate' it ('Force
activation')

Wait for DynDNS to load the records:

$ host cribbagecorner.com 204.13.249.76
Using domain server:
Name: 204.13.249.76
Address: 204.13.249.76#53
Aliases:

cribbagecorner.com has address 94.76.216.167
cribbagecorner.com mail is handled by 10
peterman.bitfieldconsulting.com.

Then go to HostMonster and enter the nameservers:

ns1.mydyndns.org ns2.mydyndns.org

It didn't accept them as IP addresses which is why it kept saying
'Invalid nameserver'. Dumb or what?

I found when the DNS was delegated over, accessing the main page as a
normal user gave 'Access denied' - though subpages worked OK. Found
this page:

http://drupal.org/node/492734

which suggests going to Site Configuration / Performance and clicking
'Clear cached data'. This worked!

# 2010-09-08 Wednesday

Working on PHP build. Downloaded PHP-FPM:

[link]
Generated patch:

php-fpm-0.6-5.3.0/generate-fpm-patch

Downloaded PHP:

[link]

Patched:

patch -p1 < ../fpm.patch

Prepared for Debian build:

dh_make -e sysadmin@example.com -f ../php-5.3.0.tar.gz

Edited control file and added custom configure line to debian/rules:

override_dh_auto_configure:
./buildconf --force
./configure --with-fpm \
--with-libevent=shared,/usr/local/lib/libevent.so \
--with-mcrypt \
--with-zlib \
--enable-mbstring \
--with-openssl \
--with-mysql=/usr/bin/mysql_config \
--with-mysqli=mysqlnd \
--with-pdo-mysql=mysqlnd \
--with-mysql-sock \
--without-sqlite \
--with-curl

per

http://www.debian.org/doc/maint-guide/ch-dreq.en.html

Then:

dpkg-buildpackage -b -uc -rfakeroot

In general, whenever I have to expend brainpower to solve a problem, I’ll consider it worth recording the solution (and the problem) in the notebook. Similarly, if there’s a command line that took some trial and error to construct, it’s worth a couple of seconds to copy and paste it into the notebook.

Start today

It doesn’t matter if you haven’t been keeping notes up to now and think it’s too late to start. It’s not. Just open a new text file, put the date, and type in what you did today.