Managing information in your personal and professional life

While all of us suffer from an information overload, how we deal with that information is often what separates us on a professional level. This blog entry is a review of my experiences managing this information, leading up to Evernote, a product that works well in practice.

Here at work we are slowly implementing a records management policy, and a probable outcome is that we will be unable to keep email older than one year. As an IT professional this has implications, because my email is full of useful information that I want to keep. After discussing this at length with our records manager, I came to the conclusion that I really need some sort of "personal information manager". In about March 2009 I discovered Evernote, an online service.

But first a little history, and for that we can go back a long way to software products like Lotus Agenda and others whose names I have forgotten . One of them lost all my personal information when the computer crashed, and I never did find out who had borrowed some things of mine. But none of those products really worked well enough to "stick".

The internet has become a huge source of this information, and managing browser bookmarks has always been a problem for me. From about 2000 I used a Lotus Notes database to track my bookmarks and other information, and it worked very well. One really nice feature was the ability to add notes to each bookmark to explain why you bookmarked it. Another useful feature was replicating the database between systems so that I really had just one bookmark list to manage, even on different systems. However, there were several problems with the Notes database; possibly the biggest was that getting anything in the database involved a multi step cut and past operation. This is where Evernote really shines because (in Firefox) you can use a button that automatically puts the bookmark in your database with just one click. If you want the content of the page, there is just one box to check and you have saved it for future searching. Another really nice feature of Evernote is that I don't have to maintain any code, nor even a server. In fact I can get to my Evernote database from any browser. Evernote also features a stand alone client so in the event of them disappearing at least I can still have my information.

Evernot really shines when it comes to adding keyword tags to notes. Just like Taglocity, keyword tags actually work much better than folders because you can search on multiple keywords. Its difficult to search a Notes database and find the documents in multiple folders, but Evernote easily lets you find all the notes with a particular keyword tag. It also maintains a full text index so you can search for multiple keyword tags and even inside attached files. Another nice feature is that you can mail into your Evernote account, so if you get that technical information in your work email that you want to keep, you just forward it to your Evernote account. That gets around the limitation of having your work email deleted after a year.

All of this is not to say that it is perfect. For example, the rich text is limited - you can't paste in pictures. This is a strange limitation because you can import pdf files, and you seem to get everything - probably the case of the product not being fully developed yet. Another limitation is printing, where even though you set the font to one size on the note, when you print it out, the font is much bigger. There are feature missing: the ability to warn me when adding a duplicate bookmark. and a tool to find duplicate bookmarks. But these are minor limitations that should be resolved as the product matures.

I have been using Evernote for about 9 months now and currently have over 1400 items in my database. I really love how easy it is to clip web pages and add notes, and the fact that I can seamlessly integrate it into my personal and profession information management needs. Evernote allows all that technical information to follow be wherever I go. The searching works well, and you can add keyword tags to all Notes. There is a free version of Evernote, and a paid version at $5 per month. The paid version allows you to add any files to a note, and allows you to search inside those files. In all, Evernote has proved to be a very practical way to manage all that person and professional information that I collect very well - highly recommended.

DocMaster - Single Database Document Manager

When I started working with Notes in 1994, one of the first things I developed was a "Help" database. Over the years this has travelled with me and been used at almost every company I have worked at in one form or another. With input from dozens of people the original Help database has grown into a single database document manager called "DocMaster". A number of people have asked me to post the code, so here it is.

Bear in mind that this is NOT a commercial product; it is unpolished working code with some documentation. The database is both the help file, and the template. Email me with any questions.

DocMaster

Document management systems range from managing papers in a folder up to managing libraries of documents. There are many high end document management systems on the market, for example Domino.Doc or Documentum. If you think of these systems as a library, you can think of DocMaster as a filing cabinet. While there are many libraries in the world, there are many more filing cabinets. And if document collections grow to big for filing cabinets, they can be moved into the larger systems.

DocMaster is a "Single Database Document Manager" that is a place to manage collections of loosely related documents. Although DocMaster is structured like a traditional book, you don't read it as you would read a novel. Rather you read it as you would read a technical manual - typically you find the content you want and then do something with it, e.g. make a decision.

Typical Uses for DocMaster

• Corporate policies, procedures and standards.
• User manuals.
• Help documentation for Notes databases.
• Design notes for in-house software.
• All information on a particular project, e.g. a software upgrade. The next time that software is upgraded, all the notes etc. are in one place.
• All information on a particular piece of equipment, including user manuals, repair notes etc.
• A shared collection of links, pdf docs etc. on a particular subject.
• "Living" documents that often change.
• Meeting agendas and minutes.

DocMaster is easy for end users

• It uses a "Book" metaphor with a Table of Contents, an index, glossary of terms and a FAQ.
• It can use the same 3 pane interface as email, and uses folders as email does.
• It supports full text searching of all content.

DocMaster is easy to work with

• It is simple to set up, and easy to customize and includes tools to customize it for common uses.
• There is security, with document owners, editors, authors and readers.
• Work is tracked as it progresses, and you can easily see what remains.
• Audit trails. Tracks who did what, when and why. You can view or recover earlier  versions of documents.
• Highly customizable sorting: by date (up or down), by title or by doc number.
• Templates to control title format, date, author or many other fields. Templates can include content. Folders can be linked to templates so documents are created with standard settings (e.g. sorting).
• Document counter show total number of documents per folder, person, department etc.
• Tracks the size of documents.
• Any folder can have its own help documents, or one help document can be shared amongst several folders. The folder help document describes the purpose of the folder.

Advanced features

• Use Template documents to create documents with the same initial content.
• Add your own document types.
• Use "shared text" to include the same text in multiple places. Update the shared text once, and it is updated everywhere.

IT Projects and Documentation

Hands up all you developers out there who document your in-house IT projects properly. I thought so! Almost nobody does. Good documentation makes software projects easier to use and maintain, but is seldom written well, and sometimes not written at all. Documentation is usually left to the end of the project when interest has wained, and not seen as adding much value. It is something developers would rather not do, something that is put off as long as possible, hoping the need for it  will fade away. Unfortunately all that fades is the memory of the project, making the code yet harder to maintain.

An ounce of prevention is worth a pound of cure. Let me illustrate that with a personal experience from last winter in San Diego, California, when we had the 3rd wettest winter on record. For some time I had suspected my roof might be developing problems, and early in the previous summer I had hired a roofing company to check out and fix the roof. During this process my neighbor (who has the same house design) asked the price. Being neighborly I told him, and got a look that said... "You haven't got leaks - you must be nuts!". In due course my neighbor was transferred to Japan for a 3 year tour of duty, and he let his house to tenants. Came the winter rains and water started pouring though his roof, damaging dry wall inside. Of course tenants don't get problems fixed as fast as home owners, which made it even more expensive. I'm guessing, but he probably paid at least 5 times what I spent for his repairs. Had he been proactive in fixing his roof, his costs would have been much lower.

Likewise, good documentation significantly reduces the cost of using and maintaining software down the line. Timing is everything. The sooner documentation is written, the less effort it takes (because it is fresh in peoples' minds) and the better the result. But the real trick for producing good documentation with minimum effort is to turn it into a collaboration project between developers and users.

Collaboration to the rescue

If we in the IT world are so interested in electronic collaboration, why don't we harness it to help us solve the documentation problem? The secret to improving documentation is to efficiently harness the thoughts and input of all involved in the project when they have thoughts and input. The labor of writing the documentation is not so much the actual writing of the text, as knowing what to write. Timing is everything, for example developers should document the purpose of a field when that field is created. And if end users have a problem with that documentation, they must be able to add comments, ask questions while they are working with the software. A good example of a collaborative documentation project is Wikipedia, the on-line encyclopedia. You could consider collaboratively maintained documentation a moderated wiki project.

Essentially the idea is to turn every page into a collaborative effort. To implement this, end users and developers must be able to add comments to a page while they are reading it, because that is when they see the problems. Typically users will find that...

• Documentation is ambiguous.  Other people don't make the assumptions the author made, and immediately see ambiguity problems that the author will never notice. And as more users from different cultures mix, the potential of ambiguous writing increases.
• Documentation is wrong - it is out of date, or there is a bug in the code - and it does not behave as expected.
• Documentation is incomplete, either partly or totally missing.

Every page of documentation has an owner. When users add comments to that page, the owners are notified by email. After reading the comment the owner then initiates an action, e.g. posting a potential bug to development, correcting the documentation etc. Once the problem is taken care of, the comment discussion is archived and the updated page is ready for use again. In this way documentation becomes almost "self maintaining". You can see simple examples of this on the web at the bottom of technical pages, e.g. on the Lotus Notes Knowledgebase they ask several questions and solicit you input. While this is a step in the right direction, it usually is a very weak implementation of collaboration because there is no ways for the doc owner to contact the person who made the comments.

One further collaborative though it to use IM (Instant Messaging) in the documentation. This is an intriguing idea that I have yet to try in a meaningful way. However, if you use the Lotus Notes client and Sametime together it is trivial to add.

What is software documentation?

Let's begin by defining what we mean by software project documentation: It is a collection of documents related to a software project, aimed at a target audience for specific purpose. Any software project has several kinds of documents associated with it. At a high level these are...

Document Type	Target Audience	Purpose / Description
Requirements	Software architects, end users.	Defines what the end users want, and is the"target" the development team aims for.
Software Design	Software developers	Describes how the requirements will be metwith the developed software, and how that software will be built.
The code itself	Software developers	(special case)
Test cases	Software developers	Describes how to verify that the user requirementshave been met.
User instructions and help	End users	Describes how to use the software.
Development Standards (usually not projectspecific)	Software Developers	Describes conventions adopted by the organization,and the reasons for those conventions.

Most important is what documentation is not: Documentation is not a story. Many people approach documentation as though they were writing an essay at school. This bad approach is positively encouraged by the page layout of word processors, e.g. MS Word. It is important to realize that documentation is NOT a story, but is a collection of many documents, all related to the same subject.

When should you write documentation?

One mistake that is often made is to leave the documentation to the end of the project, which invites problems. Documentation is a continuous process that starts before the official start of the project, and continues after the official end of the project. The tail end of the documentation can be considered part of the project maintenance.

The raw materials for documentation are notes, questions, ideas etc. To get the highest quality documentation content it is very important to capture that raw material while it is fresh in people's minds, so make it easy for them. Some examples are:

• Suppose Mike is a software developer working on the project, and adds a field to a form. He should immediately add that field to the help documentation and explain the high level purpose of that field, for example why that field was required. (No need to add where it is used etc., as an up to date version of that is best extracted from the code). He can do it in point form, and flesh it out later. But capturing that information early is vital, because it is fresh in his mind. Even leaving it to the next day will cause him to occasionally forget important details and subtleties.
• Suppose Jack is using the software and finds a problem. He adds a comment to the manual, which emails Mary, the owner of that page. Having a free moment, Mary reads Jack's comment, contacts Jack and has a short discussion about the problem. Since the problem is fresh in Jack's mind, the two of them can effectively resolve the problem quickly. As an added bonus, Jack feels that the software development team is really "on the ball", and they are all one team working together for the good of their company. Of course, had Mary taken a month to get back to Jack he would only remember the frustration with the software, and would have forgotten all the details. Not only is it more difficult to resolve the problem, but the "on the ball" team building effect has been lost.

At the project start take a very high level approach to documentation by developing an outline. Realize that the outline is very fluid, and will change substantially as the project proceeds and the documentation is fleshed out and refined. An initial outline will typically cover requirements, design and architecture. A useful technique is to flesh out the end user manual before the code is written. Of course as the product is developed, the user manual is updated. This forces a number of design decisions to be considered early, and results in a higher quality end product.

When writing the documentation always capture the reason why a decision was made. You are capturing the context of a decision, which makes it easier to maintain. Later on people forget the context of that decision, and then have difficulty deciding to change it. For example, I was recently trying to fix a problem with some code I had written a few years back. I had not recorded the reason why it was written using such a peculiar way. Having forgotten the context, I rewrote it in a more convectional way to fix a problem and add an enhancement. When testing the modified code I discovered why the other peculiar approach was used, and had to rewrite it the original way (due to the nature of the change, I could not simply roll back to an earlier version). Of course, the code now includes the reason why this peculiar approach was used so I won't make that mistake again!

Managing documentation

As mentioned above, the problem with using word processor for documentation is that it forces you into "story" or "book" mode, whereas project documentation is a large collection of relatively short, related documents. If you don't use MS Word for documentation, what do you use? I have found a Notes database to be a very good tool. Potentially there are other ways to do this, but because Notes works so well, I have never needed to explore any further.

Once you realize that project documentation is a collection of many related documents that are collaboratively maintained, project documentation becomes a mini document management project. This naturally leads to the document lifecycle: create, publish, collaborate and update, retire and archive, something that is well handled by standard workflow techniques in a database. Managing project documentation as a collection of documents in a Notes database makes the whole task much easier.

All documents should have an audit trail or revision history so that you can go back to old versions. This audit trail should be entirely automatic, requiring nothing from the user except possibly the reason why the document is being updated. Disk space is cheap - you can keep almost any number of old version for as long as is required. Again this is relatively easily implemented in a Notes database.

One of the biggest mistakes we make is to think that documentation is complete. It very rarely is. Since the documentation is a database, updating one page does not require re-issuing the entire package. Actually, if the documentation is really "alive", it will be being updated continuously. Since documentation is a collaborative "living" effort, it must be on-line, and can't be on paper. While it can be printed out, paper is obsolete the moment it is printed (but realize that it can be hard to get people off paper). One very useful concept I have employed is to track the "percentage done" of each page. You create a few set "percentage done levels", e.g.

• 0% - Title done (placeholder, used to build the outline)
• 25% - Started work
• 50% - About half way
• 75% - Nearly complete
• 100% - Done

You assign one of these levels to each document. To start, you create the outline by creating pages for each logical heading in the documentation. Don't worry if you create too many or too few - the pages changes as the documentation is worked on. At the document level this is a very crude indicator of the work done, but at the chapter and database levels (i.e. in a categorized Notes view) it provides a very good indication of how much work still needs to be done. It also shows where effort must be focused to complete the work.

Where do you document?

As mentioned earlier, I like to use Notes databases for project documentation, which allows me to keep most related information on the project in one database. Typically requirements, design, and user and code documentation exist in that database, and this forms the bulk of the documentation. Documentation with a specialized workflow such as bugs, enhancement requests, ownership etc. is better managed in a project management database. (See also Thoughts on Managing In-House Software Development Projects in this blog.) And of course the best place to document the code is in the code itself. If you have a tool to generate the documentation from the code even better. Best of all is to use such a tool to create the documentation on demand (because the documentation is always up to date), but that isn't always practical.

User Documentation for Notes Projects

This is a section specific to Notes, and Notes application development. Once developers decide to use Notes for on-line user documentation (i.e. help) they must decide where to put it. The worst place is to put the help in documents in the database. Reason? - at least 2 versions of the help must be maintained - one in the production database, one in the development database. And if the database design is ever cloned, the problem compounds. As we know, it is impossible to manually keep nontrivial collections of documents in sync.

This leaves two choices - either hard code your help into the database design, or put it in a separate database. Both of these allow you to maintain only one version of help. For small, relatively simple database,adding the help as design elements is a reasonable approach. Typically if I was create a template that formed part of a database design, e.g. config documents - I might put the help docs in the design. But in recent years I have all but moved away from this approach.

For anything substantial (i.e. more that 2 or 3 days work) it makes a lot of sense to put the help into a stand alone documentation database. This database should be a standard template developed for the purpose and reused across multiple projects. As the design elements are created, the programmer can add notes to the help documentation database. For example, if the developer creates a field on a form, he can jot down a few notes on why that field was created as he creates it. As the project develops, so this documentation gets fleshed out. Finally, documentation can be upgraded as required, e.g. if a users has a problem, add the answer to the documentation and send a link. Thus the documentation is continuously improved.

Conclusion

To sum up, the main points to consider are:

• Documentation is a continuous process that starts before a project formally starts, and continues well after the formal end of the project.
• Timing is critical: capture documentation content when raw content is fresh in people's minds.
• Project documentation is not a story; it is a collection of related documents that must be managed.
• Treat documentation as a collaborative effort, much like a wiki to reduce costs and improve quality.

I trust this gives you some ideas on how to effectively create and manage project documentation.

Using code headers to make LotusScript easier to maintain

This article is an example of a simple technique that, when put into practice, can make LotusScript much easier to maintain. Code headers are text comments at the start of procedures that describe what the procedure does, what is passed in, what it returns, any special notes and a brief summary of changes. When you open a function or sub with a code header you immediately know what the code does because it is clearly explained at the top. No more hours of studying the code to figure out how it all fits together.

---

Function RemoveFolderFields(doc As NotesDocument) as Variant
%REM
=========================================================
PURPOSE
This function removes the hidden $FolderRef , $FolderRefId, $FolderRefFlags fields from the doc

PASSED
doc - The current document we are updating

RETURNS
True - We updated the DocFoldersTX field
False - There was no change to the DocFoldersTX field

SPECIAL NOTES
Checks the exisitence of folders that are in $FolderRef field and updates that field with folders that are present in the database.

HISTORY
25 Jun 03        Prasad KVS -         Initial version
12 May 04        Chris Doig -          Removed references to OldDocFoldersTX. Also deleted commented out old code.
=========================================================
%END REM
Your code goes here...

---

Purpose
The purpose paragraph explains why the procedure exists. Well written purpose paragraphs save hours when debugging complex programs.

Passed
On occasion I have opened the code starting with pages of global declarations. Such code is easy to write, but difficult to maintain. When debugging such code you have little idea what is changing those global variables without extensive study of the code. My preference is to avoid passing via global variables, to pass everything though the calling line, and fully describe each item passed. (There is normally no need to describe the variable type, because that is defined in the first line of the procedure. Don't duplicate information - you can never manually keep it up to date.) The passed paragraph describes what is passed into the procedure. This can be another time saver, especially variable names are reused, e.g. exactly which document does doc refer to?

Returns
The RETURNS paragraph explicitly states what the function should return. For example, if the returns paragraph stated the function should return true or false, and something else was returned, you would have a good idea of where to start looking for the bug.

Special Notes
The special notes paragraph describes anything unusual about the code, e.g. why a particular approach was taken. It can also be used for programming notes, e.g."if the folder name is missing, then an error occurs that is not caught by the error handler."

History
The history paragraph is simply a list of dates, programmer names and change summaries. This is particularly useful when projects mover around between several programmers - one can always ask a previous programmer why something was done that way.

An IT department could define a corporate standard that requires all LotusScript code to include code headers, and these headers be maintained. Such a standard helps you climb the CMMI scale. (Of course you have to enforce the standard, or it is worse than useless!) Developers can use something like Teamstudio Snippets (or even a home grown tool) to add this code header to the start of every procedure in LotusScript. You can also configure automated testing tools (e.g. EasyDesignScanner) to check that the code header is at the start of all procedures. Once you have these code headers in place you will have take a significant step towards making your code more maintainable.