As a practising project manager, I am always interested in finding new tools to help with key task. Project planning and integrating uncertainty of estimates is one such point of interest. One way to do this is through Monte Carlo simulation and over the years a few products have appeared with varying degrees of utility, ease of use and not to say price points.
These tools fall into three basic categories from the point of view of project managers.
- Stand alone tools like statistics packages or research applications. Generally not so easy to use for project planning.
- Excel plug-ins or add-ons which are basically visual basics applications that run inside Excel. Usually easy enough to install and run, but the round trip from e.g., MS-Project or whatever other application you use (e.g. Primavera) to Excel and back can be a bit tortuous.
- Plug-ins or add-ons for common project management applications like MS-Project or Primavera. These tools run inside the application and eliminate the exporting and importing necessary with Excel based tools.
Over the years I have used (although sparingly to say the least) a couple of different tools, but never any that worked directly inside MS-Project.
That is until now which is what this review is about: a new application entrant into the market called Full Monte.
What is Full Monte?
Full Monte is a plug-in to MS-Project that allows the user to perform Monte Carlo simulation of a project created in MS-Project (MSP).
Monte Carlo simulation is a way to simulate the effects of variation in project estimates for task durations and costs by calculating a large number of repetitions (i.e., samples). This effectively creates an experiment with a distribution of the calculated outcomes (the expected start and finish times for tasks, summary tasks and the overall project). From this we can make statements about how confident we are in e.g., any one particular finish date or range of dates.
Full Monte promises to help us do this efficiently so let’s see.
The vendor offers a free trial which is what I went with at this time.
Note: Full Monte is available only for Windows, not Apple OSX, which makes sense because MSP does not run under OSX either.
Objectives
The objective of the review was simply to assess for myself whether this is a product I could and would use for Monte Carlo simulation if it came to that.
In my review I looked at the following:
- What’s in the download?
- What is the documentation like?
- How easy is this to install?
- What does it take to learn the basics to actually do something?
- What exactly does the software do?
- What are the reports and reporting capabilities like?
- Where would I go from here, e.g., what else is there to learn about this product?
What’s in the download?
The download was simple enough to kick off from the website and downloaded without problems.
The download contains:
- The Full Monte application
- A Guided Tour PDF document
- A User Guide PDF document
- A demonstration project in MS-Project format (Full Monte Demonstration.mpp)
Documentation
The documentation is the Guided Tour PDF and the User Guide PDF files. Both documents are accessible from within MSP under the Full Monte menu (see below).
The Guided Tour is essentially a quick-start guide using the provided demonstration project as a play ground.
Installation
Installation was a snap; I just clicked through from the download. Hint: make a note of where you install this (if not blindingly obvious) because you need to find the folder to start up the demonstration project later on.
Note: I installed this with MS-Project 2007 so my examples will look different from MS-Project 2010 which is used in all the Full Monte documentation. Other than that, the functionality is the same.
Learning the basics
Whenever I install something new I expect to spend a little time up front to learn the basics, that is, enough ‘stuff’ so I can start using the application for something useful. Full Monte helps out by giving us a Guided Tour (GuidedTour.pdf) which I did first.
First step is simply to start up MSP. Then go find the demonstration project (the file is called Full Monte Demonstration.mpp).
Next open the guided tour which is accessible from the Full Monte menu:
After that, just read the document and repeat the exercises as you see fit.
The tour uses simple and clear language and provides a lot of illustrations (screen shots). The content covers the following:
- Start-up (find the Full Monte menu, load the demonstration project)
- Data entry (quick pass through the data entry options to set up a simulation)
- Risk and sensitivity analysis (a simulation run)
- Reporting (a look at the output from the simulation)
- Customizing graphical content (quick introduction to how you can make changes to standard reports)
Couldn’t be simpler and this is all you need to take a simple project of your own and start playing around with a simulation.
Now, all this gets you to the point where you know where the basics are and how to work use this. The assumption is that you have an idea about what Monte Carlo simulation is about and what the different options available to you are for.
For example, you have to select a distribution and make estimates about task durations. The User Guide (accessible from the Full Monte menu) discusses the distributions and helps out a little. However, depending upon your knowledge of Monte Carlo simulations, you may have to go to other sources to beef up your understanding so you can make the right choices.
The software makes it easy to set up the problem and execute the experiment, but it is up to the user to ensure the experiment makes sense and the output has any relevance to the real world.
For the record, Full Monte provides the following distributions:
- None (default, deterministic)
- Beta (i.e., a variant of the family)
- Lognormal
- Normal
- Triangular
- Uniform
The Guided Tour was simple to follow and the software behaved pretty much like the examples in the document.
One small quibble about branching:
In the guided tour, five tours have task 13 as a predecessor and show up in the branching window.
On my machine (i.e., with my version of the demo project), I only see two choices even though the predecessor relationships appear identical.
I am not sure why this is, perhaps the demo project in the download is slightly different somewhere, somehow. (For the record, I did reset the demo project form the Full Monte menu.)
Core functionality
Taking a guided tour is a little like a ‘monkey see, monkey do’ exercise. It’s not always clear what is going on and what it means, we’re just clicking along.
So after the tour, what will we know? Well, we will know how to set up and execute a simulation for a particular project and at least run some standard reports to analyse the outcome.
The trick is setting up the simulation because this is where the tough decisions are made. Running the simulation itself is trivial after that. Then comes looking at the output and understanding what all the numbers and graphs mean. Partly this is helped along by the basic report design, but the user’s brain definitely has to be engaged.
Setting up a simulation
I like Full Monte because setting up the simulation is handled in a very compact way. First you select either Edit or Global Edit form the full Monte menu.
Edit is for editing individual tasks while Global Edit is simply the ability to make settings for more than one task at a time, essentially bulk data entry if you wish.
I’ll talk about the Edit option in the following.
Setting up a simulation involves addressing the options in the Edit window.
As you can see from the graphic, you select a task in the left scrolling window which is simply all the tasks identified in your project (i.e., in MSP).
Then, for each task you decide on the distribution, add duration estimates, decide on a correlation group and % and finally decide on any branching.
Let’s go through the basic choices.
Distributions
Monte Carlo simulation is useful because it can calculate the effects of variation (read: uncertainty) in the estimates. Full Monte doesn’t know what your uncertainty is, so you have to tell the software.
Your choices are None (default), Beta, LogNormal, Normal, Triangular and Uniform. Assuming you know what you’re doing, you pick the one that makes most sense for the respective tasks.
Durations
The durations are the estimates of the time you think a task could take. It is quite common to think about a three point estimate like the optimistic, most likely and pessimistic durations.
These estimates imply a distribution and some degree of variation. You will see that for technical reasons, some of the distributions let you set all three estimates, while some only let you specifically set two in the Edit window.
The Most Likely estimate that shows in the Edit window is simply the duration entered for the task in your MSP project plan. (Obviously, this means that if you change the duration in the plan, the Edit screen will also show a change.)
The software automatically populates the optimistic and pessimistic values based on defaults. Consider this just a starting point, because the default values may have no relevance to your project’s inherent level of variation.
The bottom line is that you have to think through your project and provide the estimates as best you can (and perhaps get some input from others as well).
Note: If your plan has buffers or ‘safety’ or slack built into the task durations, you should strip all that out and only enter durations as net expected work time. Otherwise, you build buffers on top of buffers and your schedule will be blown out in time much more than it needs to be. The whole point of the simulation is to establish sensible buffers in the first place.
For example, if you have said to yourself, well, I have this two day task that I need to get done and if I get it done in this particular week, I’m OK, so let me just put that in as a 5 day duration. This is not as crazy as it seems, because, like it or not, many projects use shared resources. This means that we are often also prevented from simply picking two days in a week to get a two day task done. Thus we end up with a duration saying that, in between other things, this resource will do this two day task in this five day window. You can’t simulate with this kind of estimate, you have to restate it as a two day task and instead cover your risk through your estimate of variation.
Besides entering durations as specific time estimates in hours, days, weeks etc., you can also enter them as percentages from the baseline duration in your MSP plan. This can be handy if you have an idea about the range of your values or if you simply want to model things like Most Likely plus or minus 10% (symmetrical) or, say, plus 30% and minus 10% (asymmetrical).
Technically, getting all this into the software is trivial; coming up with quality data, not so much.
This is about as complicated as it needs to get for a lot of projects.
Correlation
Correlation is a more advanced function you can use to link risk to other tasks or, indeed, to external forces (assuming you have a way to monitor and estimate this). The example used in the Guided Tour is weather. It is easy to see that if your are building a house, the weather could affect your project plan.
Technically, the way you do it is define what Full Monte calls a Correlation Group (e.g. weather) and then link tasks to this correlation group. Then you set a correlation value and you can now theoretically model the effect of weather changes on your plan.
This is easy to say, but takes a lot more work to actually set up, manage and interpret in a meaningful way, so consider this an advanced topic. Just know that there is functionality to deal with scenarios like this if you ever need them.
Branching
Similarly, branching is an advanced topic. The simple version is that branching allows you to specify what Full Monte should do once it gets past a task that can in the plan branch off into more than one continuing path.
As with correlations, we are now into complex network behaviour which takes a fair amount of technical know how to get right, so for anyone but an advanced users, this is not really applicable to most projects.
That said, there are two technical options: conditional and probabilistic. Conditional simply means you have the ability to specify dates as the branching logic, as in “if the task finishes before this date, go on to task A, but if it finishes after this date, do task B next.”
Probabilistic means you set a probability for each successor task.
Again, how and why you would use this is another story and beyond the scope of this review. The functionality is there for those who know how to use it.
Running a simulation
At some point, you will have entered all the estimates in the tool and now you’ll want to see the what happens. The simulation is kicked off by selecting Analysis from the Full Monte menu.
This presents you with the following screen of options.
You have four basic options to start: Number of simulations to run and then to Perform Risk Analysis, Perform Sensitivity Analysis or both.
Risk Analysis means to simulate start and finish dates plus costs for tasks and the overall project.
Sensitivity Analysis means to estimate how critical the respective tasks are and how strongly they affect the overall timeline.
For Risk Analysis you have a further three options: use fixed seed, display project finish histogram and display total cost histogram.
Fixed seed means you have ability to replicate simulations with a similar distribution. For some types of comparisons, this can be very helpful, for example to isolate the effects of certain decisions. If you do not check this, each simulation run will generate its own distribution. With large number of repetitions, the differences between runs become smaller, but won’t be identical. Consider Fixed Seed a special case.
Project Finish Histogram is a report showing you the confidence levels for the expected range of finish times.
This chart pops up automatically upon completion of the simulation. (The example used the Normal distribution set through Global Edit).
The chart simply tells you that, for example, you have 0% confident that you can reach the theoretical early finish date of 2/3/2013, but can have 90% confidence that you can finish before 5/8/2013. The red line is the cumulative confidence level for the resulting distribution of finish times.
Clicking on this chart flips you over to a similarly structured cost chart.
Again, you can see the confidence levels for realizing various final cost amounts.
Obviously, both of these reports are crucially dependent upon the quality of the data and the assumptions made about the distributions and their variance.
The sensitivity analysis also produces a report, the so called tornado graph, but you have to navigate to it: Full Monte menu/Report/File/Open Report Specification/Tornado.
The sensitivity analysis calculates a sensitivity index for each task which indicates its relative importance to the overall timeline. The tornado graph simply shows bars for each task in descending order of importance. In the example below (using the demo project), it is clear that task 22 is the one to worry about, no so much the rest.
Reports
Above we looked at a couple of the standard reports. There are a few others delivered with the software. The standard reports are available from the menu path:
Full Monte menu/Report/File/Open Report Specification/
The options drill down to tabular presentations of the underlying data. Some of the reports allow you to click a button to produce a graph for, e.g., a task. In this way you can see a distribution of finish times for individual tasks which can be useful to validate your assumptions about the distributions and ranges of outcomes.
For example, clicking on the Graph button for Dig Foundation/Early Start Histogram gives you this:
The User Guide explains how to customize existing reports or make new ones. The procedure is relatively straight forward, but as always, this requires you to become familiar with the underlying data model and where Full Monte stores the output from its many calculations.
This raises another issue common to plug-ins. Plug-in manufacturers do not necessarily talk to each other some if you have other plug-in running in MSP, then you may experience a data conflict where the plug-ins try to access the same field.
The solution is to change the mapping and Full Monte helps you out with that. One particular use for this is if you migrate from another Monte Carlo tool and already have simulation data in your project plan. Again, the User Guide covers this.
Where to go from here
The Full Monte software is easy to use and runs plenty fast on a normal computer. I ran all my simulations at 10000 repetitions which only tool a few seconds with the demo project. Obviously, the size and complexity of the project determines the run time, but that is nothing new.
My personal experience showed that the documentation is basic, but adequate and it is easy and quick to get up to speed on full Monte with the basic software solution.
Ultimately, the user has to become educated about Monte Carlo analysis in the first place before the tool is safe to apply for real live project planning where important money hangs in the balance.
This review has not explored the use of Full Monte for larger, more complex projects.
Related posts:
Tagged with: Full Monte • monte carlo simulation • project planning
Filed under: Monte Carlo Similation
Like this post? Subscribe to my RSS feed and get loads more!