Using real project schedule data to compare earned schedule and earned duration management project time forecasting capabilities
Paulo André de Andradea, Annelies Martensb, Mario Vanhouckeb,c,d,
a Techisa do Brasil, Brazil
b Faculty of Economics and Business Administration, Ghent University, Tweekerkenstraat 2, Gent 9000, Belgium c Technology and Operations Management, Vlerick Business School, Reep 1, Gent 9000, Belgium d UCL School of Management, University College London, 1 Canada Square, London E14 5AA, United Kingdom
A R T I C L E I N F O A B S T R A C T
Keywords:
Project management
Earned Duration Management
Earned schedule
Time forecasting
Empirical database Project regularity
Project control system Since project control involves taking decisions that affect the future, the ability to accurately forecast the final duration and cost of projects is of major importance. In this paper, we focus on improving the accuracy of project duration forecasting by introducing a forecasting approach for Earned Value Management (EVM) and Earned Duration Management (EDM) that combines the schedule performance and schedule adherence of the project in progress. As the schedule adherence has not yet been defined formally for EDM, we extend the EVM-based measure of schedule adherence, the p-factor, to EDM and refer to this measure as the c-factor. Moreover, we aim to improve the ability to indicate the expected forecasting accuracy for a project by extending the EVM concept of project regularity to EDM. The introduced forecasting approach and the EDM project regularity indicator are applied to a large number of real-life projects, mainly situated in the construction sector. The conducted empirical experiment shows that the project duration forecasting accuracy can be increased by focusing on both the schedule performance and schedule adherence. Further, this study shows that the EDM project regularity indicator is indeed a more reliable indicator of forecasting accuracy.
1. Introduction
Uncertainty and risk often cause delays and budget overruns during project execution. Therefore, project forecasting is an important aspect of the project control phase to accurately predict the final project duration and cost. Project control entails that the progress of projects is monitored during project execution to predict the final project outcome and to take corrective actions when necessary.
A well-known technique to monitor the progress of projects and to forecast project cost and duration is Earned Value Management (EVM, [1]). While EVM provides performance metrics for both the cost performance and schedule performance of projects, all EVM metrics are cost-based. As a result, it is known that the EVM schedule performance indicators are less reliable towards the end of a project. The earned schedule concept (ES, [2]) has been created to tackle this disadvantage. Project time management could thereafter be managed with the same kind of method as cost. However, while Vandevoorde and Vanhoucke [3] demonstrated that ES provides, on average, more accurate project duration forecasts than any other technique available at the time, the main drawback of ES is that it uses the EV as a proxy to get the duration, which entails that the ES is still cost-based. Earned Duration Management (EDM) has been developed by Khamooshi and Golafshani [4] to eliminate this drawback. EDM is a novel technique that creates duration-based performance metrics to remove one of the ES perceived problems, namely its time indicator sensitivity to activity planned cost. The aforementioned monitoring techniques all provide formulas to forecast the project duration, which is the main topic of this paper.
The goal of this paper is twofold. First, we aim to improve the accuracy of project time forecasting. While the standard EVM/ES and EDM time forecasting formulas only consider the schedule performance as an indicator for future performance to predict the final project duration, we propose a forecasting approach that focuses on two aspects of projects in progress, namely the schedule performance and the schedule adherence. The latter is incorporated in the proposed forecasting approach since a low schedule adherence can lead to the occurrence of rework [5]. Second, we aim to improve the ability to indicate a priori the expected accuracy of forecasting methods based on the project characteristics. Currently, the project serial/parallel (SP) topological characteristic is the most used as an indicator of the potential accuracy of project time forecasting. Nevertheless, there are
E-mail addresses: pandre@techisa.net (P.A.d. Andrade), annelies.martens@ugent.be (A. Martens), mario.vanhoucke@ugent.be (M. Vanhoucke).
https://doi.org/10.1016/j.autcon.2018.11.030
Received 10 July 2018; Received in revised form 24 October 2018; Accepted 29 November 2018
0926-5805/ © 2018 Elsevier B.V. All rights reserved.
Table 1
General concepts and EVM and EDM metrics.
General concepts
AT Actual Time The date of project data collection
BPD Baseline Planned Duration The project approved duration
RD Real Duration The project actual duration
SP Serial/Parallel-indicator A topological network indicator
MAPE Mean Absolute Percentage Error A measure for the forecasting accuracy
Earned Value/Earned Schedule
AC Actual Cost The costs incurred up to AT
BAC Budget At Completion The approved budget cost for the project
PV Planned Value The monetary value planned te be earned up to AT
EV Earned Value The value earned up to AT
ES Earned Schedule The time at which the EV was planned to be earned
EAC(t) Estimate at completion (time) The prediction of the RD made at AT
SPI Schedule performance index SPI=EV/PV
SPI(t) Schedule performance index (time) SPI(t)=ES/AT
SV(t) Schedule variance (time) SV(t)=ES-AT
p-factor Schedule adherence A measure for the schedule adherence
RIEVM EVM Regularity/Irregularityindex A project regularity indicator
Earned duration management
PD Planned Duration The number of work periods of an activity planned to be earned up to AT
TPD Total Planned Duration The cumulative number of planned working periods up to AT
TED Total Earned Duration The cumulative number of working periods earned up to AT
ED Earned Duration The time that TED was planned to be earned
EDAC Estimated Duration at Completion The prediction of RD mate at AT
DPI Duration performance index DPI=ED/AT
EDI Earned Duration Index EDI= TED/TPD
c-factor Schedule compliance A measure of compliance to the schedule
RIEDM EDM Regularity/Irregularity-index A project regularity indicator
continuing efforts to create more reliable predictors for the accuracy of the estimates of project completion time. Batselier and Vanhoucke [6] proposed the regular/irregular-indicator (RI), a novel project characteristic based upon the EVM technique, which reflects how value is accrued during project execution. Their study regarded this indicator as a stronger influencer to time and cost forecasting accuracy than the SP indicator. In this paper, we extend the concept of RI to EDM and evaluate the reliability of this project characteristic as a predictor of the expected accuracy of duration forecasting methods.
The remainder of this paper is structured as follows. In Section 2, the concepts of EVM and EDM that are important for this study are discussed. Subsequently, the research approach and methodology is described in Section 3. The results of the empirical experiment are discussed in Section 4. Finally, the conclusions of this study are discussed in Section 5.
2. Literature review
In this section, the most important concepts that are used in this study are discussed. Further, Table 1 summarises the concepts and abbreviations used in this paper. For the sake of clarity, we grouped the general concepts, EVM/ES concepts and EDM concepts.
2.1. Project monitoring
2.1.1. Earned Value Management
EVM is an established methodology to monitor projects during execution that uses three key metrics to measure the progress of projects, namely the Planned Value (PV), the Earned Value (EV) and the Actual Cost (AC). The PV represents the cumulative value that is planned at every period during the project makespan. Further, the EV and AC indicate the cumulative created value and incurred costs at every time period. The PV, EV and AC are used to construct schedule and cost performance indicators. More specifically, two schedule performance indicators can be constructed using the PV and EV, namely the Schedule Variance (SV= EV-PV) and the Schedule Performance Indicator (SPI=EVPV). Consequently, when the value earned is lower than planned and the project suffers from a delay, the SV and SPI will be lower than 0 and lower than 1, respectively.
It should be noted that the EV is a measure for the actual progress of the project which does not account for the congruence with the schedule. However, performing work not according to the baseline schedule often indicates activity impediments (i.e. activities are performed less efficient than planned) or is a likely cause of rework (i.e. activities that are performed ahead of schedule are performed under a degree of risk). Therefore, Lipke [5] introduced the concept of schedule adherence, or the p-factor, as a tool to reveal these impediments and to detect the portion of work that is performed under risk. More specifically, the pfactor measures the portion of earned value accrued in congruence with the baseline schedule and is calculated using the following equation
[7]:
min(PVi,ES, EVi,AT) p = i N
i N PVi,ES (1) with p the schedule adherence and N the set of activities in the project. Using the p-factor, the EV can be split up in a part that has been performed in congruence with the baseline schedule (EV(p)) and a part that is performed under risk (EV(r)). Using this information, the possibility of rework can be taken into account by determining the effective EV (EV(e), as defined in Eq. (2)).
EV(e) = EV(p) + R% EV(r) (2) with R% the estimated portion of the EV(r) that is usable and does not require rework. Using the EV(e) instead of the EV, the performance measures SPI and SPI(t) are decreased to account for possible rework. Vanhoucke [7] have examined the impact of this correction on the duration forecasting accuracy of earned value metrics using Monte Carlo simulation. The computational experiment has shown that the pfactor might help in predicting the project duration forecasting accuracy. For a more extensive overview on the concepts and metrics of EVM, the reader is referred to Vanhoucke [8] and Fleming and Koppelman [1]. Further, an overview of recent developments and extensions of EVM is given by Willems and Vanhoucke [9].
2.1.2. Earned schedule
Since all EVM key metrics are cost-based, ES [2] has been developed as an extension of EVM to monitor the schedule progress in time units. More specifically, ES informs the time at which the current earned value (EV at actual time AT) was planned to be earned. ES is calculated using the following equation:
EV PVt ES= +t PVt+1 PVt (3) where t is the number of integer time periods for which EV≥PVt and EV ≤PVt+1. The ES schedule performance indicators are determined as SV
(t)= ES-AT and SPI(t) when the project suffers from a delay.
2.1.3. Earned Duration Management
While the ES is expressed in time units, it is calculated based on the EV and PV and therefore still depends on the cost structure of the project. EDM has been developed by Khamooshi and Golafshani [4] to resolve this issue. The key metrics of EDM are the total planned duration (TPD), total earned duration (TED) and the earned duration (ED), which are all time-based metrics. Rather than considering the value of project activities in monetary units, the value of the activities is expressed in work periods.
First, the total planned duration (TPD) represents the cumulative number of planned work periods throughout the project makespan and should thus not be confused with the project planned duration or baseline planned duration (BPD). The TPD is determined using the following equation:
n
TPD= PDi
i=1 (4) where PDi is the portion of the planned duration for activity i that occurs up to (AT), and n is the number of completed or in-progress activities up to AT. For in-progress activities, only the duration of the activity to the left of the AT counts to determine the TPD. Hence, the TPD can be seen as the time-based equivalent of the PV.
Further, the total earned duration (TED) represents the cumulative number of working periods earned throughout the project makespan and is calculated using the following equation:
n
TED= EDi
i=1 (5) where EDi is the earned duration for activity i up to AT, and n the number of completed or in-progress activities up to AT. Thus, the TED is the time-based equivalent of the EV.
Finally, the earned duration (ED) informs the time at which the current total planned duration (TPD at AT) was planned to be earned and can be considered to be the EDM equivalent of the ES. The ED is calculated using the following equation:
TED TPD ED
+1
where t is the number of integer time periods for which TED≥TPDt and
and the Duration Performance Index are respectively the EDM equivalents for the SPI and SPI(t).
2.2. Project forecasting
The project monitoring information collected during project progress can be used to produce forecasts for the final project duration. The general formula for the EVM time forecasting concept, the Estimate at Completion for time or EAC(t), is defined as follows:
EAC(t) = +AT PDWR (7) with AT the actual time of the project and PWDR the Planned Duration of Work Remaining. Three EVM time forecasting methods can be distinguished, namely the Planned Value Method (PVM) proposed by Anbari [10], the Earned Schedule Method (ESM) proposed by Lipke [2] and the Earned Duration Method (EDM) proposed by Jacob and Kane in 2004 and discussed in Vanhoucke [8]. It should be noted that this EDM method is not the same as the EDM method of Khamooshi and Golafshani [4]. Vandevoorde and Vanhoucke [3] compared these forecasting techniques using an extensive simulation study and recommend the use of the ESM method, since it is the only method which showed reliable results during the whole project duration. The formula to determine the EAC(t) using the ESM method is as follows:
BPD ES
EAC(t) = +AT PF (8) with PF a performance factor indicating the expected performance of the future work. The most commonly used performance factors are 1 and SPI(t), when the future performance is expected to follow the baseline schedule or the current time performance, respectively [3]. While the empirical study of Batselier and Vanhoucke [11] showed that a performance factor of 1 resulted in a higher accuracy, it can be argued that this method is not realistic since it does not take the current schedule performance into account [12].
Further, in recent literature, several studies focused on improving the project duration forecasting accuracy using historical data or Monte Carlo simulation. Elshaer [13] integrated activity sensitivity measures in the duration forecasting process in order to improve the forecasting accuracy. In Wauters and Vanhoucke [14] and Wauters and Vanhoucke [15], the forecasting accuracy is improved by using Artificial Intelligence methods such as Nearest Neighbours and Support Vector Machines. Finally, Batselier and Vanhoucke [6] integrate EVM with reference class forecasting (RCF) to enhance the accuracy of project duration forecasting.
Lastly, the EDM equivalent of the EAC(t), the Estimated Duration at Completion or EDAC, can be defined as follows:
BPD ED
EDAC AT= + PF (9)
Since the DPI is the counterpart of the SPI(t), it can be used as a performance factor when the future performance is expected to follow the current time performance.
2.3. Project characteristics
The expected accuracy of forecasting methods varies highly among different types of projects. Projects can, however, be classified into different categories for which the expected forecasting accuracy is more
Table 2
Overview of different composite factors.
EAC(t) EDAC
Notation PF Eq. (8) PF Eq. (9)
WA9010 0.9 ⋅SPI(t) + 0.1 ⋅p-factor 0.9 ⋅DPI + 0.1 ⋅c-factor
WA8020 0.8 ⋅SPI(t) + 0.2 ⋅p-factor 0.8 ⋅DPI + 0.2 ⋅c-factor
WA7030 0.7 ⋅SPI(t) + 0.3 ⋅p-factor 0.7 ⋅DPI + 0.3 ⋅c-factor
WA6040 0.6 ⋅SPI(t) + 0.4 ⋅p-factor 0.6 ⋅DPI + 0.4 ⋅c-factor
WA5050 0.5 ⋅SPI(t) + 0.5 ⋅p-factor 0.5 ⋅DPI + 0.5 ⋅c-factor
WA4060 0.4 ⋅SPI(t) + 0.6 ⋅p-factor 0.4 ⋅DPI + 0.6 ⋅c-factor
WA3070 0.3 ⋅SPI(t) + 0.7 ⋅p-factor 0.3 ⋅DPI + 0.7 ⋅c-factor
WA2080 0.2 ⋅SPI(t) + 0.8 ⋅p-factor 0.2 ⋅DPI + 0.8 ⋅c-factor
WA1090 0.1 ⋅SPI(t) + 0.9 ⋅p-factor 0.1 ⋅DPI + 0.9 ⋅c-factor
homogeneous. In the remainder of this section, we will discuss two project characteristics that can be used for such a categorisation, namely the serial/parallel-indicator and the regular/irregular-indicator.
The serial/parallel indicator SP [16,17] is a project characteristic that informs how close the project activities network is to either a serial or parallel network. The SP values vary from 0 to 1, the closer to 1 the more serial the network is. It is calculated by the following formula:
ns 1
SP=
nt 1 (10) where ns is the maximum number of subsequent activities in the network (or the maximum progressive level, [18]) and nt is the total number of activities in the project.
The regular/irregular-indicator for EVM (RIEVM, [6]) is a project characteristic that measures the PV-curve degree of closeness to a perfectly linear curve. The RIEVM is calculated using the following equation:
RIEVM =
(11)
with mi the maximal possible deviation of the PV-curve from the perfectly linear curve at time instance i, ai the actual deviation of the PVcurve from the perfectly linear curve at time instance i and r the number of equidistant evaluation points. For this paper, the evaluation points were chosen to be each workday of the project duration.
3. Research approach and methodology
3.1. Research approach
The goal of this paper is twofold. First, we aim at improving the project time forecasting accuracy in order to improve the reliability of the forecasted project duration. Second, to provide the project manager with a more reliable indication of the expected accuracy of the forecasting method he or she is using, we aim at improving the ability of project characteristics to indicate the expected accuracy of forecasting methods.
Goal 1: improvement of forecasting accuracy Rather than focusing on a singular PF in project time forecasting formulas, we introduce composite performance factors for project time forecasting. In project cost forecasting, a composite performance factor has been introduced to assume that the cost of future work would follow the current time and cost formula. More precisely, the composite PF ‘0.8 CPI + 0.2 SPI(t)’ has been used [19]. In this study, we introduce a composite PF that is a linear combination of the schedule performance and schedule adherence in order to improve the project time forecasting performance. The composite factors are thus of the form k ×schedule performance + (1 − k) ×schedule adherence, with 0 ≤ k ≤ 1. In order to evaluate which weights result in the highest forecasting accuracy, we evaluate the forecasting accuracy for k ∈ [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]. Since we construct the composite factors for both EVM and EDM, we will evaluate the forecasting accuracy of 9 composite factors for the EAC(t) and EDAC. For the EAC(t), the schedule performance and schedule adherence are represented by the SPI(t) and the p-factor, which are both established EVM concepts. For EDM, we represent the schedule performance and schedule adherence by the DPI and the c-factor. Since the c-factor has not yet been formally determined, we define the c-factor as follows:
min(PDi,ED, EDi,AT) c = i N
i N PDi,ED (12) with c being the compliance to the schedule. Hence, similar to the pfactor, the c-factor represents the portion of the executed project that is performed in accordance with the baseline schedule.
To summarise, Table 2 lists the composite factors that are used in this study. The performance of these composite factors will be compared to the performance of the standard formulas. Since the composite factors are a weighted arithmetic average (WA) of the schedule performance and schedule adherence, we denote the composite factors that are calculated in the experiment. For instance, EAC(t)-WA9010 refers to the composite factor using SPI(t) and the p-factor, with k = 0.9 and 1 − k = 0.1.
Goal 2: improvement of ability to indicate expected forecasting accuracy By using the most accurate indicator of expected forecasting accuracy, the project manager can have the most reliable indication of the forecasting accuracy that is to be expected for a certain project. Batselier and Vanhoucke [6] found that the RI, which is calculated using EVM information of the project, has a higher ability than the frequently used SP-indicator to indicate the expected accuracy of project time forecasts. Since the RI defined by Batselier and Vanhoucke [6] is calculated using EVM information, we refer to this concept as RIEVM. In this study, this concept is extended to EDM and referred to as RIEDM. The RIEDM is calculated as follows:
RIEDM =
(13)
with mi′ the maximal possible deviation of the TPD-curve from the perfectly linear curve at time instance i, ai the actual deviation of the TPD-curve from the perfectly linear curve at time instance i and r the number of equidistant evaluation points. Although the RIEDM concept is similar to RIEVM, the RIEDM provides different values than the RIEVM by using the TPD-curve rather than the PV-curve. For the most accurate time forecasting approaches, we will compare the ability of the SP, RIEVM and RIEDM to indicate the expected accuracy of the produced forecasts. 3.2. Illustrative example
The calculations of the c-factor, the RIEDM and the forecasting formulas are illustrated with an example project consisting of 10 activities. Fig. 1 depicts the project network and lists the planned duration and planned cost of the activities below the activity nodes.
3.2.1. Baseline schedule
Fig. 2 shows the baseline schedule of the illustrative example. The BPD and BAC of this example are equal to 30 time units and €96, respectively.
Further, the PV-curve and TPD-curve of the project are depicted in Fig. 3. Using the information from Figs. 2 and 3, the project characteristics SP, RIEVM and RIEDM can be determined as follows. The SPvalue of the project can be determined using Eq. (10). Since the project consists of 10 activities and the maximum number of subsequent activities in the network is 4, the SP of this project is equal to 33.33% . The project can thus be considered to be a parallel project. Further, the RIEVM can be determined using Eq. (11). For instance, for
Fig. 1. Illustrative example: Baseline schedule.
time point 18, the mi and ai are equal to 57.6 (=BACBPD × = ×t 18) and 14.4 (= |mi −PVi| = |57.6 − 43.2|), respectively. By determining the mi and ai for all time units, the RIEVM can be determined to be 73.9%. Similarly, RIEDM is calculated using Eq. (13), with mi′ and ai′ equal to 43.2 (= ×18) and 3.8 (= |43.2 − 47|) respectively. The RIEDM of the project is equal to 85.8%.
3.2.2. Project monitoring
Fig. 4 shows the illustrative example in progress until t=18. Using this information, both the schedule adherence and schedule performance can be calculated for EVM and EDM. This information can then be used to produce the duration forecasts.
The schedule performance of this project can be expressed using the SPI(t) or the DPI. For these metrics, the ES and ED of the project are required. Fig. 5 shows the EV-curve and TED-curve up to t=18 such that the ES and ED of the project can be determined. As can be seen in this figure, the project is ahead of schedule in terms of EVM (ES = 21.3) and behind schedule in terms of EDM (ED = 17). Consequently, the schedule performance of the illustrative example is equal to 118.4% (= ) in terms of SPI(t) and 94.4% (= ) in terms of DPI.
Further, the schedule adherence of this example can be expressed using the p-factor (Eq. (1)) or the c-factor (Eq. (12)). In order to use these formulas, the EV or ED of the activities and the ES or ED of the project are required. The EV and ED of the activities are listed in Fig. 3 and the ES and ED of the project are determined in Fig. 5. Therefore, the p-factor and c-factor are equal to 72.4%
(= 1 + +4 112+ + +×4 8.3 / 126 20+ + + + +×20.3 / 33 + + + + + +6 3 206 1010× 10.3 / 126 9 + +0 12 0 = ) and 86%
(= = respectively.
The duration forecasts can be constructed using the available information on the schedule performance and schedule adherence. In Table 3, the results of the EAC(t), EDAC, EAC(t)-WA9010 and EDACWA9010 are depicted. This table shows that the results for different forecasting formulas might vary substantially. In this example, the EAC(t) and EAC(t)-WA9010 forecasts (=25.34 and 25.64 respectively) are more positive than the standard EDAC (=31.76) and weighted EDAC formula with k = 0.9 (=31.89).
3.2.3. Project completion
Finally, Fig. 6 shows the project at completion. As can be seen from this figure, the actual duration of the project equals 32 days. The accuracy of the forecasting formulas can be determined in retrospect. For t = 18, the absolute percentage errors for the EAC(t), EAC(t)-WA9010, EDAC and EDAC-WA9010 are 23.21% (=|33 3325.34|), 22.30%
(=|33 3325.64|), 3.76% (=|33 3331.76|) and 3.36% (=|33 3331.89|) respectively. Thus, for the example project, the standard and weighted EDAC forecasting formulas are more accurate than the EAC(t) formula at t = 18. The MAPE of the forecasting formulas can be determined as the mean of the absolute percentage errors over all tracking periods.
3.3. Research methodology
The research methodology of this study consists of three steps, namely the data selection, data analysis and performance evaluation (Fig. 7). In the remainder of this section, these steps are discussed in greater detail.
3.3.1. Data selection
In this study, we evaluate the performance of project duration forecasting methods on real life projects from the database of Batselier and Vanhoucke [20]. This database is publicly available on the supporting website [21] and documents the baseline information, risk information and real life progress of 125 projects from varying sectors (e.g. construction, production and software development). During the project selection procedure, we identified several reasons to exclude certain projects from the analysis, namely (i) lacking actual data, (ii) inadequate or strongly invalid data, (iii) most activity durations measured in hours rather than days, (iv) work in week-ends and (v) outliers, e.g. only MAPE values (Section 3.3.3) between 0% and 50% are considered. Table 4 summarises which projects have been excluded from further analysis. Further, this table shows that 57 projects were deemed adequate for the purposes of this paper. From these projects, 50 projects belong to the construction sector, 5 are IT projects and 2 are engineering projects.
3.3.2. Data analysis
In order to evaluate which characteristic is the best indicator of the potential accuracy of project time forecasting, we determine the SP, RIEVM and RIEDM of the 57 retained projects. Based on these values, each project will be classified in categories ranging from parallel to
Fig. 2. Toy example: Baseline schedule.
Fig. 3. Determining the project regularity.
serial projects and from strongly irregular to regular projects. Table 5 shows the categories for SP-values and RI-values as defined by Batselier and Vanhoucke [20] and Batselier and Vanhoucke [6] respectively and indicates the number of projects in each category.
3.3.3. Performance evaluation
In Section 4, we review the performance of two components of the empirical experiment. First, in Section 4.1, we evaluate the forecasting accuracy of the composite forecasting formulas introduced in this paper. Second, we examine the ability of the SP, RIEVM and RIEDM to indicate the expected forecasting accuracy in Section 4.2. In the remainder of this section, we discuss the performance measures that are used to evaluate both components.
3.3.3.1. Performance of forecasting formulas. The performance of the standard and composite forecasting formulas for EVM and EDM will be evaluated using the Mean Absolute Percentage Error (MAPE). The MAPE is frequently used in studies on EVM time forecasting [22,13,11] and is calculated as follows:
1 T A Ft
MAPE=
T t=1 A (14) with A the actual duration at project completion, Ft the forecasted duration at tracking period t and T the number of tracking periods.
In order to compare the performance of the forecasting methods, we evaluate the avg MAPE and the % better, which indicates in how many cases the evaluated forecasting method performs better (i.e., has a lower MAPE) than the method it is compared to.
The % better of a forecasting method a compared to forecasting method b is thus calculated as follows:
ni=1 (MAPEi a, < MAPE )i b,
% better =
n (15)
with n the number of projects, MAPEi,x the MAPE of forecasting method x for project i and the indicator function which is 1 if the event is true and 0 otherwise.
3.3.3.2. Performance of forecasting accuracy indicators. In order to compare the performance of the SP, RIEVM and RIEDM, the R-squared statistic will be used. This statistic presents the proportion of the variance in the dependent variable (i.e., the MAPE) that is predictable from the independent variable(s) (i.e., the SP, RIEVM or RIEDM) and has been used by Batselier and Vanhoucke [6] to indicate the relation between the project characteristic and forecasting accuracy.
4. Results
4.1. Forecasting performance of composite factors
In this study, we have developed a new project time forecasting formula for EVM and EDM that is based on the schedule performance and schedule adherence of projects that are being executed. First, we make a comparison of the standard EVM and EDM forecasting formulas. Subsequently, we compare the performance of the EVM and EDM composite factors to the standard formulas for EVM and EDM, respectively. As a result, we compare the difference in performance between two project monitoring methodologies (EVM and EDM), and between the two considered approaches (standard formulas and composite factors).
The performance of the EVM and EDM standard formulas is described in Table 6. As can be seen from this table, the avg MAPE of EDAC is lower compared to the avg MAPE of EAC(t). Further, the table shows that EDAC presented a smaller MAPE than EAC(t) in 64.9% of the projects while EAC(t) was better in 33.3% of the cases. The difference to 100% (1.8%) is due to one project for which EAC(t) and EDAC had the same performance. Therefore, we can conclude that the standard EDM forecasting formula using DPI performs better than the standard EVM forecasting formula using SPI(t).
In Table 7, the performance of the composite factors is summarised
Fig. 4. Illustrative example: Project monitoring.
Fig. 5. Determining the project performance.
Table 3
Duration forecasts for the example project.
Formula
EAC(t) Eq. (8) EDAC Eq. (9)
EAC(t)-WA9010 0.9⋅SPI(t)+0.1⋅p-factor 0.9 118.4% 0.1 72.4%
EDAC-WA9010 0.9⋅DPI+0.1⋅c-factor 30 17
and compared to the standard formulas for EVM and EDM project time forecasting. The ‘Standard’ row of this table shows the avg MAPE of the standard formulas, the following rows depict the performance of the 9 composite factors in terms of avg MAPE and % better for both EVM an EDM. The values in bold indicate the best performance in terms of avg MAPE and % better than the standard formula, for both EVM and EDM.
For the EVM composite factors, only factors WA2080 and WA1090 perform worse (i.e., have a higher avg MAPE) than the standard EVM formula. Further, for the EDM composite factors, only WA1090 performs worse than the standard EDM formula. Hence, when the weights are very low for the schedule performance (SPI(t) or DPI) and very high for the schedule adherence (p-factor or c-factor), the composite factors have a lower performance than the standard formulas. In all other cases, the composite factors perform better. More specifically, the lowest avg MAPE is reached by WA7030 and WA6040 for EVM and by WA5050 for EDM. Further, the MAPE of the WA9010 composite factor is better than the EVM and EDM standard formulas in 75.4% and 78.9% of the cases, respectively. Therefore, the composite factors WA9010, WA6040 and WA7030 for EVM and the composite factors WA9010 and WA5050 for EDM can be considered to be the best performing composite factors.
4.2. Performance of forecasting accuracy indicators
Section 4.1 showed that, for the standard formulas, the EDAC formula is more accurate than the EAC(t) formula. Further, the composite factors introduced in this paper show to be more accurate than the standard formulas. Therefore, in this section, we evaluate the ability of the SP, RIEV M and RIEDM to indicate the expected forecasting accuracy for the standard EDAC formula (Section 4.2.1) and the most accurate composite factors for EDM (Section 4.2.2). As discussed in Section 3.3, we compare the R2 of plotting the MAPEs against the SP and RIEDM to determine which metric has the highest ability to indicate expected forecasting accuracy.
4.2.1. Forecasting accuracy indicators for EDAC
In Figs. 8 and 9, the MAPE of each project is plotted against the SPindicator and RIEDM indicator. Both figures show a decreasing trend. More specifically, for increasing SP-values, the forecasting accuracy increases. Similarly, the higher the RIEDM, the higher the forecasting accuracy is. It can be noted that the R2 for the RIEDM indicator (8.9%) is substantially larger than the R2 for the SP-indicator (3.6%). We can thus conclude that the RIEDM indicator is a better indicator for the EDAC standard formula accuracy than the SP-indicator.
Since the RIEDM indicator is the best indicator for the forecasting accuracy of the standard EDAC formula, we discuss the performance of the EDAC standard formula for the different RI-categories in Table 8. As this table shows, there are no strongly irregular projects in our dataset. Further, most projects (46 of 57) are regular projects. While the avg MAPE for irregular projects is 14.5%, the avg MAPE for regular projects improves to 11.3%.
4.2.2. Forecasting accuracy indicators for EDM composite factors Table 7 showed that the EDAC composite factors WA5050 and WA9010 perform the best in terms of avg MAPE and % better, respectively. Therefore, we evaluate the ability of the SP and RIEDM to indicate the expected forecasting accuracy for these composite factors as well.
Fig. 6. Toy example: Project completion.
Fig. 7. Research methodology.
Table 4
Selection of projects for the study.
Reasons for removal Project IDs Quantity
Projects without actual data C2011-1, C2011-02, C2011-3, C2011-4, C2011-6, C2011-08,
C2011-09, C2011-11, C2011-14,
C2012-01, C2012-02, C2012-03,
C2012-04, C2012-05, C2012-06,
C2012-07, C2012-08, C2012-09,
C2012-10, C2012-11, C2012-12,
C2012-14, C2012-16 23
Projects with inadequate or strong invalid data C2013-13, C2013-15, C2013-16, 5
C2013-17, C2015-32, C2011-07,
C2011-10, C2012-15, C2012-17, 20
Projects with most activities with duration measured in hours C2014-03, C2015-10, C2015-11,
C2015-12, C2015-13, C2015-14,
C2015-15, C2015-16, C2015-17,
C2015-18, C2015-19, C2015-20,
C2015-21,C2015-22, C2015-23,
C2015-24
Project with work in week-ends C2015-25, C2013-02, C2013-08, 1
C2013-10, C2013-12, C2013-14, C2015-01, C2015-02, C2015-09 19
Projects with outlier MAPE values C2016-16, C2016-17, C2016-18,
C2016-19, C2016-20, C2016-21,
C2016-22, C2016-23, C2016-24,
C2016-25, C2016-26
Number of projects unfit for the study 68 Number of projects in Batselier and Vanhoucke [20] database 125 Number of projects used in the study 57
Table 5
Classification of selected projects.
SP-indicator
Category Values # projects
Parallel projects [0%, 40%) 23
Serial-parallel projects [40%, 60%] 19
Serial projects (60%, 100%] 15
RI-indicator
Category Values # projects
RIE V M RIE D M
Strongly irregular projects [0%, 60%) 1 0
Irregular projects [60%, 80%] 12 11
Regular projects (80%, 100%] 44 46
Table 6
Comparison of technique results for estimates at completion using all 57 projects.
Metric EAC(t) EDAC
avg MAPE 12.9% 11.9%
% better 33.3% 64.9%
Figs. 10 and 11 illustrate the scattered diagrams of the forecasting MAPEs plotted against the SP and RIEDM-indicator. Similar as for the standard EDAC formula, the forecasting accuracy of the composite factors WA9010 and WA5050 shows a decreasing trend for increasing SP and RIEDM values. Hence, the forecasting accuracy is higher for projects that are more serial (Fig. 10) or more regular (Fig. 11). However, contrarily to the standard EDAC formula, the differences in R2 between the use of the SP or RIEDM-indicators are substantial. The R2 of
Table 7
Comparison of standard vs PF1 composite for 57 projects.
Standard 12.9 − 11.9 − WA9010 11.9 75.4 11.0 78.9
WA8020 11.3 71.9 10.4 71.9
WA7030 10.9 70.2 10.0 70.2 WA6040 10.9 68.4 9.8 63.2
WA5050 11.2 66.7 9.7 59.6 WA4060 11.8 61.4 10.0 57.9
WA3070 12.8 59.6 10.5 54.4
WA2080 14.4 52.6 11.4 47.4
WA1090 18.2 50.9 12.9 43.9
Fig. 8. EDM: scattered diagram of MAPEs for EDAC vs SP indicators.
Fig. 9. EDM: scattered diagram of MAPEs for EDAC vs RI indicators.
Table 8
EDAC forecasting accuracy for different RI-categories.
Brackets
[0%, 60%) 0 − −
[60%, 80%] 11 14.5% 63.6% (80%, 100%] 46 11.3% 65.2%
using the SP as an indicator for forecasting accuracy is very low for both WA9010 and WA5050 (3.33% and 0.4% respectively). The R2 of using the RIEDM as an indicator for forecasting accuracy is more substantial (10.31% for WA9010 and 15.41% for WA5050). Consequently, the RIEDM clearly outperforms the SP as an indicator for forecasting accuracy for the best performing EDAC composite formulas WA9010 and WA5050.
Since the RIEDM indicator is the best indicator for the forecasting accuracy of the best performing EDAC composite formulas, we discuss the performance of these composite formulas for the different RI-categories in Table 9 as well. This table shows that, for irregular projects, the WA9010 composite formula outperforms the WA5050 composite formula both in terms of avg MAPE (13.9% vs 14.4%) and % better (81.8% vs 36.4%). For the regular projects, however, the WA9010 composite formula performs best in terms of % better (78.3% vs 65.2%) while the avg MAPE of the WA5050 composite formula is lower
Fig. 10. EDM: scattered diagram of MAPEs for WA9010 and WA5050 vs SP indicators.
Fig. 11. EDM: scattered diagram of MAPEs for WA9010 and WA5050 vs RIEDM indicators.
Table 9
EDAC forecasting accuracy for different RI-categories.
Category
[0%, 60%) 0 − − − − [60%, 80%] 11 13.9% 81.8% 14.4% 36.4%
(80%, 100%] 46 10.3% 78.3% 8.6% 65.2%
compared to the WA9010 composite formula (8.6% vs 10.3%). Hence, for irregular projects, WA9010 is clearly the preferred composite factor to use. For regular projects, WA9010 is the preferred factor in terms of % better while WA5050 is preferred in terms of avg MAPE.
4.3. Limitations of the experiment
First, we examined the ability of different project characteristics to indicate the expected forecasting accuracy for specific projects. Therefore, we followed the approach of Batselier and Vanhoucke [12] by plotting the MAPEs of the project against the SP and RIEDM. Although the resulting R-squared values of using the SP and RIEDM differ substantially, they are relatively low for both methods. Hence, while the RIEDM is clearly a better indicator of expected accuracy than the SP according to the R-squared value of the linear regression, this might suggest that the relation between the project characteristics and the forecasting accuracy could be nonlinear. Second, in this study, we used 57 real-life projects to evaluate the performance of different project duration forecasting techniques and to assess the ability of different project characteristics to indicate the expected accuracy of these techniques for a specific project. Although this set of projects is substantial, including more data would increase the validity of the results even further.
5. Conclusions
This paper focused on forecasting the final project duration for projects in progress. The contribution of this study is twofold. First, we aimed to improve the accuracy of the project duration forecasts by introducing composite performance factors that combine the schedule performance and schedule adherence as an indicator for the future performance of projects. Therefore, the schedule adherence concept has been extended to EDM and is referred to as the schedule compliance. The resulting c-factor is a measure for schedule adherence that is independent of the activity costs. Second, we intended to improve the ability to indicate the expected accuracy of the duration forecasts for a specific project by extending the project regularity concept of Batselier and Vanhoucke [6] to EDM.
In order to review the performance of the composite factors and the ability of the EDM regularity indicator to indicate the expected accuracy, an empirical experiment has been conducted on 57 projects from the database of Batselier and Vanhoucke [20]. The experiment has shown that composite performance factors that combine the schedule performance and schedule adherence have a higher accuracy than the standard forecasting formulas that focus on the schedule performance only. Further, it was shown that the RIEDM is a better indicator for the expected accuracy of the best performing forecasting techniques that use composite performance factors.
A future research avenue could consist of investigating different weights for the composite performance factors for different stages of the project in order to evaluate which weights for the beginning, middle and end stage of the project result in the highest forecasting accuracy. Moreover, the type of relationship between project characteristics and the expected forecasting accuracy could be investigated by considering nonlinear regression. Finally, the dataset can be expanded further by extending the existing database of real-life data and by conducting simulation experiments on artificial data. The main advantage of using artificial data is that a vast amount of project executions can be generated from projects with varying characteristics. However, empirical data are more realistic than artificial data and include more projectspecific information such as the industry. Therefore, by extending the empirical dataset, insights into the differences between the industry types of the projects can be examined. References
[1] Q. Fleming, J. Koppelman, Earned Value Project Management, ISBN 9781935589082, 3rd ed., Project Management Institute, Newton Square, Pennsylvania, 2010.
[2] W. Lipke, Schedule is different, The Measurable News Summer (2003) 31–34 http://earnedschedule.com/Papers.shtml last visited 09/01/2017.
[3] S. Vandevoorde, M. Vanhoucke, A comparison of different project duration forecasting methods using earned value metrics, Int. J. Proj. Manag. 24 (2006) 289–302, https://doi.org/10.1016/j.ijproman.2005.10.004.
[4] H. Khamooshi, H. Golafshani, EDM: Earned Duration Management, a new approach to schedule performance management and measurement, Int. J. Proj. Manag. 32 (6) (2014) 1019–1041, https://doi.org/10.1016/j.ijproman.2013.11.002.
[5] W. Lipke, Connecting earned value to the schedule, The Measurable News Winter 1 (2004) 6–16 URL http://earnedschedule.com/papers.shtml last visited 18/10/ 2018.
[6] J. Batselier, M. Vanhoucke, Project regularity: development and evaluation of a new project characteristic, J. Syst. Sci. Syst. Eng. 26 (1) (2017) 100–120, https://doi. org/10.1007/s11518-016-5312-6.
[7] M. Vanhoucke, The impact of project schedule adherence and rework on the duration forecast accuracy of earned value metrics, in: E.C. Hoffmann (Ed.), Project Management: Practices, Challenges and Developments, NOVA Publishers, 2013, pp.
95–131 ISBN 978-1629481838.
[8] M. Vanhoucke, Measuring Time - Improving Project Performance using Earned
Value Management, of International Series in Operations Research and
Management Science, vol. 136, Springer, 2010ISBN 978-1-4419-1014-1.
[9] L. Willems, M. Vanhoucke, Classification of articles and journals on project control and earned value management, Int. J. Proj. Manag. 33 (7) (2015) 1610–1634, https://doi.org/10.1016/j.ijproman.2015.06.003.
[10] F. Anbari, Earned value project management method and extensions. Proj. Manag. J. 34(4) (2003) 12–23, https://doi.org/10.1177/875697280303400403.
[11] J. Batselier, M. Vanhoucke, Evaluation of deterministic state-of-the-art forecasting approaches for project duration based on earned value management, Int. J. Proj.
Manag. 33 (7) (2015) 1558–1596, https://doi.org/10.1016/j.ijproman.2015.04. 003.
[12] J. Batselier, M. Vanhoucke, Improving project forecast accuracy by integrating earned value management with exponential smoothing and reference class forecasting, Int. J. Proj. Manag. 35 (1) (2017) 28–43, https://doi.org/10.1016/j. ijproman.2016.10.003.
[13] R. Elshaer, Impact of sensitivity information on the prediction of project's duration using earned schedule method, Int. J. Proj. Manag. 31 (2013) 579–588, https://doi. org/10.1016/j.ijproman.2012.10.006.
[14] M. Wauters, M. Vanhoucke, A comparative study of Artificial Intelligence methods for project duration forecasting, Expert Syst. Appl. 46 (2016) 249–261, https://doi. org/10.1016/j.eswa.2015.10.008.
[15] M. Wauters, M. Vanhoucke, A Nearest Neighbour extension to project duration forecasting with Artificial Intelligence, Eur. J. Oper. Res. 259 (3) (2017) 1097–1111, https://doi.org/10.1016/j.ejor.2016.11.018.
[16] L. Tavares, J. Ferreira, J. Coelho, The risk of delay of a project in terms of the morphology of its network, Eur. J. Oper. Res. 119 (1999) 510–537, https://doi.org/ 10.1016/S0377-2217(99)00150-2.
[17] M. Vanhoucke, J. Coelho, D. Debels, B. Maenhout, L. Tavares, An evaluation of the adequacy of project network generators with systematically sampled networks, Eur. J. Oper. Res. 187 (2008) 511–524, https://doi.org/10.1016/j.ejor.2007.03.032.
[18] S. Elmaghraby, Activity Networks: Project Planning and Control by Network Models, ISBN 9780471238614 New York: John Wiley and Sons, Inc., 1977.
[19] M. Vanhoucke, Project Management with Dynamic Scheduling: Baseline Scheduling, Risk Analysis and Project Control, ISBN 978-3-642-25175-7, vol. XVIII, Springer, 2012.
[20] J. Batselier, M. Vanhoucke, Construction and evaluation framework for a real-life project database, Int. J. Proj. Manag. 33 (2015) 697–710, https://doi.org/10.1016/ j.ijproman.2014.09.004.
[21] OR-AS, Online Consultation of the Real-life Project Database, last visited 09/01/ 2017, 2017. http://www.projectmanagement.ugent.be/research/data/realdata.
[22] O. Zwikael, S. Globerson, T. Raz, Evaluation of models for forecasting the final cost of a project, Proj. Manag. J. 31 (1) (2000) 53–57, https://doi.org/10.1177/ 875697280003100108.
Subscribe by Email
Follow Updates Articles from This Blog via Email
No Comments