When Planners Lie with Numbers
The text below is an excerpt from the article "How (In)accurate Are Demand Forecasts in Public Works Projects? The Case of Transportation," by Bent Flyvbjerg, Mette Skamris Holm, and Søren Buhl, published in Journal of the American Planning Association, vol. 71, no. 2, Spring 2005. For notes and references, please see the article.
In the present section we consider the situation where planners and other influential actors do not find it important to get forecasts right and where planners, therefore, do not help to clarify and mitigate risk but, instead, generate and exacerbate it. Here planners are part of the problem, not the solution. This situation may need some explication, because it possibly sounds to many like an unlikely state of affairs. After all, it may be agreed that planners ought to be interested in being accurate and unbiased in forecasting. It is even stated as an explicit requirement in the AICP (American Institute of Certified Planners) Code of Ethics and Professional Conduct that "A planner must strive to provide full, clear and accurate information on planning issues to citizens and governmental decision-makers" (American Planning Association 1991, A.3), and we certainly agree with the Code. The British RTPI has laid down similar obligations for its members (Royal Town Planning Institute 2001).
However, the literature is replete with things planners and planning "must" strive to do, but which they don't. Planning must be open and communicative, but often it is closed. Planning must be participatory and democratic, but often it is an instrument to dominate and control. Planning must be about rationality, but often it is about power (Flyvbjerg 1998, Watson 2003). This is the "dark side" of planning and planners identified by Flyvbjerg (1996) and Yiftachel (1998), which is remarkably underexplored by planning researchers and theorists.
Forecasting, too, has its dark side. It is here "planners lie with numbers," as Wachs (1989) has aptly put it. Planners on the dark side are busy, not with getting forecasts right and following the AICP Code of Ethics, but with getting projects funded and built. And accurate forecasts are often not an effective means for achieving this objective. Indeed, accurate forecasts may be counterproductive, whereas biased forecasts may be effective in competing for funds and securing the go-ahead for construction. "The most effective planner," says Wachs (1989, 477), "is sometimes the one who can cloak advocacy in the guise of scientific or technical rationality." Such advocacy would stand in direct opposition to AICP's ruling that "the planner's primary obligation [is] to the public interest" (American Planning Association 1991, B.2). Nevertheless, seemingly rational forecasts that underestimate costs and overestimate benefits have long been an established formula for project approval (Flyvbjerg, Bruzelius, and Rothengatter 2003). Forecasting is here mainly another kind of rent-seeking behavior, resulting in a make-believe world of misrepresentation which makes it extremely difficult to decide which projects deserve undertaking and which do not. The consequence is, as even one of the industry's own organs, the Oxford-based Major Projects Association, acknowledges, that too many projects proceed that should not. We would like to add that many projects don't proceed that probably should, had they not lost out to projects with "better" misrepresentation (Flyvbjerg, Holm, and Buhl 2002).
In this situation, the question is not so much what planners can do to reduce inaccuracy and risk in forecasting, but what others can do to impose on planners the checks and balances that would give planners the incentive to stop producing biased forecasts and begin to work according to their Code of Ethics. The challenge is to change the power relations, which governs forecasting and project development. Here better forecasting techniques and appeals to ethics won't do; institutional change with a focus on transparency and accountability is necessary.
Two basic types of accountability define liberal democracies: (1) Public sector accountability through transparency and public control, and (2) Private sector accountability via competition and market control. Both types of accountability may be effective tools to curb planners' misrepresentation in forecasting and to promote a culture which acknowledges and deals effectively with risk. In order to achieve accountability through transparency and public control, the following would be required as practices embedded in the relevant institutions:
- National-level government should not offer discretionary grants to local infrastructure agencies for the sole purpose of building a specific type of infrastructure, for instance rail. Such grants create perverse incentives. Instead, national government should simply offer "infrastructure grants" or "transportation grants" to local governments, and let local political officials spend the funds however they choose to, but make sure that every dollar they spend on one type of infrastructure reduces their ability to fund another.
- Forecasts should be made subject to independent peer review. Where large amounts of taxpayers' money are at stake, such review may be carried out by national or state accounting and auditing offices, like the General Accounting Office in the US or the National Audit Office in the UK, who have the independence and expertise to produce such reviews. Other types of independent review bodies may be established, for instance within national departments of finance or with relevant professional bodies.
- Forecasts should be benchmarked against comparable forecasts, for instance using reference class forecasting as described in the previous section.
- Forecasts, peer reviews, and benchmarkings should be made available to the public as they are produced, including all relevant documentation.
- Public hearings, citizen juries, and the like should be organized to allow stakeholders and civil society to voice criticism and support of forecasts. Knowledge generated in this way should be integrated in planning and decision making.
- Scientific and professional conferences should be organized where forecasters would present and defend their forecasts in the face of colleagues' scrutiny and criticism.
- Projects with inflated benefit-cost ratios should be reconsidered and stopped if recalculated costs and benefits do not warrant implementation. Projects with realistic estimates of benefits and costs should be rewarded.
- Professional and occasionally even criminal penalties should be enforced for planners and forecasters who consistently and foreseeably produce deceptive forecasts. An example of a professional penalty would be the exclusion from one’s professional organization if one violates its code of ethics. An example of a criminal penalty would be punishment as the result of prosecution before a court or similar legal set-up, for instance where deceptive forecasts have led to substantial mismanagement of public funds (Garett and Wachs, 1996). Malpractice in planning should be taken as seriously as it is in other professions. Failing to do this amounts to not taking the profession of planning seriously.
In order to achieve accountability in forecasting via competition and market control, the following would be required, again as practices that are both embedded in and enforced by the relevant institutions:
The decision to go ahead with a project should, where at all possible, be made contingent on the willingness of private financiers to participate without a sovereign guarantee for at least one third of the total capital needs. This should be required whether projects pass the market test or not, that is, whether projects are subsidized or not or provided for social justice reasons or not. Private lenders, shareholders, and stock market analysts would produce their own forecasts or would critically monitor existing ones. If they were wrong about the forecasts, they and their organizations would be hurt. The result would be more realistic forecasts and reduced risk.
Full public financing or full financing with a sovereign guarantee should be avoided.
Forecasters and their organizations must share financial responsibility for covering benefit shortfalls (and cost overruns) resulting from misrepresentation and bias in forecasting.
- The participation of risk capital should not mean that government gives up or reduces control of the project. On the contrary, it means that government can more effectively play the role it should be playing, namely as the ordinary citizen's guarantor for ensuring concerns about safety, environment, risk, and a proper use of public funds.
If the institutions with responsibility for developing and building major transportation infrastructure project would effectively implement, embed, and enforce such measures of accountability, then the misrepresentation in transportation forecasting, which is widespread today, may be mitigated. If this is not done, misrepresentation is likely to continue, and the allocation of funds for transportation investments is likely to be wasteful.
We conclude that the patronage estimates used by planners of rail infrastructure development are highly, systematically, and significantly misleading (inflated). This results in large benefit shortfalls for rail projects. For road projects the problem of misleading forecasts is less severe and less one-sided than for rail. But even for roads, for half the projects the difference between actual and forecasted traffic is more than ±20 percent. On this background, planners and decision makers are well advised to take with a grain of salt any traffic forecast which does not explicitly take into account the uncertainty of predicting future traffic. For rail passenger forecasts, a grain of salt may not be enough.
The risks generated from misleading forecasts are typically ignored or downplayed in infrastructure planning, to the detriment of social and economic welfare. Risks, therefore, have a doubly negative effect in this particular type of planning, since it is one thing to take on a risk that one has calculated and is prepared to take, much as insurance companies and professional investors do, while it is quite another matter--that moves risk-taking to a different and more problematic level--to ignore risks. This is especially the case when risks are of the magnitude we have documented here, with many demand forecasts being off by more than 50 percent on investments that measure in hundreds of millions of dollars. Such behavior is bound to produce losers among those financing infrastructure, be they tax payers or private investors. If the losers, or, for future projects, potential losers, want to protect themselves, then our study shows that the risk of faulty forecasts, and related risk assessment and management, must be placed at the core of planning and decision making. Our goal with this article has been to take a first step in this direction by developing the necessary data and approach.
The policy implications of our findings are clear. First, the findings show that a major planning and policy problem--namely misinformation--exists for this highly expensive field of public policy. Second, the size and perseverance over time of the problem of misinformation indicate that it will not go away by merely pointing out its existence and appealing to the good will of project promoters and planners to make more accurate forecasts. The problem of misinformation is an issue of power and profit and must be dealt with as such, using the mechanisms of transparency and accountability we commonly use in liberal democracies to mitigate rent-seeking behavior and the misuse of power. To the extent that planners partake in rent-seeking behavior and misuse of power, this may be seen as a violation of their code of ethics, that is, malpractice. Such malpractice should be taken seriously by the responsible institutions. Failing to do so amounts to not taking the profession of planning seriously.
Back to Main Page on the Relationship between Truth and Lying in Policy and Planning