Monday, October 8, 2012

Formal Theory and Policy Relevance

Suppose you are a policymaker and you are facing strong pressure from your core supports to enact policy \(p\).  However, there are those who are concerned that \(p\) will bring about an outcome even worse than the status quo, and regardless of the outcome it produces, the policy will be costly to implement.  While your supporters find the status quo deeply objectionable, polling tells you that the average citizen considers it acceptable.  What should you do?

Fortunately, assembled before you are three academics who study the relevant policy domain.  They are all employed by prestigious universities and have long, distinguished careers.

When you ask expert 1 whether \(p\) is worth enacting, s/he responds "Xh ssh oah."  You ask him to clarify, but s/he only responds with slight variations on the same sounds.  To your ears, it is pure gobbledygook.

When you ask expert 2 whether \(p\) is worth enacting s/he responds, "Absolutely.  In fact, it is imperative that you do so.  Just ask the people of [some country that historically adopted policy \(p\)].  I therefore urge you to show leadership on this issue."

When you ask expert 3 whether \(p\) is worth enacting, s/he responds, "I honestly don't know.  There appears to be some evidence that \(p\) brings about the desired outcome, but there's also some evidence that this relationship is being driven by [some third factor].  One thing I can say for sure is that the only arguments that have been put forth in favor of \(p\) so far are logically inconsistent."

Whose advice should you follow?

Well, I really haven't given you enough information.  If possible, you might try to find someone who can translate the statement made by expert 1 into English.  You might solicit the opinions of several more experts, or look at opinion polls, or listen to your advisers.  So assume, if only for the sake of argument, that you have to make a decision now and there is no time to consult anyone else.

If your goal is to choose the best policy, you might give some real weight to the opinion of expert 3.  Expert 2 hasn't given you anything but a single anecdote.  Expert 3 isn't very confident, but s/he seems to have thought a lot more carefully about the issue.

If your goal is to stay in office, though, you probably go with policy \(p\).  It would be misleading to say that you are heeding the advice of expert 2, of course, because the real reason you are going to enact the policy is because your core supporters want you to.  However, it's awfully convenient that you've got a bigwig you can point to as having endorsed your policy, innit?

Now suppose that you do actually have one adviser in the room.  This adviser knows nothing about the relevant policy domain, and admits as much.  But as the experts leave the room, your adviser remarks that expert 1 didn't even answer your question.  Is that fair?

Suppose I tell you that what expert 1 said roughly translates to, "Well, careful theoretical analysis indicates that it is likely that the costs will exceed the benefits in the short run, at least provided that conditions [x, y, and z] hold, and provided that condition [q] does not hold.  But within five to eight years, the policy will probably pay for itself, and within a decade or two, the welfare gains will be enormous.  Unfortunately, it's hard to tell whether conditions [q] and [z] in particular are going hold."  That would be, you know, relevant.  It doesn't give you a clear course of action, but to say that it doesn't speak to the question you asked would not be fair.  Except, of course, that s/he said what s/he said in Exotic Foreign Language, and thus they've done you no good.  At least, not given the assumption that you've got to make a decision now.

But we can all agree that it's not strictly true that expert 1 failed to answer the question, right?

More importantly, suppose that a large number of the experts you consult, time after time, on issue after issue, responds solely in Exotic Foreign Language.  If a young intern of yours who's thinking about applying to graduate schools asks whether it would be worth studying Exotic Foreign Language, what would you say?  What if you overheard a colleague tell this intern that it would be a mistake to do so, that studying Exotic Foreign Language is a waste of time because no one who speaks Exotic Foreign Language ever says anything of any relevance to policy debates anyway?  Would you perhaps think to yourself "How the hell do you know?"

As you can probably guess, I'm using Exotic Foreign Language as a metaphor for formal theory.

To those who say, "I don't speak this language, you are of no use to me if you can't learn to speak plain English," I say, "That's reasonable."  Maybe I have some high-minded notion that it is impossible to translate from Exotic Foreign Language to English without losing some of the nuance in meaning, or without turning three words into sixty, or a short policy brief into a doorstopper that I know you won't read.  But, sure, I'm still of no use to you.

To those who say, "Speakers of Exotic Foreign Language don't have anything relevant to say", who discourage future graduate students from learning formal theory for this reason, I say, "EAT FECAL MATTER AND CHOKE ON IT."  Forgive me for being crude.  I obviously don't really mean that.  But, in all honesty, I see no reason to abide those whose fear of math -- which seems to be the real issue here -- leads them to make such grand assertions about the inability of formal theory to so much as speak to important questions of the day.  The fact that you don't understand what formal theorists have to say does not mean that they in fact lack something to say.  It does mean that there is unlikely to be a constructive dialogue, as I just acknowledged in the previous paragraph, but you know what would help with that a great freaking deal?  If we didn't tell college students that it would be a waste of their time to learn formal theory because formal theorists have nothing to say.

Now, I don't actually think it impossible to translate arguments originally expressed in the language of formal theory into ordinary English while retaining most of the nuance.  It's very challenging, but it can be done.  I've even found that math-phobic undergrads can be made to do some low-level translation over the course of a single semester.  Getting them to do so often makes me feel more like a therapist engaged in immersion therapy than an educator, and it took me a long time to realize that the primary obstacle between them and the goals I wanted them to achieve was not primarily cognitive but largely emotional.  But still.  It can be done.

In the classroom.

I'll grant that the policy crowd is not interested in overcoming any phobias and learning new languages just so they can hear what one of three experts has to say.  Fair enough.  Formal theorists can and should devote more effort to translating their arguments into English as best they can, even if that means losing some of the nuance (which it inevitably does).  But I don't think I can possibly overstate the extent to which I find it frustrating when I hear people say that formal theorists have nothing "relevant" to say.

This is particularly frustrating because, so far as I can tell, a great many of those who are concerned with being "relevant" are comfortable emulating the behavior of expert 2.  But maybe I'm just being cranky.  Perhaps those political scientists who think that our greatest failure as a discipline is the relative infrequency with which politicians cite our work are well aware that just because our voice are more likely to be heard when we unequivocally endorse or condemn policies doesn't mean that we should be doing so when we can't back those positions up.  That it is grossly unfair to assume that people who engage in esoteric debates about research methodology don't care about policy, unless of course one assumes that "caring" about policy does not involve caring about getting the right policy.  Maybe it's wrong of me to assume that many of those who cry out for more relevant work crave recognition and influence more than they do good policy, just as it is perhaps too cynical of me to think that policy makers are interested in retaining office more than they are good policy.  I don't know.  But one thing I know for sure is that people who confidently assert that formal theorists have nothing relevant to say don't know what they're talking about.

How about some quick examples?

We have a major candidate running for office whose primary foreign policy prescription is to assemble a military "a military so strong, no one wants to test it" indicating that he assumes that the primary determinant of war onset is whether the challenger expects to prevail or not.  In the current politicized environment, all my progressive readers are probably already saying to themselves, "Sure, but no one really believes that except conservative hacks anyway."  Maybe.  But a lot of academics have either asserted that states do not start wars they expect to lose, or consider it profoundly puzzling that they sometimes do so (which they do, fairly often in fact.)\(^1\)  I know that I myself once thought it self-evident that states don't start wars they expect to lose.  The reason why I no longer think so is because I've spent time studying formal models of crisis bargaining.

Consider the following standard crisis bargaining model.  Some challenger, C, proposes a division of some disputed benefits, denoted \(x\), to some defender, D.  If D accepts, the benefits are divided accordingly, with C receiving \(x\) and D receiving the remainder (i.e., \(1-x\).)  If D rejects, the two fight a war.  In that war, C expects to acquire \(w\) of the good.  Each side will suffer some loss of utility when incurring the costs of war, leaving C with a net payoff of \(w - c_C\) and D with \(1 - w - c_D\).

How does the size of \(w\) affect the likelihood that they fight a war rather than reaching an agreement?

It doesn't.

At least, not if we assume complete information.  Or if we assume incomplete information over \(c_D\).

If we assume that C does not know how the war will end, well, that's a different matter.  Suppose that \(w = \displaystyle \frac{e_Cm_C}{e_Cm_C + e_Dm_D}\), where \(e_C\) denotes C's martial effectiveness, \(m_C\) denotes C's material capabilities, and \(e_D\) and \(m_D\) denote the same for D.  Suppose further that C does not know the size of \(e_D\), though she knows the size of \(m_D\).  Let's keep things simple and assume that she also knows that \(e_D = \underline{e}_D\) with probability \(\phi\) and \(e_D = \overline{e}_D\)  with probability \(1-\phi\), where \(\underline{e}_D < \overline{e}_D\).  That is, C knows that there is some probability that D is relatively good at the conduct of war and some probability that he is not, and there are no other possibilities to consider.  Then C expects to acquire \(\overline{w}\equiv \displaystyle \frac{e_Cm_C}{e_Cm_C + \underline{e}_Dm_D}\) of the benefits with probability \(\phi\) and \(\underline{w}\equiv \displaystyle \frac{e_Cm_C}{e_Cm_C + \overline{e}_Dm_D}\) of the benefits with probability \(1-\phi\), where \(\underline{w} < \overline{w}\).

I'll skip most of the details.  But I can prove (and often have done so; see here for example) that C will propose terms that D accepts for certain when \(\phi \leq \hat{\phi}\), where \(\hat{\phi} = \displaystyle \frac{c_C + c_D}{\overline{w} - \underline{w} + c_C + c_D}\), and chooses terms that D will accept if and only if D is relatively low in martial effectiveness when \(\phi > \hat{\phi}\).  And since \(\hat{\phi}\) decreases when the difference between \(\overline{w}\) and \(\underline{w}\) increases, meaning that C will find it worthwhile to risk war under ever more permissive conditions, we can say that war is more likely when \(\underline{w}\) and \(\overline{w}\) are far apart.  The distance between \(\overline{w}\) and \(\underline{w}\) is greatest when \(m_C = m_D\).  That is, war is most likely when C is maximally uncertain about the outcome, not when C is very sure that C will win.  If D is currently stronger than C, becoming stronger still will discourage war.  But if D is currently weaker, becoming stronger will promote war.

Okay, that's probably too much Exotic Foreign Language for most of you.  Allow me to translate into English.

Suppose you owe me $5.  I offer to let you out of your debt through a bet, which we'll settle with a roll of a fair die.  If it comes up six, I'll pay you $600.  If it comes up anything else, you owe me $6.  You want to take the bet, don't you?

What if I told you that you don't owe me anything, that I owe you $500.  For whatever ridiculous reason, we again agree that, if you so desire, we can roll a die, and I'll pay you $600 if it comes up 6, but you'll owe me $6 if it comes up anything else.  You're no longer very interested in taking the bet are you?

You know what didn't change?  Your chance of winning.

It is absurd to think that states make decisions over war and peace without paying any attention whatsoever to what they can get without fighting.  To assume that war only occurs when the initiator expects to win is to assume precisely this.  To assume that there is such a thing as a level of military strength that would ensure that no one would ever dare test the US is to assume precisely this. If, instead, we assume that states gamble when there is a lot to gain and/or little to lose, but do not gamble when there is little to gain and/or a lot to lose, we would expect the likelihood of war to depend, among other things, on much uncertainty there is over the likely outcome of war.  That is, when challengers expect almost exactly the same outcome from a war against a defender who is relatively high in martial effectiveness as they do one who is relatively low in martial effectiveness, it doesn't really make sense for them to propose terms that would only be accepted if the defender is low in martial effectiveness.  That would be like turning down $500 for sure in hopes of getting lucky enough to win $600, despite knowing that there's a very real chance that you'll end up in the red.

And, it just so happens that the historical record provides strong evidence that wars have occurred more frequently when the two sides were roughly equal in material capabilities, a condition that exacerbates uncertainty over the likely outcome of war.

The simple model above tells us that wars are actually less likely to occur when the challenger can be extremely confident that they will win, and that challengers might well propose terms that they believe are more likely to be rejected than accepted.  Those are not trivial conclusions.  Very smart people have been known to express arguments fundamentally at odds with these claims.

Granted, we didn't need a formal model to figure any of this out.  Folks have been arguing that parity promotes war for a long, long time.  That people will gamble when the odds that their bet pays off are less than 50% is hardly a revelation.  I could, and under different circumstances would, argue that there is some value to identifying the precise logic by which such results emerge, but forget that.  The incredibly simple model I very briefly discussed above is more useful for foundational purposes than anything anyway.  A thorough understanding of it prepares one to study more complicated models.  And those models often produce truly surprising implications, even when starting from relatively intuitive assumptions.  In my own work, I have proven that relatively straightforward assumptions might lead us to conclude that even if war is caused by uncertainty, gathering accurate intelligence can make war more likely, and this is especially true as the quality of that intelligence improves; that poverty might promote popular participation in those wars that do occur, but decreases the likelihood of civil war onset in the first place; and that it is precisely because electoral competition induces accountability that democratic states are prone to, at times, fighting prolonged, politically divisive wars that they have no hope of winning.

There's a lot of criticisms you can make of these papers, not least of which being that my writing is atrocious.  But one thing you can't say is that the arguments they advance lack relevance to real world questions of policy.  Whether better intelligence is always and everywhere a force for peace, whether alleviating poverty strictly reduces the likelihood of war, and whether democratic institutions induce more careful policy making are deeply important questions of policy.  They also happen to be ones where a strong consensus holds that the precise opposite of what I have proven should be the case.

So the next time you feel the temptation to pronounce authoritatively that formal theory lacks policy relevance, stop and ask yourself -- what is your basis for saying that?  Aren't you really saying "Eww, it's got, like, math and stuff"?  Are you really saying "even if I could speak the language, I just know that I'd get nothing out of it"?  If you are saying that, do you have any real basis for saying so?

Finally, look down your nose at me for all too often allowing myself to be expert 1 if you must, but for the love all that is good and holy, don't let yourself be expert 2.  The world does not need more of that.

1. Focusing only in interstate wars (data taken from the Correlates of War project), which initiators win far more often than irregular was such as civil wars and extra-state wars, initiators have won roughly 64% of bilateral wars between 1816 and 2007.  Not all of the 36% of cases where the initiator lost are cases where the initiator should have known that they would lose, but a very substantial number of them are.


  1. "that poverty might promote popular participation in those wars that do occur, but decreases the likelihood of civil war onset in the first place"

    I looked at the abstract for this paper and maybe my neurons (such as they are) are not firing today, but I didn't really get from the abstract why the paper is arguing what it is arguing. Specifically, the last sentence of the abstract says the results indicate that poverty and low state capacity may reduce the likelihood of rebellion/civil war, but the preceding sentences don't seem to explain why (at least as I read them).

    Also, does the paper address whatever (non-formal) lit. argues that poverty makes civil war onset more likely? If not, don't you think it might be worth some day writing a paper that analyzes a model, gets the result, but then also proceeds to show that the empirical (or other) lit/evidence that cuts vs. that result is wrong for reasons X, Y and Z? I realize this type of paper is prob not written v. often; at least my impression is that most papers that analyze models do that and then pretty much stop. But perhaps that is a wrong or unfair characterization...

    1. Hey LFC.

      You're right, the abstract doesn't explain it. I probably should have found a way to fit that in. (150 word limits are tough.)

      Basically, conditions under which the public is expected to support the rebels give the government reason to fear war more than they give the rebels reason to demand more from the government. The reason for this is that the government expects to lose a lot more when the rebels have popular support, but the rebels only expect to gain a little bit more. Why the difference? Because some of the additional spoils that will be wrested from the government will be given to the public. A we move from conditions where the rebels would go it alone to conditions where they'd have popular support the government's willingness to offer concessions increases faster than the rebels' willingness to reject demands, and this leads the overall likelihood of peace to increase. Poverty and low state capacity are both factors that increase the likelihood that the rebels would have the support of the population in the event of a conflict.

      In the current version of the paper, no, we don't devote much attention to empirical evidence. In a previous version we did, but we ran into some problems in the revisions. Basically, there is no good measure of popular support for the rebels during civil war, and simulation results tell me that absent a reliable measure of this, I wouldn't expect to find support for the model's implications even if the model was exactly right. So I can't show robust empirical evidence that conflict (of some form) is more likely in rich countries with high state capacity, while those conflicts that do occur in poorer and lower capacity states are more likely to have popular support. Some of the statistical models using very crude proxies support this, some do not. If I thought the fact that we lack robust evidence in support of our claim was dispositive, I'd report it. But since we'd expect to observe exactly what we do observe given the measurement problem we face, I decided to just omit that discussion from the paper entirely. I may add some discussion of it back in if the paper is rejected though.

  2. Thanks for the reply.

    I want to focus briefly on this part of it:

    "The reason for this is that the government expects to lose a lot more when the rebels have popular support, but the rebels only expect to gain a little bit more. Why the difference? Because some of the additional spoils that will be wrested from the government will be given to the public. As we move from conditions where the rebels would go it alone to conditions where they'd have popular support the government's willingness to offer concessions increases faster than the rebels' willingness to reject demands, and this leads the overall likelihood of peace to increase."

    This assumes of course that the more poverty there is, the more popular support for the rebels there will be. Seems like a reasonable assumption, or reasonable enough to grant for the sake of argument.

    However there are 2 other assumptions here which seem to me more open to question: first, that the rebels are consciously thinking in terms of division of 'spoils' or gains betw themselves and 'the public', and second and more fundamentally, that rebels and govt engage in a process of bargaining to begin with.

    In some civil wars, though, there probably is no bargaining at all and no real possibility of bargaining. E.g. Libya: what cd Gaddafi have offered that wd have satisfied the rebels except to completely relinquish power? E.g. Syria: prob. same thing w/r/t Assad, though there might have been a chance for bargaining at the beginning. E.g. Somalia: the recognized govt (the one supported by African Union forces) may not have bargained at all with al-Shabab
    and other rebels (though I'm not sure on this). There was likely nothing to bargain about b.c the issue basically is who controls the country, not 'spoils', and the govt can't grant any spoils anyway since it prob doesn't -- or didn't at the relevant time -- have effective control over much. (It may have more now, as al-Shabab has apparently been driven out of its last urban foothold, or so I heard on the news a few days ago.)

    So, if I'm understanding the argument (and admittedly w/o having read the paper), the model's applicability would seem limited to a subset of civil wars, namely those where is some possibility of bargaining. Which is still possibly useful but does suggest an arguably rather important limitation.

    1. Excellent points. Those are indeed critical assumptions.

      Let me start by saying that I agree completely that the model only applies to a subset of cases. I will try to convince you below that this subset is perhaps not *quite* so small as it may seem, but you're absolutely right that there are some cases to which the model clearly does not apply.

      There are two reasons I think the subset of cases to which the model *does* apply is broader than it may initially seem. First, "bargaining" can be implicit. When a government changes its policies unilaterally in hopes of discouraging rebellion, that is not the same as sitting down at the negotiating table with the leadership of a dissident group and hashing out terms. But, in *some* respects, it might as well be. The "bargaining" that takes places in bargaining models really only deals with strategic decisions over the allocation of things that people want. It's mostly an aesthetic choice to refer to this as "bargaining." This doesn't speak to the examples you provided, but I just wanted to clarify the point because there are lots of cases where we do not see negotiation but where I would still say that the "bargaining" envisioned by these types of models still took place.

      Second, and more importantly, the decision to attempt to topple a government through force of arms is largely endogenous to the war process itself. That is, groups who might have chosen not to fight at all if they had been granted greater autonomy, or offered a power-sharing agreement, or who were provided greater government subsidies for housing or food, might ultimately seek to overthrow the government once they find themselves in a war that they are winning. I'm not at all sure that the rebel groups in Benghazi would have marched on Tripoli with the goal of overthrowing Qaddafi had he never cracked down on protests, or had he announced government subsidies that would have lowered the effective cost of food. Ditto Syria. I don't know that for sure, of course. And even if what I say is true, that doesn't mean that I'm arguing that Qaddafi should have known that the war would occur and that it would end the way it did. I view the decision to suppress protests rather than trying to buy the people off (which is precisely what Saudi Arabia and other Gulf countries did -- see as a form of gambling. But regardless of whether Qaddafi made a grievous error or a calculated decision that just so happened to end poorly for him, either way., I'm reluctant to say that there was no possibility of bargaining in Libya.

      Note too that there are many examples of groups that have sought independence, and claimed that they would not settle for anything less, who essentially gave up violence after they were granted regional autonomy. Consider the ETA and the Basque territories or the IRA and Northern Ireland. The frequency with which groups abandon violence after achieving outcomes that fall short of their espoused goals makes me reluctant to accept the claim that there is nothing to bargain over. In politics, actors often claim that no middle ground is possible when in fact it is.

      However, as I said above, I will readily acknowledge that the model simply does not apply to some cases. Somalia might well be one of them. As you rightly note, the government didn't have control of much, so they couldn't really promise to give much away.

  3. Thanks (again) for the reply. I take the point that the subset of applicable cases may be larger than it at first seems to be. I'm not at my home computer right now, so for the time being I'll leave it at that.