‘Normative audit’ as a method for political philosophy

by Ingrid Robeyns on April 3, 2020

So here’s an idea I’d like to float in our virtual common room. In the last 10-20 years, there has been a lot of discussion in contemporary normative political philosophers on methods. Yet in my view, methods are always depending on the aim/goal/function of the analysis one wants to do. So, what is the work we are doing aiming at, exactly? The work of that part of political philosophy does different things, including spelling out what normative notions mean (conceptual work), developing theories of certain values (e.g. a theory of justice, a theory of freedom as non-domination), developing theories on particular problems (e.g. a theory arguing for open borders), and of course, many of us spend a lot of our energies showing that certain arguments other philosophers advance in pursuing the above research agenda’s are wrong, or have shortcomings and how these could be fixed; or whether the many views and reasons advanced in this kind of research are philosophically distinct, i.e. whether they cannot be reduced to a more fundamental reason given for a certain view.

But one could also do philosophical work in this tradition whereby one is, as a philosopher, firstly, not explicitly introducing one’s own values to this debate; yet, secondly, one is nevertheless trying to provide a constructive input for politics (public policy making and the democratic debate), and, thirdly, one is not coming up with a specific recommendation for a policy or an institutional change, but instead making some less specific recommendations.

We could call this method a ‘normative audit’ (or ‘ethical audit’ might also work, but let’s proceed with one name). How would this work?

For a specific institution or problem, the philosopher’s task would be to collect and discuss all the arguments that have been advanced arguing for (a specific design of) that institution, or a specific policy/political treatment of that problem. The philosopher would have to try to be complete, that is, bring all relevant considerations to the table. Ideally, the philosopher would be creative and smart enough to see whether important arguments have been lacking, and would be able to add them to the discussion. (The possible relevance of standpoint epistemology comes in here, but I’ll bracket that for now).

For example, typical arguments would be that institution A would increase our wellbeing and our freedoms more than institution B, yet that A scores lower with respect to the value of personal autonomy than B. One important task would be to spell out under which conceptualisations of those values/normative notions these claims are true, and how the arguments would change if one were to opt for different conceptualisations.

This type of work would allow the philosopher conducting the normative audit to deliver the following:
(i) Weeding out bad arguments, which could be advanced either in public debate or in scholarship. The arguments would be bad if they are based on logical errors, e.g. inconsistencies, or empirical premises that are false.
(ii) Spelling out possible inconsistencies between different parts of the institutional design. For example, institutional feature 1 (e.g. a tax deduction) could only be based on principle X, yet principle X would also imply institutional feature 2, which is currently lacking in the overall institutional design.
(iii) Laying out a range of possible options open to the citizens, politicians and policy makers, with a clear explanation how, if one weighs value/principle X more than value/principle Y, this would favour option 1, versus how a different weighing of values would favour another option.

This normative audit would essentially weed out bad arguments, reduce the range of possible options that can be justified, and give a map to those who have to make choices (as voters, citizens, politicians, policy makers or otherwise) to clearly see how different normative concerns (values, principles) relate to different options.

The normative audit method is very much in line with a method that has been used for a long time in normative political philosophy, but where the overall goal was to defend a certain position, that is, defend option 1 or 2. Many of us have used it; for example, about ten years ago Anca Gheaus and I used it in a joint paper in which we show how the values of gender equality, the pursuit of individual plans and goals, and the good of parental care (especially for the wellbeing of children) can be balanced in different ways in the design of parental leave schemes, and that the scheme we propose balances these values in the way in which we think best balances those three values. Our own normative position is clearly present, since several of these values are either contested or one might not care much about them. Moreover, we say nothing (and hence can be read as caring less) about other values, including economic production efficiency.

Jonathan Wolff has recently spelled out this method in which the goal is to come to a concrete normative recommendation in his chapter in the Routledge Handbook of Ethics and Public Policy; and it is, in my view, very similar (perhaps even the same, I am not sure) to the Rawlsian method of reflective equilibrium. But both reflective equilibrium and the method that Wolff describe, are for analyses that aim to come to a decisive conclusion about what to do/choose. The normative audit method is much less prescriptive, in that its aim is rather to support others (citizens, politicians, policy makers) to make choices that are better informed about the normative implications, and about the soundness of different options. In the normative audit method, ideally the philosophers should not let their own normative views play a role (although I believe that there will always be some personal values involved, if only by the choice to work on topic T rather than topic S.)

I’ve used the ‘normative audit’ method two-and-a-half years ago in a Dutch paper with my colleague Sem de Maagt in which we critically analyse Dutch inheritance taxation; we don’t defend a particular scheme, but rather show all the values at stake, as well as point out some internal inconsistencies and other aspects which we think cannot be justified, of the current Dutch legislation.

Now, as does happen once in a while in academic common rooms, I am ready for one of you to tell me that this ‘normative audit’ method has been spelled out in great detailed by a philosopher (or social scientist!) twenty years ago. If so, I’d love to hear it.



John Quiggin 04.04.20 at 3:49 am

Interesting. This has echoes of the positive-normative distinction adopted by a lot of mainstream economists. The typical version of the claim is that a sound economic analysis lays out the choices and trade-offs, independent of the preferences of the economist doing the work.

That distinction has been the subject of a lot of criticism, which I assume you are familiar with. So, it would be interesting to consider whether those criticisms apply to the normative audit idea, and whether there are good responses.


Max Heermann 04.04.20 at 10:15 am

Very nice outline of this method, Ingrid! Good to see it spelled out like that. I would argue, though, that a “normative audit” is well suited to also make mare specific recommendations. Why do you think this is not/should not be the case?

In political science, David Beetham has developed what he calls a “democratic audit” in the 90s and set up the think tank(?) “UK Democratic Audit” now located at LSE I believe. The democratic audit as a method/approach has also been applied to the European Union (Chris Lord 2012). Now for a bit of shameless self-promotion: In a recent paper, Dirk Leuffen and myself have “audited” European Parliament decsion-making rules in the context of a differentiated EU (https://doi.org/10.1111/jcms.13015).


Ingrid Robeyns 04.04.20 at 12:38 pm

John, thanks, that is an interesting comparison. I think it is true that the similarity is that this normative audit method starts from the claim that (when using this method!) the philosopher should lay out the choices and value-trade-offs, independent of her own explicit political values; however, a very important qualification is that there is no claim in the normative audit method that it follows that ‘sound philosophical analysis must follow this method’. Rather, the view would be that political philosophers (as a group) can and must do many different things, and there may be circumstances in which their own values should not play an explicit role, and hence they should stop short of making the explicitly prescriptive-normative recommendations that most political philosophers (including me) typically make.

So I would think that a main difference between the mainstream economic view you refer to, and the view I hold, is that mainstream economists believe that they cannot and should never be prescriptive, whereas I think that political philosophers can and should be prescriptive in some circumstances, and can and should not be explicitly prescriptive in other circumstances (in the latter case they are not prescriptive, but still normative in a weaker sense, e.g. by choosing the topic of their research). Hence, I am a method-pluralist, and think the first question a scholar should ask is what type of knowledge they are producing, for whom, in which (social, political, historical etc.) context, and then, derived from this, ask which method they need. I think that’s very different from what mainstream economists who endorse the positive-normative distinction believe.


Ingrid Robeyns 04.04.20 at 12:46 pm

Max – thanks. SO I think it depends on what we mean with ‘specific recommendations’. It is true that the normative audit method can and does make specific recommendations; for example, when Sem de Maagt and I used it to analyse Dutch inheritance taxation leglisation, we argued that the different tax rates between children, other relatives, and non-relatives cannot be justified, so we argue that shoudl be abolished. But we didn’t argue for, say, a change in the structure of inheritance taxation, the types and levels of exemptions, etc – so we didn’t come up with a concrete institutional design. We tried to show that this depends in part how much weight one attaches to different values, such as ‘freedom’ and ‘constraining inequalities’.

Thanks also for the reference to your paper and to the ‘democratic audit’ model more generally. Is that method the same to what I lay out in the OP?


Max 04.04.20 at 1:50 pm

As far as I can tell, your depiction of the method fits very well with my understanding of the “democratic audit” approach but I’m not sure the latter has been spelled out in such a form. Beetham himself (and I think the UK Democratic Audit folks) was also very much occupied with measurments and operationalization.

I’d say that the “democratic deficit” debate in EU studies (including in constitutional law) implicitly applies something that reflects the method as proposed by you. This debate has also always been about institutional design, ie making proposals on how to strengthen the EU’s democratic credentials. Some authors propose solutions modelled on nation state parliamentary democracy, others argue that this is a misnormer and discuss how principles from international law/relations should be weighted against criteria for (national) democracy (e.g. the demoi-cracy literature). Quite a number of authors active in the debate, including myself, have more of an empiricist background/training, which might explain why these approaches are often not as formalized methodologically.

“But we didn’t argue for, say, a change in the structure of inheritance taxation, the types and levels of exemptions, etc – so we didn’t come up with a concrete institutional design. We tried to show that this depends in part how much weight one attaches to different values, such as ‘freedom’ and ‘constraining inequalities’.”
In our paper we show that a trade-off between to democratic principles – autonomy and equality – if not all EU member states take part in a common policy. We discuss how different reform proposals weigh these principles and in the end declare our own support for one of the proposals which in our judgement minimizes the trade-off.


Jonathan Wolff 04.05.20 at 8:00 am

Hi Ingrid. Thanks so much for the mention. First I should make clear that I don’t think what I call ‘Engaged Philosophy’ is a new method. Rather it’s a name and a spelling out in detail of a method that many people in applied moral and political philosophy have been using implicitly for decades, maybe even centuries.

There are no doubt affinities with reflective equilibrium but the aims and context are very different. For Rawls reflective equilibrium is a method to come to a determinate, consistent, account of justice. The output for him is his two principles of justice.

Engaged philosophy, as I understand it, is a way of approaching particular pressing questions in moral and political life (for example how to allocate scarce intensive care beds in a pandemic). The contrast I have in mind is with a sort of textbook caricature of applied moral philosophy where a theorist approaches a problem with a philosophical theory in hand and tries to solve the problem by working out the consequences of their theory for the subject matter before us.

Instead I say we should take a detailed look at the problem and the conflicts of values involved. Look at the current situation and how we got into it. See what other countries do. Look in detail at the work of specialists and activists in the area, both for their description of the facts and their moral reasoning. Construct a menu of policy options. Consider the likely effects of introducing them, including unintended consequences, and whether the public and officials would actually follow new policy or try to resist. See which options are possible from where we are, and best supported by values, using whatever philosophical resources we have at our disposal. Make policy recommendations, rooted in values, that will constitute an improvement while being open to the fact that further improvements may later become available.

This last point draws out a possible ambiguity in the term ‘decisive outcome’ you use. Yes, a decision is needed (otherwise we stay where we are, which is a decision by default too). But it is not decisive in the sense of ‘once and for all’. In my view it is provisional, experimental, and can be changed or reversed.

One similarity with reflective equilibrium is that going through this process can show us that currently dominant theories in political philosophy are inadequate. I found this in relation to disability where much philosophical work of the 70s and 80s simply lacked the concepts needed to make a worthwhile contribution to active debates.

One obvious differences is that there is no attempt here to come to a theoretical equilibrium. Values may well remain in tension. The point is to do the best we can in a practical sense rather than to cleanse our theoretical ideas as a goal in itself (even though there can be a theoretical pay off).

If I understand it, your own audit methodology is very similar to engaged philosophy, but draws back from making any actual recommendation. I’m not sure how much attention it gives to the empirical, historical, and comparative elements but knowing your work, and Anca’s, no doubt it is very inclusive.

I’ve spoken on this topic in several countries and the most common response is ‘but this is what we already do.’ I’m happy to hear this. All I’ve done is provide a framework so that people can be more reflective about their methodology.


Adam Swift 04.05.20 at 2:15 pm

At one point Erik Olin Wright, Harry and I had a plan to write a book called Capitalism: A Moral Audit. The idea was to identify a number of different values – wellbeing, liberty, equality, democracy, community etc – and say something about how capitalism fared with respect to each. This was before Erik’s magisterial Envisioning Real Utopias, chapter 3 of which did some of this under the title ‘What’s So Bad About Capitalism?’.

That project would have involved more judgments than what you have in mind, which makes me wonder whether ‘audit’ is the right term for you. If I’m right that a company is audited when outsiders come along and check whether it really is worth what it says it’s worth, or that its dealings have been properly conducted, then that sounds more substantially evaluative than what you’re talking about.

Of course, had we proceeded with that project we would immediately have confronted the fact that each of those values is subject to competing interpretations. So the book would inevitably have had to identify and discuss the various different things people might be valuing under those labels (and the level of detail at which it did that would have depended on its intended readership). So we might well have ended up not offering judgments at all but more modestly clarifying the different considerations – conceptual, normative and empirical – that are inevitably involved in making such judgments.

That, in effect, is what Harry and I, together with Helen F. Ladd and Susanna Loeb, did in our book Educational Goods: Values, Evidence and Decision-Making. (There is a journal article summary called ‘Educational Goods and Values: A Framework for Decision Makers’ in Theory and Research in Education). Of course the subject is much more modest than anything like ‘capitalism’, but the aim there was precisely to provide readers with the resources needed to evaluate policy options in the sphere of education. So we offer a list of different ‘goods’ that are distinctive to that sphere, explain the different distributive ideals at stake, and identify a range of other, non-educational, values that also need to be taken into account when making, or assessing, decisions. (We mainly talk about collective policy decisions, but in principle the same considerations will be relevant to individuals.) We deliberately don’t offer any determinate conclusions about any of the issues we discuss (school finance, school autonomy and choice, school accountability). So, like what you have in mind, the idea is not to argue for any particular view but rather to help readers see the range of issues at stake, the kind of weighing judgments that will have to engage in, and the relevance of empirical evidence from the social sciences, so that they can make their own judgments.

Like Jo, we don’t see ourselves as suggesting anything new or radical. Whether they like it or not, and whether they know it or not, the ‘method’ we propose is inevitably being used by anybody offering an all-things-considered critique, or defence, of any policy. As he says, ‘all we’ve done is provide a framework so that people can be more reflective about their methodology’.

I wouldn’t claim that we have ‘spelled out in great detail’ the method you propose, but from what you’ve said here I do think it might be helpful to look at our approach, if only to clarify where the two differ.

PS I would insert links to the things referred to but I don’t know how. Sorry.


Ingrid Robeyns 04.06.20 at 8:18 pm

Thanks Jo and Adam, for very interesting and valuable discussion!

I had read Jo’s Chapter and account of ‘engaged philosophy’ also as the spelling out of what many people do, and I am not surprised that Adam makes the same claim (I need to read that work Adam!), and I should of course have made the same disclaimer. Still, I think there is a lot of value in spelling out the methods that we’ve been trained with, since, in my view, normative political philosophers have been rather weak in spelling that out, and this has in fact probably harmed our possibility to communicate how we do what we do. (I wrote a blogpost about this in 2007 [ https://crookedtimber.org/2007/09/04/methods-in-political-theoryphilosophy-bleg/%5D, and I think we have made progress in the meantime, thanks to chapters such as yours!

Adam (or Harry?) – I know it’s off-topic, but can I ask you a question that’s also been on my mind for a long time: Why do you and others use the term ‘good’ rather than ‘value’? Do they refer to the same, and hence are they synonyms, and if not, how do they differ?


Tom 04.07.20 at 12:57 am

John @1: “That distinction has been the subject of a lot of criticism, which I assume you are familiar with.”

I am familiar with the distinction positive/normative in economics but what are the key references for its criticism? Are they mainly from philosophers or also economists? I would appreciate a few references if possible. Thanks anyway.

Ingrid, the idea of normative audit seems very good to me. In my experience it is actually crucial also to the teaching of philosophy itself. Indeed, I believe that the value of studying philosophy, at least as an undergraduate, is to be exposed constantly to a normative audit of your own beliefs and values (a bit like talking with Socrates, if you will).

Comments on this entry are closed.