Identifying Key Risks to the Effectiveness of Water Use Authorization Systems through Theory of Change (ToC): The Case of South Africa

Abstract

South Africa is a water-stressed country with significant water management challenges. In response, progressive and advanced water management policy and legislation have been developed that include the regulation of water uses through a water use license application (WULA) system. In recent years, the effective functioning of the WULA system has come under increased scrutiny. However, a comprehensive systematic evaluation of the effectiveness of the WULA system has not yet been conducted. This paper aims to identify key risks to the effectiveness of the WULA system in South Africa through the application of a theory of change (ToC) methodology. Workshops with more than 60 participants were held with different stakeholder groups. Ultimately, a total of 21 key risks to the effectiveness of the WULA system were identified. The results suggest that various risks have already been realized and that the WULA system is not functioning as intended and is not achieving its outcomes. It is recommended that the identified key risks provide the basis for a comprehensive system evaluation to inform a more effective WULA system design.

1. Introduction

Internationally, water is governed differently depending on the particular policy and legislative context governing water resources [1,2,3]. Within the South African context, water governance necessitates effective and responsive policy and legislation to deal with the implications of being one of the most water-stressed countries in the world [4]. To this end, the country has a long and rich history of forward-thinking and internationally acclaimed water management policy and legislation [5]. Following the first democratic elections in 1994, the country embarked on a comprehensive and far-reaching water policy reform process. This process culminated in what was at the time internationally considered some of the most progressive and advanced water management legislation in realizing basic human and environmental rights and guided by the principles of integrated water resource management [1,6]. Central to these reforms was the National Water Act No. 36 of 1998 (NWA) [7], which is the primary framework legislation that embodies several progressive approaches to water governance, including the regulation of different water uses [8,9,10]. The most significant of these water uses, such as taking water from a water resource, discharging water containing waste, and altering the bed, banks, course, or characteristics of a watercourse, are regulated through the so-called water use license application (WULA) system [9,11,12,13].

The WULA system has, however, been subject to scrutiny and criticism since its introduction, and anecdotal evidence suggests that it has been plagued with implementation challenges. These challenges include, for example, process inefficiencies, poor report quality, lack of data, weak enforcement, limited public participation, lack of skills and capacity, political interference in decision-making, and corruption. Consequently, the effectiveness of the WULA system has been questioned [14,15,16,17,18,19]. There have, however, been no attempts at evaluating the overall effectiveness of the system. Typically, effectiveness evaluation determines if a policy intervention works as intended or designed, and if it is achieving its policy objectives. The lack of effectiveness research is mainly due to the serious methodological challenges it presents.

Since the early 1990s, the theory of change (ToC) has been a popular method for evaluating complex systems, implementing change processes, and reflecting on strategies adopted to achieve certain outcomes [20,21]. Although the ToC approach to evaluation might battle with the oversimplification of the underlying assumptions and the causal linkages contained within complex systems [20,21], it is still considered a favorable approach to evaluation with its ability to produce an explicit theory of the system and the development of pathways which facilitate evaluation in the attempt to highlight and address complexity [22]. More recently, ToC has been widely applied for the evaluation of complex policy-based systems and the subsequent identification of risks to such systems functioning as intended [22,23,24,25,26,27,28,29]. For this reason, the South African government introduced ToC as the prescribed method to deal with policy and system evaluation [27,30].

ToC is described as a method that illustrates the causal links and sequences of events needed for a policy intervention to lead to a desired outcome or impact and articulates the assumptions underpinning its design. These assumptions also represent the risks to the effectiveness of the system and the extent to which it can be expected to work as intended or as designed [22,30]. Accordingly, this paper aims to identify key risks to the effectiveness of the WULA system in South Africa through the application of ToC, and in so doing further aims to inform future system evaluations and influence system design. The next section explains the ToC methodology, after which the results are discussed, and conclusions made.

2. Methods

2.1. Application of the Theory of Change (ToC) Approach

Two definitive approaches to the application of ToC approach to evaluation exist [24,30]. The first approach is to apply the ToC when designing a new policy-based implementation instrument to realize the specific policy objectives. This is achieved by involving different stakeholders and specialists through a process of information and knowledge sharing on how to best design the policy-based implementation instrument and the subsequent implementation, monitoring, and evaluation requirements [24,30]. The second approach in the application of ToC is then to evaluate an already existing policy-based implementation instrument, by unpacking the key underlying assumptions in the pursuit of highlighting the associated key risks to the instrument in achieving its objectives. In such specific cases, the policy-based implementation instrument has already been designed and the evaluation needs to deal with an ‘existing design’ and not a ‘preferred design’.

In this specific paper, the ‘design’ of the South African WULA system already exists and is mandated by policy and legislation, and therefore the second approach in the application of ToC is followed. Based on the outcome of the evaluation results, policymakers may consider a redesign or improvement to the existing design of the system by means of policy and legislative reform.

The ToC approach was therefore selected for this research due to its wide adoption as a policy evaluation method and its status as the preferred policy evaluation method in South Africa [30]. The next sections explain the systematic application of the ToC method and the particular six steps for the ToC development and validation process.

2.2. Results-Based Pyramid, ToC Map, and Causal Narrative

The ToC approach to evaluation produces a causal understanding of the existing design of the policy-based implementation instrument and is presented in the form of a ‘causal narrative’ and an illustrative ‘ToC map’ [30]. The causal narrative and ToC map are then developed against the following evaluation components namely, design, inputs, activities, outputs, outcomes (immediate and intermediate), and impacts [25,30,31,32,33]. These evaluation components are typically conceptualized and explained in the form of the so-called ‘results-based pyramid’ [30], shown in Figure 1. The results-based pyramid requires the evaluator to answer the following questions related to the system under evaluation: What is the underlying design (i.e., design component)? What do we use to perform the work (i.e., input component)? What do we do (i.e., activity component)? What do we produce (i.e., output component)? What do we wish to achieve (i.e., outcome component)? What do we aim to change (i.e., impact component) [30]? For the purpose of this research, the results-based pyramid was purposefully adapted for the evaluation of the WULA system. The pyramid illustrates the conceptualization of the evaluation components and guides the development of the ToC map and related causal narrative [23,28,34] by posing the following generic questions [25,32,33]:

• What is the legal mandate for the WULA system (design)?
• What inputs are required to implement the WULA system (inputs)?
• What actions are required to implement the WULA system (activities)?
• What outputs do the WULA system produce (outputs)?
• What immediate and intermediate outcomes do the WULA system deliver (outcomes)?
• What does the WULA system aim to achieve (impact)?

By systematically answering the results-based pyramid questions, an in-depth understanding is achieved of how the WULA system is expected to work and what it is expected to achieve. It provides an exploded view of the particular policy mechanism that can be translated into a ToC map (illustrating causal linkages between the evaluation components) and the ToC causal narrative (explaining these causal linkages and their key assumptions). The assumptions that underpin the ToC map and narrative are important because they provide the inner logic of the policy intervention design. If these assumptions are shown to be false or questionable, they present a risk to the policy intervention not working as intended and/or not achieving its intended results. The next section explains how the ToC map and causal narrative with the key assumptions were developed and verified.

2.3. Development and Validation of the ToC Map and Causal Narrative

There are different approaches recommended for the development of the ToC map and related causal narrative and assumptions [35]. These include mere desktop studies as well as specific reliance on specialist inputs. However, such approaches, although mostly technically accurate, sometimes lack verification and validation of the results by broader stakeholders. Therefore, it is recommended that, to obtain the best results, a range of stakeholders should be involved in developing the ToC map and narrative. The ToC map and causal narrative were also supported through the collection of descriptive qualitative data from existing literature and verified by the stakeholders involved in the workshops. Stakeholders involved in the development and validation of the ToC map and causal narrative were selected based on their experience, knowledge, and direct involvement in the South African WULA system. For the purpose of this research, the following six steps were followed in the development of the ToC map and causal narrative, with key assumptions and risks, to the effectiveness of the South African WULA system:

• Step 1—Inception workshop: The content of the initial ToC conceptual framework and ToC narrative was developed based on the understanding of the WULA system in South Africa and outputs from the workshop, involving internationally recognized researchers in the field of the ToC approach to evaluation, ecological water requirements, water law, as well as professional legal experts on the WULA system. The 12 researchers and professionals attending the inception workshop had more than 150 years combined experience in the water sector. The researchers and professionals applied the components (design, input, activity, output, outcome, impact) of the ‘results-based pyramid’ to the South African WULA system based on their experience and professional judgement and produced the first version of the ToC map and causal narrative with key assumptions.
• Step 2—Specialist/Consultant workshop: The ToC map and narrative developed during the inception workshop were presented to consultants and specialists directly involved in the WULA process. The purpose of the workshop was to obtain a water consultant and specialist perspective on the initial ToC map and narrative. In total, 16 consultants and specialists participated from across the country with a combined experience in the water sector of more than 200 years.
• Step 3—Applicant workshop: The ToC map and narrative were next presented to applicants directly involved in applying for WULAs. The purpose of this workshop was to obtain perspectives from the range of applicants seeking a water use license through the South African WULA system. A total of nine applicants from various sectors, including mining, industry, electricity generation, agriculture, forestry, and local government, participated in the workshop, each having been intimately involved in various WULA processes.
• Step 4—Regulator workshop: The ToC map and narrative were next presented to regulators directly or indirectly involved with reviewing and issuing of water use licenses. A total of 11 regulators from national and regional offices and catchment management agencies (including Gauteng, Mpumalanga, KwaZulu-Natal, North-West, Western Cape and Inkomati-uSuthu Catchment Management Agency (CMA), and Berg-Gouritz CMA) participated. The purpose of this workshop was to obtain a regulators perspective on the WULA system. The administrators and officials involved in the assessment, review, and decision-making process of the WULA system had a combined working experience of more than 90 years.
• Step 5—Public workshop: The ToC map and causal narrative were also presented to 13 members of public forums, including the MooiRiver Catchment Management Forum (CMF), uSuthu to Mhlathuze CMF, Upper Olifants CMF, and Berg River CMF. The purpose was to obtain a civil society perspective on the WULA system.
• Step 6—Finalization workshops: A finalization workshop was held to review and reflect on the comments and inputs from the key stakeholder workshops during steps 1 to 5 that included specialists, consultants, applicants, the public, and regulators. The ToC map and narrative with key assumptions presented in this paper are the results from these workshops.

Ultimately, the six-step approach ensures that the ToC map and related narrative and assumptions are accurate, agreed upon and validated by 61 individuals representing a broad spectrum of different stakeholders in the water sector. The process was only concluded once a saturation point was reached where more consultation and workshops would not reasonably produce any new insights or additional perspectives.

Prior informed consent was obtained from all key stakeholders who participated in the workshops. This research was subjected to ethics approval by the Ethics Committee of the Faculty of Natural and Agricultural Sciences (FNAS), North-West University (NWU-01661-20-A9).

3. Results—ToC Map, ToC Causal Narrative, and Underlying Assumptions

The research results deal with the ToC map, the ToC causal narrative, and the key underlying assumptions. The ToC map is presented in Figure 2 and is conceptualized based on the six evaluation components (design, input, activity, output, outcome, and impact) from the results-based pyramid in Figure 1. The ToC map should be read together with Figure 1, from left to right starting with the design and input components and ending with the impact components. Certain key assumptions (numbered 1–21) underpin the causal narrative. Table 1 provides a summary of the key assumptions and key risks to the effectiveness of the WULA system in relation to the different evaluation components. To summarize, the WULA system causal narrative supports the following statement:

“The South African WULA system is guided by policy and mandated trough legislation and regulations (design component), and requires skills and competencies, information, data, co-operative governance, time and money (input component) to administer and implement a prescribed process (activity component), which produces high quality information, communicated in technical and specialist reports (output component) to inform a licensing decision-making process (outcome component) for specific water uses towards the progressive realization of our environmental and water rights as stipulated in sections 24 and 27 of the Constitution (impact component)”.

3.1. Design Component

The design component of the WULA system refers to the design of the system [30] as guided by policy and prescribed in the NWA [7] and water use license application and appeals regulations [36] and further informed by published guideline documents [37,38]. The causal narrative for the design component is guided by question of the ToC results-based pyramid namely, “What is the legal mandate for the WULA system?”. It is only through a clear understanding of the design component of WULA system that the causal narrative between the input, activity, output, outcome (immediate and intermediate) and impact components can be understood [39].

A review of the existing literature on the water legislation in South Africa indicated a highly discriminatory approach to water use and water allocation under the apartheid regime; however, since the introduction of a democratic government attempts have been made to address these challenges with the introduction of new water governance approaches under the NWA [2,5,40,41]. The NWA is a framework act which gives effect to the ‘Fundamental principles and objectives for the new water law in South Africa’ and the White Paper on a National Water Policy (NWP) [42]. The ultimate objective of the NWA is the progressive realization of the environmental and water rights as stipulated in the Constitution [43] through reasonable legislative and other measures. The NWA has since introduced numerous policy-based implementation instruments to achieve these objectives such as the water use authorization (WUA) system. This system makes provision for permissible water uses including the use of water use by means of a license (i.e., WULA system). The NWA serves to create a regulatory framework under which water use licenses are required and sets out the process to be followed to apply for such a license. As of 2017, the WULA system design has been prescribed in the water use license application and appeals regulations [36] and is supplemented by guideline documents that aim to provide operational guidance and a standardized approach to the WULA and assessment process [37,38]. The South African WULA system also qualifies as an administrative action or decision-making process. This process is grounded on the fundamental human right to just administrative action [43] and implemented through the Promotion of Administrative Justice Act No. 3 of 2000 (PAJA) [44], which is based on the principles of lawfulness, procedural fairness, and reasonableness.

Ultimately, the legal mandate for the South African WULA system is well described and embedded in legislation. Therefore, the design component of the WULA system governs decision-making (NWA and regulations) related to environmental rights and water rights, based on a just administrative process [44]. It is from the design component of the WULA system that the causal narrative of the required input and activity components can be described and what is to be expected from the system in terms of the output, outcome, and impact components.

3.2. Input Component

The input component relates to the skills, competencies, infrastructure, co-operative governance, access to data and information, and time and money required for the WULA system to function effectively [25,31,33]. The causal narrative of the input component deals with the question, “What inputs are required to implement the WULA system?”. From the stakeholder workshops it was evident that skills and competencies are essential to make an informed decision, based on the information generated and provided. These required skills and competencies are associated with the seven main role players and entities required as ‘inputs’ to the WULA system and include, the landowner, the applicant, consultant, and specialists, public or civil society, administrators and officials from the responsible authority, and the Water Tribunal. In this case, the applicant is responsible for gathering and collating detailed information in support of the application, whilst the consultant is responsible for the application process and requires an in-depth understanding of the WULA process integrated with sound scientific knowledge and skills to generate the required technical reports necessary for decision-making. Specialists involved in the application process are necessary to undertake sector-specific scientific studies in support of the application. The skills and competencies of these individuals are regulated by the South African Council for Natural Scientific Professions (SACNASP) in terms of the Natural Scientific Professions Act No. 27 of 2003 [45]. Through consultation and participation civil society provide inputs based on experience and indigenous knowledge. The administrators/officials provide inputs through administrative review skills and competencies. These skills and competencies rely on a holistic and integrated understanding of the design and input components provided by the applicant/consultant, the specialist as well as the public/civil society.

The WULA system also requires reliable data, information, infrastructure, and established institutions for informed decision-making. One of the most important sets of information required as an ‘input’ comes in the form of the National Water Resource Strategy [46] which provides the framework for the protection, use, development, conservation, management, and control of water resources at a national, regional, and catchment level. The strategy is also essential for informing the development of catchment management strategies [47], which sets out the objectives, plans, and procedures for the protection and use water resources at a catchment level. Even though both these sets of information are developed at a strategic level, the importance of aligning source directed controls, such as water use licenses, with the objectives of the strategies cannot be underestimated [48,49]. The establishment of institutions for informed decision-making comes in the form of catchment management agencies (CMAs), which are responsible for the investigations and the dissemination of advice to interested persons on the protection, use, and management of water resources within the designated water management area. These agencies are established to realize the subsidiarity principle which focusses on decentralization and stakeholder engagement in decision-making processes within the determined area of jurisdiction (e.g., water management area) [50].

The classification of water resources, the setting of resource quality objectives and the determination of the Reserve have also been highlighted as crucial inputs to the WULA system. Classification of water resources is seen as the first stage in the protection of water resources and provides for guidelines and procedures for determining different classes of water resources [7]. The setting of resource quality objectives (RQOs) is fundamental to the classification of water resources and provides for numerical and descriptive statements related to the biological, chemical, and physical attributes of the water resource [51]. The Reserve consists of two aspects, namely the basic human needs and ecological reserve and provides for the essential needs of individuals as well as the water required for the protection of aquatic ecosystems [7]. A total of 25 L per person per day has been earmarked as sufficient for the basic human needs, and it is expected that this volume should increase as the standard of living becomes higher [42]. The ecological reserve relates to the water necessary for the protection of the aquatic ecosystems and should be determined for all individual water resources [52].

The WULA system also requires sufficient infrastructure, time, and money as inputs. Infrastructure necessary for the effective implementation of the WULA system is mainly associated with information technology platforms such as the water authorization and registration management system (WARMS) and the electronic water use license authorization application system (e-WULAAS). The WARMS has been established to capture the information of water users and registrations which is essential for water allocation and reallocation within catchments [9]. The e-WULAAS makes provision for the registration and submission of water use license applications and is intended to provide a seamless interface between the applicant and responsible authority for the management, coordination, finalization, and issuing of water use license applications.

The WULA system also relies on money as an input component. In this regard the direct financial cost burden relates to the applicant undertaking the said process needing to appoint a representative/consultant and the procurement of the specialists involved in the process. The financial input, in terms of administration, including review and decision-making, is borne by the responsible authority and relies on government funding.

From the above input component narrative, the following five key assumptions have been identified (numbered 1–5 in Figure 2):

• Assumption 1: Sufficient skills and competencies are in place to implement the WULA system.
• Assumption 2: Necessary infrastructure, communication, data, information are available, up to date, and adequate to support the WULA system.
• Assumption 3: Catchment Management Agencies have been established and are functioning.
• Assumption 4: Resource classification, resource quality objectives, and the Reserve have been determined.
• Assumption 5: Funds are available to support and sustain the WULA system

3.3. Activity Component

The activity component of the ToC is concerned with the actions needed to achieve specific outputs [25,31,33] and guided by the causal narrative, “What actions are required to implement the WULA system?”. The activity component of the South African WULA system is well aligned and similar to other international systems [1,3,53]. However, in recent years the activity component of the South African WULA system has been increasingly criticized due to procedural inefficiencies leading to backlogs of applications and questionable conditions included and/or excluded from licenses [16,53]. The ‘steps’ within the activity component of the WULA system include the following phases and are regulated by the water use license application and appeals regulations [36]:

• Phase 1 Pre-application enquiry: The pre-application enquiry is initiated by the applicant planning to apply for a water use license [7] with the intent of information dissemination between the applicant and responsible authority regarding the application.
• Phase 2: Application and information gathering: The applicant should gather and collate information in support of the application for informed decision-making by administrators/officials. The public consultation and participation process should also be initiated during this phase, contributing towards information gathering and involvement of all relevant stakeholders [7].
• Phase 3: Legal and technical assessment, evaluation, and input: The administrators/officials assess and review the application to determine completeness. If the application contains all the necessary information for decision-making, a technical assessment will be undertaken to verify whether all findings and recommendations are in line with relevant operational policies and strategies [7].
• Phase 4: Assessment review, recommendation, decision, and appeal: After the legal and technical assessment, recommendations, as well as a draft license with conditions, are submitted to the relevant authority for final consideration. After a decision has been reached on the water use license application, the decision and the license must be returned to the appropriate office, which then informs the applicant of the outcome as well as other interested and affected persons. The issuing of the license and any condition contained in the license may be appealed through the formal appeals process which is subjected to a review by the Water Tribunal [7].

The following 10 assumptions have been identified from the activity component narrative of the WULA system (numbered 6–15 in Figure 2):

• Assumption 6: All relevant water uses have been identified during the site inspection.
• Assumption 7: It is possible to agree on the requirements for the technical review.
• Assumption 8: The public is willing to participate and to do so in good faith.
• Assumption 9: Scientific/technical reports are valid, of high quality, and comprehensive.
• Assumption 10: Impacts on the water resource can be accurately predicted by means of an established method, criteria, and credible baseline information.
• Assumption 11: Administrators/officials read applications/scientific/technical reports.
• Assumption 12: Administrators/officials understand the content of the application/scientific/technical reports.
• Assumption 13: Administrators/officials are rational, impartial, unbiased, and objective during the review process.
• Assumption 14: Water use license applications are processed within the set time frames.
• Assumption 15: The Water Tribunal is objective and impartial.

3.4. Output Component

The output component narrative is guided by the following question, “What outputs do the WULA system produce?”. In this case, the activity component of the WULA system produces outputs [25,31,33] in the form of reports and documentation which are fundamental to decision-making. Reports submitted in support of an application may differ depending on the type of water use(s) applied for and can include the site inspection report, section 27 motivation statement stipulating the socio-economic benefits of the proposed water use, relevant technical and specialist reports, and the public participation report.

The output components of the WULA system are expected to be high-quality reports, containing adequate information to inform the licensing decision-making by the responsible authority. The following key assumptions are identified for the output component of the South African WULA system (numbered 16 and17 in Figure 2):

• Assumption 16: An effective and efficient process leads to high-quality reports.
• Assumption 17: High-quality reports lead to informed decisions.

3.5. Outcome Component

The outcome (immediate and intermediate) component of the WULA system refer what should be achieved by particular outputs [25,31,33] and is guided by the output narrative question, “What immediate and intermediate outcomes do the WULA system deliver?”. The main immediate outcome of the WULA system is a water use license, which authorizes the undertaking of a specific water use(s) applied for. The water use license is informed by the output components in the form of reports and based on a decision made by the responsible authority. Such a decision by the responsible authority should be governed and guided by principles and criteria as explicitly described in Sections 2 of the National Environmental Management Act No. 107 of 1998 (NEMA) [54] and the NWA [7], including relevant regulations in the water use license application and appeals regulations [36] and published guideline documents:

The abovementioned sections, regulations, and guideline documents guide decision-making and inform the content and conditions of the issued license. Ultimately, the decision-making process by the responsible authority is an administrative action and therefore regulated by a set of decision-making principles as stipulated in PAJA [44], including the requirement of lawful administrative action, which is procedurally fair and reasonable [52,55,56].

It is only through the implementation and compliance with the said conditions defined in the water use license that the intermediate outcomes are realized [48,49]. Several intermediate outcomes have been identified through the ToC workshops and were associated with ‘values’, ‘transparent decision-making’, ‘the realization of administrative justice’, and ‘capacity and knowledge building’. Essentially these identified intermediate outcomes are reflected in the Section 2 NEMA principles and in the purpose of the NWA. It is therefore argued that giving effect to the principles and purpose of the NEMA and NWA represent the intermediate outcome of the South African WULA system. Furthermore, the intermediate outcomes are realized through the implementation of catchment management strategies and achievement of the resource quality objectives, as well as the realization of the National Water Resource Strategy 2^nd edition (NWRS2), aimed at managing water efficiently and effectively to ensure equitable and sustainable growth and development.

The following key underlying assumptions are identified for the outcome component of the South African WULA system (numbered 18 to 20 in Figure 2):

• Assumption 18: Decisions are lawful, reasonable, and procedurally fair.
• Assumption 19: Water use licenses contain informed conditions to protect water resources.
• Assumption 20: Decisions are underpinned by decision-making principles.

3.6. Impact Component

Finally, the primary impact of the WULA system represents the results of achieving certain outcomes [25,31,33] and asks the question, “What does the WULA system aim to achieve?”. The aim of the WULA system is associated with the progressive realization of two basic human rights as contained in sections 24 and 27 of the Constitution [43], namely:

Section 24: “Everyone has the right:

(a)

to an environment that is not harmful to their health or well-being; and

(b)

to have the environment protected, for the benefit of present and future

generations, through reasonable legislative and other measures that—

(i)

prevent pollution and ecological degradation;

(ii)

promote conservation; and

(iii)

secure ecologically sustainable development and use of natural resources while promoting justifiable economic and social development.”

Section 27: “Everyone has the right to have access to … (b) sufficient water…”

It is understood that the impact of the WULA system would be to contribute to the progressive realization of these rights and that such impact is a long-term and ongoing aspiration rather than a clearly defined aim to be achieved. Therefore, the impact component sets the ultimate change the WULA system would like to achieve and/or the direction the system needs to move towards in terms of water governance.

One key underlying assumption has been identified as it relates to the impact component of the South African WULA system (number 21 in Figure 2):

• Assumption 21: Informed decisions regulating water use that are lawful, reasonable, and procedurally fair will lead to the progressive realization of sections 24 and 27 of the Constitution.

4. Discussion

A total of 21 assumptions are described in relation to the causal narrative illustrated in the ToC map (Figure 2). These assumptions underpin the design of the WULA system and ultimately its effective implementation. Therefore, these assumptions also represent key risks for the effective functioning of the WULA system, and for the system achieving its intended outcomes and impacts. Table 1 provides a summary of the 21 key assumptions translated into key to the WULA system. Moreover, the table also makes a link between the identified risks and the existing literature in the water sector.

Although the aim of this paper was to identify the key risks to the WULA system and not to evaluate each of the key risks (this is an area for future research), it is worth pointing out the existing literature that relates to these risks. This suggests that for 11 of the 21 key risks, there is a lack of literature, and therefore more research is needed in these areas. The water sector can, however, in attempting to understand these risks, benefit from research already conducted in the field of environmental impact assessment (EIA). For example, the seminal work by Holling [57] highlights that accuracy of prediction (risk 10) is very difficult to achieve and therefore adaptive management is a key requirement for any well-functioning regulatory system in achieving its intended objectives. Kornov and Thiesen [58] point out that decision-makers are not rational (risk 13) or objectively impartial, and that to influence assessment decisions, a better understanding is required of how decision-making works. Furthermore, the EIA literature has shown that although decisions might be administratively just and therefore lawful, reasonable, and procedurally fair (risk 18), they could still be wrong because they are based on weak or flawed report substance (risk 17) [34,55,56].

For the remaining 10 key risks, the literature does exist in the water sector that shows that skills and competencies (risks 1 and 15) are lacking [8,59,60] and is one of the main contributors constraining the effective implementation of the WULA system, leading to the backlog of applications and the issuing of water use licenses with substantive defects [9,12,16,46,61]. Infrastructure, communication, data, and information (risk 2) are also of concern [8,50,61,62,63,64]. Communication between the applicant and responsible authority is essential to ensure that WULA system is effectively implemented. Recent studies have indicated that communication between the applicant and responsible authority is especially problematic and leads to confusion in the steps to be followed during the application process [65].

Over the recent past, reports of outdated and incorrect information necessary for water allocation and reallocation are captured on electronic databases, such as the WARMS, and have seen the responsible authority allocating additional financial and human resources for the upkeep of such systems [12]. The highly anticipated e-WULAAS has also been plagued with technical issues resulting in the platform frequently being off-line and applicants needing to resubmit applications.

Furthermore, not all CMAs have been established (risk 3) and those that have are not functioning optimally [62,63,64]. Currently, only two of the potential nine CMAs have been established and are functional [12,50], which makes one question the realization of the subsidiarity principle in decision-making at the lowest level. The delay in the establishment of CMAs may be attributed to the challenges associated with the delineation of water management areas as the boundaries of these areas follow hydrological boundaries and not those of administrative boundaries leading to an escalation in complexity when it comes to decision-making processes [63]. To ensure that these complexities are dealt with effectively elements such as communication, cooperation, and intergovernmental relations amongst different scales and between different entities need to be recognized [63]. However, it is evident that co-operative governance and intergovernmental relations have been difficult to establish within the South African context, especially within the water governance sector [50,62,66,67]. These difficulties can mainly be attributed to the fact of uncertainty related to decision-making powers and a lack of resources within the water governance sector [66].

Resource classification and the determination of RQOs and the Reserve (risk 4) have only been determined for certain catchments [49] and therefore an integral part of the WULA system is lacking, which is essential for the achievement of catchment management objectives and the meeting of basic human needs and ecological water requirements. The progressive implementation of water resource classification is promising and lays the foundation for sustainable water resource management, the question remains on whether the classification system will contribute to equitable and efficient water allocation in the future [68]. Once more, the implementation of water use licenses to achieve the determined RQOs should be noted seeing that the conditions within the licenses are used to set thresholds in terms of discharge quality and ecological requirements. This, however, seems more difficult to achieve and the need exists to develop relevant approaches in the setting of applicable license conditions to realize the resource quality objectives [69]. The determination of the Reserve relies on highly technical administrative processes and methods for the quantification of flow, habitat, and water quality requirements [11,52]. From the literature, it is evident that these specific processes and methods are problematic and need to be simplified [15,70,71] to ensure that the NWA achieves its main objectives [49].

Risks 6 and 19 deal with procedural and substantive shortcomings in water use licenses [16,65], which ultimately questions the ability of such source-directed control to mitigate the negative impacts on the water resource and meet strategic objectives. Public participation (risk 8) in environmental decision-making processes is fundamental to ensure the fulfilment of human and environmental rights and the achievement of sustainable development [72]. The process of public participation was subsequently included in the post-apartheid water legislation to ensure the needs and concerns of vulnerable and marginalized communities are addressed in the pursuit of equitable access to water [5]. Questions over the NWA’s [7] ability to achieve this have been argued [17], with gaping omissions related to ‘who should be involved’, ‘the manner of involvement’, and ‘the time and extent of the involvement’ [1]. It seems, however, that the 2017 regulations have addressed at least most of these questions with the promulgation of a detailed procedure to be followed regarding the public participation process.

Process efficiency (risk 14) has been seriously questioned and the delays in water use license applications have been blamed for adding an unnecessary administrative burden and green tape [12,16,53,65]. In South Africa, a lack of empirical evidence exists on the substantive quality and completeness of technical and specialist reports (risk 16). However, recent reports have surfaced indicating that sub-standard reports, including weak impact assessments, have all contributed to an ineffective WULA system [16,53,65] which in turn may lead to procedural and substantive decision-making challenges. Certain key risks are therefore already manifesting in practice, which means the system is relying on flawed assumptions and not working as intended or as designed.

Table 1. Key assumptions and key risks for the South African WULA system.

Nr	Key Assumption	Key Risk	Relevant Literature Informing the Key Risk
1	Sufficient skills and competencies are in place to implement the WULA system	There are insufficient skills and competencies in place to implement the WULA system	[8,61,64]
2	Necessary infrastructure, communication, data, information are available, up to date and adequate to support the WULA system	Infrastructure, communication, data, information are unavailable and/or outdated and/or inadequate to support the WULA system	[9,12,66,73]
3	Catchment Management Agencies have been established and are functioning	Catchment management agencies have not been established or are not functioning	[50,62,63]
4	Resource classification, resource quality objectives and the Reserve have been determined	Resource classification, resource quality objectives, and the Reserve have not been determined	[15,49,70]
5	Funds are available to support and sustain the WULA system	No funds are available to support and sustain the WULA system	Lack of literature in the water sector
6	All relevant water uses have been identified during the site inspection	Not all relevant water uses have been identified during the site inspection	[16]
7	It is possible to agree on the requirements for the technical review	There is no agreement on the requirements for the technical review	Lack of literature in the water sector
8	The public is willing to participate and to do so in good faith	The public is unwilling to participate or do not participate in good faith	[17,74,75]
9	Scientific/technical reports are valid, of high quality, and comprehensive	Scientific/technical reports are invalid, poor quality, and are incomplete	[65]
10	Impacts on the water resource can be accurately predicted by means of an established method, criteria, and credible baseline information	Impacts on the water resource cannot be accurately predicted	[57,76] in relation to EIA. Lack of literature in the water sector
11	Administrators/officials read applications/scientific/technical reports	Administrators and officials do not read applications/scientific/technical reports	Lack of literature in the water sector
12	Administrators/officials understand the content of the application/scientific/technical reports	Administrators and officials do not understand the content of reports	Lack of literature in the water sector
13	Administrators/officials are rational, impartial, unbiased, and objective during the review process	Administrators and officials are irrational/ biased and subjective during the review process	[58,77] in relation to EIA. Lack of literature in the water sector
14	Water use license applications are processed within the set timeframes	Applications are not processed within the set time frames	[16,53,65]
15	The Water Tribunal is objective and impartial	The water tribunal is not objective and impartial	[16]
16	An effective and efficient process leads to high-quality reports	An effective and efficient process does not lead to high-quality reports	Lack of literature in the water sector
17	High-quality reports lead to informed decisions	High-quality reports do not lead to informed decisions	Lack of literature in the water sector
18	Decisions are lawful, reasonable, and procedurally fair	Decisions are not lawful, reasonable, and procedurally fair	[34,55,56] in relation to EIA. Lack of literature in the water sector.
19	Water use licenses contain informed conditions to protect water resources	Water use licenses do not contain relevant information and set weak conditions	[16,53,65]
20	Decisions are underpinned by decision-making principles	Decisions are not underpinned by decision-making principles	[34,55,56] in relation to EIA. Lack of literature in the water sector.
21	Informed decisions regulating water use that are lawful, reasonable, and procedurally fair will lead to the progressive realization of the sections 24 and 27 of the Constitution	Informed decisions that are based on the principles of administrative justice do not ultimately lead to the realization of sections 24 and 27 of the Constitution	[34,55,56] in relation to EIA. Lack of literature in the water sector.

Table 1. Key assumptions and key risks for the South African WULA system.

Nr	Key Assumption	Key Risk	Relevant Literature Informing the Key Risk
1	Sufficient skills and competencies are in place to implement the WULA system	There are insufficient skills and competencies in place to implement the WULA system	[8,61,64]
2	Necessary infrastructure, communication, data, information are available, up to date and adequate to support the WULA system	Infrastructure, communication, data, information are unavailable and/or outdated and/or inadequate to support the WULA system	[9,12,66,73]
3	Catchment Management Agencies have been established and are functioning	Catchment management agencies have not been established or are not functioning	[50,62,63]
4	Resource classification, resource quality objectives and the Reserve have been determined	Resource classification, resource quality objectives, and the Reserve have not been determined	[15,49,70]
5	Funds are available to support and sustain the WULA system	No funds are available to support and sustain the WULA system	Lack of literature in the water sector
6	All relevant water uses have been identified during the site inspection	Not all relevant water uses have been identified during the site inspection	[16]
7	It is possible to agree on the requirements for the technical review	There is no agreement on the requirements for the technical review	Lack of literature in the water sector
8	The public is willing to participate and to do so in good faith	The public is unwilling to participate or do not participate in good faith	[17,74,75]
9	Scientific/technical reports are valid, of high quality, and comprehensive	Scientific/technical reports are invalid, poor quality, and are incomplete	[65]
10	Impacts on the water resource can be accurately predicted by means of an established method, criteria, and credible baseline information	Impacts on the water resource cannot be accurately predicted	[57,76] in relation to EIA. Lack of literature in the water sector
11	Administrators/officials read applications/scientific/technical reports	Administrators and officials do not read applications/scientific/technical reports	Lack of literature in the water sector
12	Administrators/officials understand the content of the application/scientific/technical reports	Administrators and officials do not understand the content of reports	Lack of literature in the water sector
13	Administrators/officials are rational, impartial, unbiased, and objective during the review process	Administrators and officials are irrational/ biased and subjective during the review process	[58,77] in relation to EIA. Lack of literature in the water sector
14	Water use license applications are processed within the set timeframes	Applications are not processed within the set time frames	[16,53,65]
15	The Water Tribunal is objective and impartial	The water tribunal is not objective and impartial	[16]
16	An effective and efficient process leads to high-quality reports	An effective and efficient process does not lead to high-quality reports	Lack of literature in the water sector
17	High-quality reports lead to informed decisions	High-quality reports do not lead to informed decisions	Lack of literature in the water sector
18	Decisions are lawful, reasonable, and procedurally fair	Decisions are not lawful, reasonable, and procedurally fair	[34,55,56] in relation to EIA. Lack of literature in the water sector.
19	Water use licenses contain informed conditions to protect water resources	Water use licenses do not contain relevant information and set weak conditions	[16,53,65]
20	Decisions are underpinned by decision-making principles	Decisions are not underpinned by decision-making principles	[34,55,56] in relation to EIA. Lack of literature in the water sector.
21	Informed decisions regulating water use that are lawful, reasonable, and procedurally fair will lead to the progressive realization of the sections 24 and 27 of the Constitution	Informed decisions that are based on the principles of administrative justice do not ultimately lead to the realization of sections 24 and 27 of the Constitution	[34,55,56] in relation to EIA. Lack of literature in the water sector.

5. Conclusions

This paper aimed to identify key risks to the effectiveness of the WULA system in South Africa through the application of ToC, with a view to inform future system evaluations and influence system design. The results highlight 21 key assumptions and risks embedded in the design of the WULA system—see Table 1. Testing these assumptions and risks is important to understand if the system is functioning as designed and/or if the system design is flawed. If in the case of the latter, it is essential to reconsider the design of the system through policy and legislative reform. It was not the aim of this paper to evaluate the risks although the existing literature and anecdotal evidence already suggest that certain risks are currently manifesting in practice. Other risks lack information and remains unaddressed in the literature. This means that a comprehensive system review is required to strengthen our understanding of those risks that are already considered in the literature and to inform those risks that have not been addressed to date.

The ToC framework presented in this paper provides for such a comprehensive evaluation with the next step being to develop evaluation criteria against the identified key assumptions and risks. The ToC approach to evaluation makes provision for the design of key performance indicators (KPIs) for evaluation and continual monitoring of systems. Based on the outcome of the system evaluation, recommendations can then be made towards a more effective system design. With the growing water management challenges facing the country, it is essential that all water governance systems are optimally designed and functioning effectively, particularly a water governance system such as the WULA system with its main objectives linked to the realization of human and environmental rights. If the WULA system is not functioning as intended, basic human rights, such as equitable access to water, especially access to water for vulnerable communities and having the environment protected for future generations, might not be realized. Additionally, much needed socio-economic development will be jeopardized if available water is not redistributed in a timely manner due to an ineffective WULA system. In conclusion, it is the view of the authors that system evaluation is a fundamentally important aspect of any well-functioning governance system, and it is our hope that this paper serves as a first step in that direction.