(This HTML file is around 65 megabytes large, so it may take a while to load. Alternatively, you can view the plots and data in the GitLab repository at https://gitlab.com/denktank/mdma-policy-mdmcda)
###-----------------------------------------------------------------------------
### Check for the required packages and set options and variables
###-----------------------------------------------------------------------------

if (!('ufs' %in% row.names(installed.packages()))) {
  if (!(packageVersion('ufs') >= "0.4")) {
    stop("You need to have R package `ufs` installed in at least ",
         "version 0.4. If that version is not yet on CRAN, you can ",
         "install it from GitLab with the `remotes` package and command:\n\n",
         "remotes::install_gitlab('r-packages/ufs');");
  }
}

producePlots <- TRUE;
produceConfidencePlots <- TRUE;
quiet = TRUE;

### Check for required packages
ufs::checkPkgs("DiagrammeR");     ### Imported by `mdmcda`
ufs::checkPkgs("DiagrammeRsvg");  ### Imported by `mdmcda`
ufs::checkPkgs("ggplot2");        ### Imported by `mdmcda`
ufs::checkPkgs("ggtext");         ### Imported by `mdmcda`
ufs::checkPkgs("justifier");      ### Imported by `mdmcda`
ufs::checkPkgs("knitr");          ### Imported by `mdmcda`
ufs::checkPkgs("kableExtra");     ### Imported by `mdmcda`
ufs::checkPkgs("openxlsx");       ### Imported by `mdmcda`
ufs::checkPkgs("yum");            ### Imported by `mdmcda`

ufs::checkPkgs("tidyr");          ### Used once for pivot_wider
ufs::checkPkgs("patchwork");      ### Combining ggplots
ufs::checkPkgs("sessioninfo");    ### For reporting session information

### Update `mdmcda` from GitLab; use development branch
ufs::quietGitLabUpdate("r-packages/mdmcda@dev", quiet=quiet);

if (knitr::is_html_output()) {
  knitr::opts_chunk$set(echo=TRUE);
} else {
  knitr::opts_chunk$set(echo=FALSE);
}

options(knitr.kable.NA = '');

knitr::opts_chunk$set(dev="png", 
                            dev.args=list(type="cairo"),
                              dpi=150,
                            comment="");

ufs::opts$ez$figSize(size = "a4",
                     fontSizeMultiplier = 1.3);

mdmcda::opts$ez$figSize(size = "a4",
                        fontSizeMultiplier = 1.3);

ufs::opts$set(ggSaveDPI = 150);
mdmcda::opts$set(ggSaveDPI = 150);

### Set regular expressions for extracting decision and criterion
### identifiers from the performance subtable filenames
mdmcda::opts$set(
  performanceTable_decisionRegex  =
    c("performance_subtable_for_(.*)_on_.*_by_all\\.xlsx$",
      "\\1")
);
mdmcda::opts$set(
  performanceTable_criterionRegex =
    c("performance_subtable_for_.*_on_(.*)_by_all\\.xlsx$",
      "\\1")
);

ggBaseSize <-
  mdmcda::opts$get('ggBaseSize');

### Set variable names
scorerId <- "all";
weightProfileName <- "meanWeights"
weightedEstimateName <- paste0(weightProfileName, "_weighted_estimate");
tempWeightedEstimateName <- paste0(scorerId, "_", weightedEstimateName);

### Set path with files
currentDir <- here::here();
dataPath <- file.path(currentDir,     "data");
workingPath <- file.path(currentDir,  "results");
scoredPST_path <- file.path(dataPath, "scored-PSTs");

### Set filenames
criteriaFile <-
  file.path(dataPath,
            "mdma-policy-mdmcda-fullCriteriaDf.xlsx");
postSetsFile <-
  file.path(dataPath,
            "postsets--(adjustments-after-collective-scoring-days).xlsx");
scenariosFile <-
  file.path(dataPath,
            "mdma-scenario-definitions.xlsx");
weightsFile <-
  file.path(dataPath,
            "mdma-criteria-weights.xlsx");

### Tell ufs::knitAndSave to always 'cat' the knitr chunk
options(ufs.knitAndSave.catPlot = TRUE);

### Set automatic figure numbering
ufs::setFigCapNumbering();

MDMA MDMCDA

Overview

This reproducible report documents the analyses done for the MDMA Policy Think Tank that was active in the Netherlands in 2019-2020 (see the Background tab for more details). This document loads the produced data and computes the results. The most recent version of this document is hosted at https://denktank.gitlab.io/mdma-policy-mdmcda, the corresponding Git repository is at https://gitlab.com/denktank/mdma-policy-mdmcda, the Open Science Framework project for this repository is https://osf.io/h58r6/, and as of November the 13th, a preprint will be available at https://doi.org/10.31219/osf.io/txy5z.

This document is organised in the tabs above:

  • Introduction: this tab with the introduction.
  • Background: details about the background of this initiative.
  • Method: details about the method we used.
  • Reading and preprocessing: mostly the code used to read and preprocess the data.
  • Input: the specifications and estimates by the experts that were the input for the analyses
  • Results: an overview of the main results.
  • First English-language manuscript: the figures and tables for the first English-language manuscript.
  • First Dutch-language report: the figures and tables for the first Dutch-language report.
  • Additional visualisations: more detailed visualisations.
  • Experts’ weights: visualisations showing the weights the individual experts assigned.
  • Confidence scores: visualisations showing confidence scores for each performance table.
  • Sensitivity analyses: sensitivity analyses to test the robustness of the results.
  • Exporting results: mostly the code used to export the results to spreadsheets.
  • Session information: the versions of R and the packages that were used, to enhance reproducibility.

App and website

There is an online app at https://mdmapolicy.com/app where you can specify your own policy model and weights for the outcomes. https://mdmapolicy.com (an alias of https://mdmabeleid.nl) is a website about this project where, as of the 13th of November, a Dutch-language report will become available.

mdmcda R package

The Think Tank used Multi Decision Multi Criteria Decision Analysis, which is an adaption of the MCDA procedure. This document and all analyses reported here use the mdmcda R package, available through https://r-packages.gitlab.io/mdmcda. Note that it is still under development, which is why this repository has been frozen as a registration in the Open Science Framework project at https://osf.io/h58r6/. If you run into any issues with reproducing these analyses, or if you would like to use this R package to conduct an MDMCDA in your own country, you can contact Gjalt-Jorn Peters (the package developer, who also wrote this reproducible document) through Twitter or email.

Background

Discussions about drug policy often polarize. Often, individuals’ morals, prejudices and emotions come to drive the discussion rather than theory- and evidence-based argument and trying to find the best solution.

To attempt to identify the best way forward without the process being derailed by idealistic and emotional arguments, formal decision-making instruments have been employed. A famous example is the application of Multi Criteria Decision Analysis to rank drug harms in the UK (Nutt, King & Phillips, 2010). This same approach can be used to not only look at drug harms, but also at the effects different drug policies have on such harms.

Røgeberg et al. (2017) did exactly this. They compared four regulatory regimes (absolute prohibition, decriminalisation, state control and free market) and used Multi Criteria Decision Analysis (MCDA) to compare their estimated effects on 27 criteria (e.g. “Reduce user harms”, “Enable medical use”, “Promote family cohesion”, “Reduce acquisitive crime”, and “Reduce economic sosts”) for alcohol and cannabis. For both drugs, state control emerged as the optimal regime.

However, a drawback of this approach is that in the process of estimating each regulatory regime’s performance on the criteria, the experts are aware of how the pattern of estimates impacts the relative performances of each regulatory model. This leaves the process relatively open to pre-existing biases.

In the Dutch think tank process, we aimed to eliminate such biases as much as possible. To do this, an R package was developed that implements Multi Decision Multi Criteria Decision Analysis (MDMCDA). The Method tab contains a description of how this relates to MCDA. In short: the scenarios (called “regulatory regimes” by Røgeberg et a. (2017) and “policy models” in the current exercise) are not predefined, but only defined afterwards. We split up the process in an “objective phase” and a “subjective phase”.

The objective phase in MDMCDA: expert input

In the “objective phase”, the first two steps were to establish:

  • the outcomes (“criteria” in MCDA terms, such as prevalence of MDMA use; health damage; MDMA-related organised crime; or damage to the enviroenment);
  • the policy instruments (‘decisions’ in MCDA terms, such as “legal status of MDMA possession”; “license required for MDMA sale to consumers”, “minimum age”); and
  • and the options that comprise each decison (‘alternatives’ in MCDA terms, such as, for “license required for MDMA sale to consumers”: “No license required”, “License required”, and “Not applicable”).

The next step was to estimate the effects of all alternatives within all decisions (all policy options within all instruments) on all criteria (outcomes). For example, if selling MDMA would require a license, how would that impact the prevalence of MDMA use, health damage, MDMA-related organised crime, and damage to the environment?

Once all these effects had been estimated, that concluded the more objective expert input part of the exercise.

The subjective phase in MDMCDA: scenarios and weights

The next two steps are mostly informed by personal or political preference and so are more subjective. These two steps are specifying the following two things.

  • The policy models: the scenarios that each consist of one selected option (alternative) for each policy instrument (decision). For example, under a prohibitive policy, MDMA sales to consumers would be illegal, so no minimum age would be defined; whereas under a regulatory policy, the minimum age can be set to 18 years.
  • The weights for the outcomes (criteria). For example, somebody might value user health very much, but international politics much lower; and somebody else might value crime reduction very highly, and economical outcomes very low.

Once these have been specified, it is possible to weigh all the estimates using the specified weights, and then compute the total scores for all scenarios given the selected alternatives that comprise them.

Characteristics of MDMCDA

This approach reduces bias because the experts estimate the effects of single alternatives (options) within decisions (instruments) on single criteria each time. Because at that point, the scenarios (policy models) have not yet been defined, and the criteria have not yet been weighted, it is not possible to change these estimates to make a given scenario “win”.

An second benefit is that defining the scenarios and specifycing the weights becomes independent of the estimates. It is wasy to change scenarios or weights afterwards and again compute how each scenario scores. In addition, it is possible to compose a scenario out of the highest-scoring alternative (option) within each decision (policy instrument), as well as compute the worst-scoring scenario.

A third characteristic of MDMCDA is instead of so-called “swing weighting”, global weighting is used (see the Method tab for details and rationale). This can be soon both as a disadvantage and an advantage, but one specific benefit is that this disentangles the “importance” and “impact” dimensions, which are combined in one weight when applying swing weighting. The impact is expressed in the estimates set in phase 1, whereas the importance is determined in phase 2.

cat('This document is the technical report with all results. It contains all the R code used to generate these results. The R code can be expanded by clicking the "Code" button that sometimes appears at the right-hand side of the page, such as just above this paragraph, which will show the `cat` command used to print this paragraph.');

This document is the technical report with all results. It contains all the R code used to generate these results. The R code can be expanded by clicking the “Code” button that sometimes appears at the right-hand side of the page, such as just above this paragraph, which will show the cat command used to print this paragraph.

To view underlying data or the R scripts themselves, check out the repository at https://gitlab.com/denktank/mdma-policy-mdmcda. This file and the other repository contents are frozen when a manuscript is submitted. These frozen public registrations are available at out Open Science Framework repository at https://osf.io/h58r6/.

Method

In regular Multi Criteria Decision Analysis (MCDA), a number of alternatives are crossed with a number of criteria as means to select the optimal alternative. The table obtained by crossing all alternatives with all criteria is called a performance table, and each cell contains an estimate. By weighting the criteria in terms of their importance the optimal alternative can be computed. Often, this weighting not only weighs each criterion’s importance, but simultaneously the relative magnitude of the impact the alternatives can have on each criterion, in which case the weighting is called swing weighting. Swing weighting also serves the purpose of calibrating differences in metric between criteria.

However, regarding substance policy, achieving consensus on a manageable set of alternatives, as well as on the weights to use, is challenging and perhaps impossible. For example, experts in legal systems and policing might define a “prohibition” policy very differently from experts in health and prevention. Similarly, experts in economy and tax policy might define a “free market” system very differently than experts in policing and prevention. Similar variation exists is preferred weighting: for example, criteria relating to crime and health may be weight differently by health professionals than by police officers.

This presents a problem for MCDA, which requires predefined alternatives. To respect these different viewpoints, a specific application of the MCDA procedure was used that works for multiple decisions as well as multiple criteria (Multi Decision Multi Criteria Decision Analysis, MDMCDA). MDMCDA enables establishing both the alternatives and the weights after the estimates in the performance table have been produced. In order to do this, it sacrifices the elegance of swing weighting. Specifically, MDMCDA assumes that each alternative can be broken down into separate decisions that each have “sub-alternatives” of their own.

For example, a “prohibiton policy” alternative may differ from a “free market” alternative in the decision whether sale to consumers is legal or not, as well as in the decision whether quality standards are defined. Each of these two decisions comprises a number of sub-alternatives: for example, sale to consumers can be legal, illegal, or only legal with a license; and quality standards can either be defined by a governmental body or not. A “prohibition policy” may consist of the alternatives that sale to consumers is illegal and no quality standards are defined; and a “free market” alternative may consist of the alternatives that sale to consumers is legal and quality standards are defined.

Once all relevant decisions and all relevant “sub-alternatives” for each decision have been identified, each alternative (in the MCDA meaning of the word) can be defined in terms of which “sub-alternative” is selected within each decision. However, defining the alternatives is now no longer a precondition for producing the estimates in the performance tables (i.e. the estimates of the effects on each criterion). These estimates can now be produced on the level of the “sub-alternatives”. In other words, Multi Decision Multi Criterion Decision Analysis breaks down the MCDA procedure into a set of related sub-MCDA procedures, that are aggregated again once all performance sub-tables have been produced.

For example, in the example introduced above, the effects of whether sale to consumers is legal, illegal, or only legal with a license would be estimated for each criterion; and the effects of whether quality standards exist or not would also be estimated for each criterion. If the criteria would be, for example, “health” and “crime”, this would mean that for the first decision (legal status of sale to consumers), three estimates would be produced for the effects on health, and three for the effects on crime. For the second decision (whether quality standards exist), two estimates would be produced for the effects on health, and two for the effects on crime.

In total, therefore, ten estimates would be produced, in two performance sub-tables (each decision has its own performance sub-table). In a traditional MCDA, only four estimates would be produced: two alternatives (prohibition versus free market) crossed with two criteria (health and crime). As will be clear, when applying DMCDA as opposed to MCDA, the number of estimates to be produced is much higher. This is a disadvantage in terms of required effort, but is an advantage in terms of how complex estimating the effect on each criterion is. Breaking down each alternative into decisions and sub-alternatives means estimating the effects on each criterion requires less mental aggregation of component effects.

Often, criteria in an MCDA can be organized hierarchically. For example, the criterion “health” may in fact be a cluster of criteria “acute risks” and “long-term neurotoxicity”, and the criterion “crime” may be a cluster of criteria “organized crime” and “petty crime”. In MCDA, the full performance table is in practice split up by criterion cluster (i.e. by cluster of columns). For example, first the effects of each alternative on both criteria in the “health” cluster would be scored, and then the effects of each alternative on both criteria in the “crime” cluster.

Since swing weighting is performed for the effects on all alternatives on each criterion, this splitting up of the performance table by clusters of columns does not pose any problems. In MDMCDA, however, the performance table is also by each decision. Since in MDMCDA, the full performance table does not consist of alternatives crossed with criteria, but instead of decisions and their sub-alternatives crossed with criteria, the resulting performance table has many more rows than the performance table in conventional MCDA. During estimating each sub-alternative’s effect on the criteria in each cluster, therefore, the full performance table is also split by decision. This means the full performance table can be viewed as a grid of performance subtables: one for each criteria cluster-decision combination.

In our example, with two clusters (crime and health) that each consist of two criteria (organized and petty crime; and acute and long-term risk), and two decisions (legal status of sale to consumers and quality criteria), this results in four performance sub-tables to be filled with estimates. However, because swing-weighting simultaneously weighs each criterion’s importance and the relative magnitudes of effect of the alternatives, this requires all alternatives to be weighted simultaneously. Once alternatives are delineated into decisions and sub-alternatives, this is no longer possible.

Therefore, instead of calibrating the estimates using swing-weighting, global weighting is used (see also Monat, 2009). With global weighting, the metrics on which the alternatives effects on each criterion are estimated are established in advance. Specifically, in MDMCDA, the status quo represents a score of 0, and for each criterion, maximum deviation in the negative and positive directions are established. For example, if the status quo is relatively dismal, the positive extreme is set to 100, but the negative extreme to a smaller value, such as -20 (if things can get only around one fifth worse than they can get better). Another criterion may be scores from -100 to 50, if the status quo is already relatively desirable, but there is room for deterioration.

This approach fixes the scoring metrics such that scorers always have two anchor points (0 and either -100 or 100), and can score the estimates relative to those anchors. This eliminates the problem produced if a fixed scale would be used for all criteria (e.g. 0 to 100): in that case, the status quo is represented by a different score for each criterion, which introduces extraneous cognitive load for the scorers.

The magnitudes of the effect of the sub-alternative within each decision then manifest in the estimates scores. This has as second benefit that the weighting is restricted to the value one attaches to the criteria. In other words, this approach dissociates the more objective aspect of swing weighting fro its more subjective, moral aspect. That dissociation enables application of multiple weight profiles, enabling comparison of the effects of variations in criterion priority on total scores.

The terminology of MDMCDA is slightly different from the terminology of MCDA to retain unequivocal reference of ‘alternatives’ to the options comprising a decision. Therefore, in MDMCDA, each decision (e.g. legal status of sale to consumers) consists of two or more alternatives (avoiding the term “sub-alternatives”), and each specific configuration of alternatives (e.g. “prohibition”) is called a scenario. These scenarios, therefore, correspond to what in regular MCDA are the alternatives.

Reading and preprocessing

Reading data

###-----------------------------------------------------------------------------
### Read criteria, estimates, weights, scenarios, and post-sets
###-----------------------------------------------------------------------------

criteria <-
  mdmcda::read_criteria_from_xl(
    criteriaFile,
    showGraphs = FALSE);

estimates <-
  mdmcda::read_performance_tables(input = scoredPST_path);

weights <-
  mdmcda::read_weights_from_xl(input = weightsFile);

scenarioDefinitions <-
  mdmcda::read_scenarioDefinitions_in_columns_from_xl(
    scenariosFile
  );

postsets <-
  as.data.frame(openxlsx::read.xlsx(postSetsFile, sheet = 1));

###----------------------------------------------------------------------------
### Read labels
###----------------------------------------------------------------------------

### Read criterion labels
criterionLabels <-
  mdmcda::read_criterionLabels_from_xl(
    file.path(dataPath,
              "mdmcda-data--criterionLabels_en.xlsx")
  );
criterionLabels_NL <-
  mdmcda::read_criterionLabels_from_xl(
    file.path(dataPath,
              "mdmcda-data--criterionLabels_nl.xlsx")
  );

### Read decision labels
decisionLabels <-
  mdmcda::read_decisionLabels_from_xl(
    file.path(dataPath,
            "mdmcda-data--decisionLabels_en.xlsx")
  );
decisionLabels_NL <-
  mdmcda::read_decisionLabels_from_xl(
    file.path(dataPath,
            "mdmcda-data--decisionLabels_nl.xlsx")
  );

### Read decision descriptions
decisionDescriptions <-
  mdmcda::read_decisionDescriptions_from_xl(
    file.path(dataPath,
              "mdmcda-data--decisionLabels_en.xlsx")
  );

### Read scenario labels
scenarioLabels <-
  mdmcda::read_scenarioLabels_from_xl(
    file.path(dataPath,
            "mdmcda-data--scenarioLabels_en.xlsx")
);
scenarioLabels_NL <-
  mdmcda::read_scenarioLabels_from_xl(
    file.path(dataPath,
            "mdmcda-data--scenarioLabels_nl.xlsx")
);

### Read alternative labels
alternativeLabels <-
  mdmcda::read_alternativeLabels_from_xl(
    file.path(dataPath,
              "mdmcda-data--alternativeLabels_en.xlsx")
);
alternativeLabels_NL <-
  mdmcda::read_alternativeLabels_from_xl(
    file.path(dataPath,
              "mdmcda-data--alternativeLabels_nl.xlsx")
);

###-----------------------------------------------------------------------------
### Set orders for criteria, decisions, and scenarios
###-----------------------------------------------------------------------------

criterionOrder <-
  setdiff(names(criterionLabels),
          criteria$convenience$parentCriterionIds);
parentCriterionOrder <-
  intersect(names(criterionLabels),
            criteria$convenience$topCriterionClusters);

decisionOrder <- names(decisionLabels);
scenarioOrder <- names(scenarioLabels);

###-----------------------------------------------------------------------------
### Wrap the scenario labels for horizontal display
###-----------------------------------------------------------------------------

wrappedScenarioLabels <-
  unlist(lapply(scenarioLabels, function(x) 
    paste(strwrap(x, 10), collapse="\n")
  ));

wrappedScenarioLabels_NL <-
  unlist(lapply(scenarioLabels_NL, function(x) 
    paste(strwrap(x, 10), collapse="\n")
  ));

Criteria (outcomes of chosen policy)

The criteria (‘outcomes’) have a hierarchical structure to facilitate weighting them efficiently. They are clustered based on domain, such that each outcome can be considered an indicator of the cluster outcome. For example, prevalence in the general population, prevalence in vulnerable populations, and frequency and intensity of use by MDMA users are all indicative of MDMA use (the overarching cluster). Therefore, if a decision maker values MDMA use as an outcome, that cluster can received a high weight. The relative contribution of the three indicators can then be finetuned by setting the weights of the individual outcomes. Similarly, a decision maker who does not want MDMA use to play a large role when determining their policy can set the cluster weight to a low value or to zero. The latter would immediately take all three contained outcomes out of the total scores computed for the scenarios (policy models).

Estimates

In total, 2565 estimates were read. These estimates express estimated effects of 22 decisions (policy instruments), that together comprise a total of 95 alternatives (policy options), on a total of 27 criteria (outcomes).

Estimate post-sets

In total, 48 post-sets were loaded, pertaining to 10 decisions (policy instruments) and 11 criteria (outcomes).

Weights

To set weights, all experts submitted individually determined weights for each outcome and each outcome cluster on a scale from 0-100, by first setting the weight of the most important cluster and the most important outcome in each cluster to 100, and then setting the weight of the other clusters and outcomes according to their relative importance compared to the most important cluster or outcome (e.g. an outcome half as important as the most important outcome in the same cluster would get a weight of 50). These individually set weights will later be used to compute consensus weights that will be used in the rest of the procedures.

Weights were loaded from 16 scorers with identifiers ‘Scorer3’, ‘Scorer4’, ‘Scorer5’, ‘Scorer6’, ‘Scorer7’, ‘Scorer8’, ‘Scorer9’, ‘Scorer10’, ‘Scorer11’, ‘Scorer12’, ‘Scorer13’, ‘Scorer14’, ‘Scorer15’, ‘Scorer16’, ‘Scorer17’ & ‘Scorer18’.

Scenarios (“policy models”)

Scenarios are specific configurations of one selected alternative for each decision. The think-tank predefined 4 scenarios (policy models) with identifiers ‘repression’, ‘coffeeshop’, ‘adapted_coffeeshop’ & ‘free_market’.

Labels

Labels are read separately to enable easy translations to different languages.

Preparing data

In this section, the imported data are preprocessed. This can be inspected by expanding the code fragments.

Correcting weights

Initially, the “protection of the environment” criterion did not have a parent. However, the experts did not weigh this criterion taking that into account, causing the resulting mean weight to be too high. To correct this, this criterion was placed into its own cluster with the weight defined as the mean weight of all other clusters.

In addition, the experts specified weights for the “policy aligns with conservative values” and “policy aligns with liberal values”, but the Think Tank results should be apolitical, and so we override these to zero here.

###-----------------------------------------------------------------------------
### The 'planet' parent for the 'environment' criterion was added
### later; so we need to add a weight for it. As weight, we take the
### mean weight of all clusters, as agreed in the think tank meeting
###-----------------------------------------------------------------------------

planetWeight <-
  mean(
    weights$allWeights$weight[
      weights$allWeights$parentCriterion_id=="outcomes"
    ],
    na.rm=TRUE
  );

weights$individualWeights <-
  lapply(weights$individualWeights,
         function(dat) {
           dat[dat$id=="planet", "weight"] <- planetWeight;
           return(dat);
         });

weights$allWeights[weights$allWeights$criterion_id=="planet", "weight"] <-
  planetWeight;

###-----------------------------------------------------------------------------
### Set the weight of cultural values (liberal vs conservative) to 0 for the
### rest of the computations (because the think tank has no political/idealistic
### preference).
###-----------------------------------------------------------------------------

weights$allWeights[
  weights$allWeights$criterion_id=="cultural_values", "weight"] <-
  0;

Apply post-sets

Post-sets are individual estimates that were reconsidered by the think tank after the scoring days, or that were only completed after the scoring days.

### Do replacements
estimates$multiEstimateDf <-
  mdmcda::set_postsets(
    multiEstimateDf = estimates$multiEstimateDf,
    postsets = postsets,
    coder = scorerId
  );
  • replacing estimate for the effect of alternative 2 for decision b2b_sale_legal_status on criterion costs_financial_crime, which was NA, with 5.

  • replacing estimate for the effect of alternative 3 for decision b2b_sale_legal_status on criterion costs_financial_crime, which was NA, with 5.

  • replacing estimate for the effect of alternative 4 for decision b2b_sale_legal_status on criterion costs_financial_crime, which was NA, with 5.

  • replacing estimate for the effect of alternative 5 for decision b2b_sale_legal_status on criterion costs_financial_crime, which was NA, with 15.

  • replacing estimate for the effect of alternative 2 for decision consumer_sale_legal_status on criterion state_revenues_vat, which was NA, with 80.

  • replacing estimate for the effect of alternative 3 for decision consumer_sale_legal_status on criterion state_revenues_vat, which was NA, with 100.

  • replacing estimate for the effect of alternative 4 for decision consumer_sale_legal_status on criterion state_revenues_vat, which was NA, with 60.

  • replacing estimate for the effect of alternative 2 for decision consumer_sale_legal_status on criterion state_revenues_tax, which was NA, with 80.

  • replacing estimate for the effect of alternative 3 for decision consumer_sale_legal_status on criterion state_revenues_tax, which was NA, with 100.

  • replacing estimate for the effect of alternative 4 for decision consumer_sale_legal_status on criterion state_revenues_tax, which was NA, with 60.

  • replacing estimate for the effect of alternative 1 for decision crime_priority on criterion criminalisation_of_users, which was NA, with 0.

  • replacing estimate for the effect of alternative 2 for decision crime_priority on criterion criminalisation_of_users, which was NA, with 0.

  • replacing estimate for the effect of alternative 3 for decision crime_priority on criterion criminalisation_of_users, which was NA, with -50.

  • replacing estimate for the effect of alternative 1 for decision crime_priority on criterion international_trafficking_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 2 for decision crime_priority on criterion international_trafficking_mdma, which was NA, with 20.

  • replacing estimate for the effect of alternative 3 for decision crime_priority on criterion international_trafficking_mdma, which was NA, with 20.

  • replacing estimate for the effect of alternative 2 for decision international_strategy on criterion organized_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 3 for decision international_strategy on criterion organized_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 4 for decision international_strategy on criterion organized_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 5 for decision international_strategy on criterion organized_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 6 for decision international_strategy on criterion organized_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 2 for decision international_strategy on criterion organized_crime_other, which was NA, with 0.

  • replacing estimate for the effect of alternative 3 for decision international_strategy on criterion organized_crime_other, which was NA, with 0.

  • replacing estimate for the effect of alternative 4 for decision international_strategy on criterion organized_crime_other, which was NA, with 0.

  • replacing estimate for the effect of alternative 5 for decision international_strategy on criterion organized_crime_other, which was NA, with 0.

  • replacing estimate for the effect of alternative 6 for decision international_strategy on criterion organized_crime_other, which was NA, with 0.

  • replacing estimate for the effect of alternative 2 for decision international_strategy on criterion international_trafficking_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 3 for decision international_strategy on criterion international_trafficking_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 4 for decision international_strategy on criterion international_trafficking_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 5 for decision international_strategy on criterion international_trafficking_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 6 for decision international_strategy on criterion international_trafficking_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 1 for decision legal_age_enforcing on criterion liberal_values, which was NA, with -25.

  • replacing estimate for the effect of alternative 1 for decision legal_age_enforcing on criterion conservative_values, which was NA, with -25.

  • replacing estimate for the effect of alternative 1 for decision licenses_for_selling on criterion organized_crime_mdma, which was NA, with -25.

  • replacing estimate for the effect of alternative 2 for decision licenses_for_selling on criterion organized_crime_mdma, which was NA, with 50.

  • replacing estimate for the effect of alternative 1 for decision licenses_for_selling on criterion organized_crime_other, which was NA, with 0.

  • replacing estimate for the effect of alternative 2 for decision licenses_for_selling on criterion organized_crime_other, which was NA, with 30.

  • replacing estimate for the effect of alternative 2 for decision ontneming on criterion criminalisation_of_users, which was NA, with 0.

  • replacing estimate for the effect of alternative 1 for decision pricing_restrictions on criterion international_image, which was NA, with 0.

  • replacing estimate for the effect of alternative 2 for decision pricing_restrictions on criterion international_image, which was NA, with 0.

  • replacing estimate for the effect of alternative 1 for decision production_legal_status on criterion small_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 2 for decision production_legal_status on criterion small_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 3 for decision production_legal_status on criterion small_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 4 for decision production_legal_status on criterion small_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 5 for decision production_legal_status on criterion small_crime_mdma, which was NA, with 0.

  • replacing estimate for the effect of alternative 1 for decision quality_management_sanctions on criterion international_image, which was NA, with 0.

  • replacing estimate for the effect of alternative 2 for decision quality_management_sanctions on criterion international_image, which was NA, with 0.

  • replacing estimate for the effect of alternative 3 for decision quality_management_sanctions on criterion international_image, which was NA, with 0.

### Also correct the parent element for environmental protection
estimates$mergedConfidences$parentCriterion_id <-
  ifelse(estimates$mergedConfidences$parentCriterion_id == "outcomes",
         "planet",
         estimates$mergedConfidences$parentCriterion_id);

Correct alternative values for possession status

For decision (instrument) possession status, the alternative values (policy option values) were misspecified in the performance sub-tables: the values were ‘0’, ‘1’, ‘2’, ‘4’ & ‘5’ (3 was omitted). This code corrects this by shifting alternatives 4 and 5.

To verify this, the tables below were produced.

kableExtra::kable_styling(
  knitr::kable(
    table(
      estimates$multiEstimateDf[
        estimates$multiEstimateDf$decision_id=='posession_status',
        c('alternative_label', 'alternative_value')]
    )
  )
);
0 1 2 4 5
Bezit is niet toestaan 0 27 0 0 0
Bezit is toegestaan 0 0 0 0 27
Gebruikershoeveelheid gedogen 0 0 27 0 0
Gebruikershoeveelheid legaal, grootbezit gedoogd 0 0 0 27 0
Niet van toepassing 27 0 0 0 0
estimates$multiEstimateDf[
  estimates$multiEstimateDf$decision_id=='posession_status',
  'alternative_value'] <-
  
  ifelse(
    estimates$multiEstimateDf[
      estimates$multiEstimateDf$decision_id=='posession_status',
      'alternative_value'] < 3,
    estimates$multiEstimateDf[
      estimates$multiEstimateDf$decision_id=='posession_status',
      'alternative_value'],
    as.numeric(estimates$multiEstimateDf[
      estimates$multiEstimateDf$decision_id=='posession_status',
      'alternative_value']) - 1
  );

cat("\n\nFrequencies of each option after correction:\n\n");

Frequencies of each option after correction:

kableExtra::kable_styling(
  knitr::kable(
    table(
      estimates$multiEstimateDf[
        estimates$multiEstimateDf$decision_id=='posession_status',
        c('alternative_label', 'alternative_value')]
    )
  )
);
0 1 2 3 4
Bezit is niet toestaan 0 27 0 0 0
Bezit is toegestaan 0 0 0 0 27
Gebruikershoeveelheid gedogen 0 0 27 0 0
Gebruikershoeveelheid legaal, grootbezit gedoogd 0 0 0 27 0
Niet van toepassing 27 0 0 0 0

Weighting

The individual weights are now averaged, and then rescaled such that the most important outcome and cluster again had a weight of 100 (i.e. to the same metric used by the individual scorers). All outcome weights are then divided by 100 and multiplied with the weight of the corresponding clusters.

After this procedure, the most important outcome in each cluster has the weight of its cluster, and the other outcomes’ weights decrease proportionally. Finally, every weight is divided by the sum of all weights and multiplied by 100, so that every final weight expresses the relative contribution of the corresponding outcome in each model’s final scores.

Again, the visualisations can be inspected in the second tab.

###-----------------------------------------------------------------------------
### Aggregate weight estimates
###-----------------------------------------------------------------------------

weightsMeansAndSDs <- mdmcda::weightsMeansAndSDs(weights);

###-----------------------------------------------------------------------------
### Combine with criteria tree to accumulate over the hierachy. Note that
### because data.tree objects use R6, the criteria$criteriaTree object is
### updated in `criteria`, so we only need to store weightsMeansAndSDs
###-----------------------------------------------------------------------------

weightsMeansAndSDs <-
  mdmcda::combine_weights_and_criteria(
    weightsMeansAndSDs,
    criteria,
    weightCols = c(raw = 'weight_mean_proportion',
                   rescaled = 'weight_mean_rescaled_proportion')
  );

###-----------------------------------------------------------------------------
### Compile weight profile
###-----------------------------------------------------------------------------

weightProfiles <-
  mdmcda::create_weight_profile(weightsMeansAndSDs = weightsMeansAndSDs,
                                criteria = criteria,
                                profileName = weightProfileName);
weightProfileNames <- names(weightProfiles);

###-----------------------------------------------------------------------------
### Add weights and weighted estimates to multiEstimateDf
###-----------------------------------------------------------------------------

estimates$multiEstimateDf <-
  mdmcda::weight_multiEstimateDf(
    multiEstimateDf = estimates$multiEstimateDf,
    weightProfiles = weightProfiles,
    scorer = scorerId
  );

### Remove the `scorerId` from the name of the weighted estimates because we
### didn't have multiple estimators (just the Think Tank estimates)
estimates$multiEstimateDf[, weightedEstimateName] <-
  estimates$multiEstimateDf[, tempWeightedEstimateName];

###-----------------------------------------------------------------------------
### Export final average weights for use in Shiny app or other purposes
###-----------------------------------------------------------------------------

openxlsx::write.xlsx(
  weightsMeansAndSDs,
  file.path(workingPath,
            "mdmcda-data--weightsMeansAndSDs.xlsx")
);

### Note that the critera cluster weights are in column "rescaled_product"
### (criteria clusters can be recognized because their parent is the root
### element (the element with parentId "-"), in this case, "outcomes").
### The weights for the criteria themselves (i.e. the weights making up the
### weight profile we use in this analysis script) are in column
### "rescaled_total_percentage".

### Also export weight profiles directly
weightprofiles_asDf <-
  mdmcda::write_weightProfile_to_xl(
    weightProfiles,
    file.path(
      workingPath,
      "mdmcda-data--weightProfiles.xlsx"
    )
  );

###-----------------------------------------------------------------------------
### Compute the total scores for each alternative in each decision
###-----------------------------------------------------------------------------

scores_per_alternative <-
  mdmcda::compute_scores_per_alternative(
    multiEstimateDf = estimates$multiEstimateDf,
    weightProfiles = weightProfiles
  );

###-----------------------------------------------------------------------------
### Compose best and worst scenario
###-----------------------------------------------------------------------------

bestAlternatives <-
  mdmcda::compute_best_alternatives(
    scores_per_alternative=scores_per_alternative,
    ignoreRegex = "^0$"
  );

worstAlternatives <-
  mdmcda::compute_worst_alternatives(
    scores_per_alternative=scores_per_alternative,
    ignoreRegex = "^0$"
  );

### From these data frames, extract the vectors to store in the object with
### scenario definitions. Sometimes, multiple options score the same. In these
### cases, select the first option for the scenario definition.

bestScenario <-
  as.numeric(gsub("^.*\\s(\\d+)$",
                  "\\1",
                  bestAlternatives$alternative_id));
names(bestScenario) <- bestAlternatives$decision_id;

worstScenario <-
  as.numeric(gsub("^.*\\s(\\d+)$",
                  "\\1",
                  worstAlternatives$alternative_id));
names(worstScenario) <- worstAlternatives$decision_id;

###-----------------------------------------------------------------------------
### Define scenario with tweaks to make it more acceptable to list of scenarios
###-----------------------------------------------------------------------------

xShopScenario <-
  bestScenario;

xShopTweaks <-
  c('posession_status' = 3,
             ### User quantity is legal, higher quantities condoned
    'advertising' = 1,
             ### Advertising not allowed
    'consumer_sale_legal_status' = 4,
             ### Analogous to pharmaceutical law
    'legal_age' = 2,
             ### Age limit is 18 years
    'export_status' = 1,
             ### Export is illegal
    'healthpromotion_responsible_government' = 4
             ### Both governments
    );

xShopScenario[names(xShopTweaks)] <-
  xShopTweaks;

###-----------------------------------------------------------------------------
### Add these three new scenarios to the list of scenarios
###-----------------------------------------------------------------------------

scenarioDefinitions <-
  c(scenarioDefinitions,
    list(optimal_scenario = bestScenario,
         worst_scenario = worstScenario,
         x_shop = xShopScenario));

###-----------------------------------------------------------------------------
### Generate a weighted estimate dataframe with all scenarios and the selected
### alternatives
###-----------------------------------------------------------------------------

### Create dataframe for the weighted estimates
weightedEstimates <-
  mdmcda::build_weighted_estimate_df(
    multiEstimateDf = estimates$multiEstimateDf,
    criterionOrder = criterionOrder,
    decisionOrder = decisionOrder,
    scenarioOrder = scenarioOrder,
    scenarioDefinitions = scenarioDefinitions,
    scorer = scorerId,
    setMissingEstimates=0
  );

### Actually weigh the estimates
weightedEstimates <-
  mdmcda::weight_estimates_by_profile(weighted_estimate_df = weightedEstimates,
                                      weight_profiles = weightProfiles);

### Add parent criterion identifiers
weightedEstimates$parentCriterion_id <-
    criteria$convenience$parentCriterionIds_by_childId[
      as.character(weightedEstimates$criterion_id)
    ];

###-----------------------------------------------------------------------------
### Total scores per scenario
###-----------------------------------------------------------------------------

scoresPerScenario <-
  mdmcda::scores_by_scenario(weightedEstimates = weightedEstimates,
                             estimateCols = weightedEstimateName);

### Add labels
scoresPerScenario$label_en <-
  scenarioLabels[scoresPerScenario$scenario_id];
scoresPerScenario$label_nl <-
  scenarioLabels_NL[scoresPerScenario$scenario_id];

### Sort in scenarioOrder
scoresPerScenario <-
  scoresPerScenario[match(scenarioOrder, scoresPerScenario$scenario_id), ];

### Get English and Dutch clean versions
scoresPerScenario_en <-
  scoresPerScenario[, c("label_en", weightedEstimateName)];
scoresPerScenario_nl <-
  scoresPerScenario[, c("label_nl", weightedEstimateName)];

### Round the scores
scoresPerScenario_en[, weightedEstimateName] <-
  round(scoresPerScenario_en[, weightedEstimateName]);
scoresPerScenario_nl[, weightedEstimateName] <-
  round(scoresPerScenario_nl[, weightedEstimateName]);

### Pretty column names
names(scoresPerScenario_en) <- c("Policy model", "Overall score");
names(scoresPerScenario_nl) <- c("Beleidsmodel", "Totaalscore");

###-----------------------------------------------------------------------------
### Compute the scores and plots for all scenarios
###-----------------------------------------------------------------------------

scenarioScores <-
  lapply(
    names(scenarioDefinitions),
    function(x) {
      return(
        mdmcda::scenario_overview(
          scenario = scenarioDefinitions[[x]],
          scenarioLabel = scenarioLabels[x],
          multiEstimateDf = estimates$multiEstimateDf,
          estimateCol = weightedEstimateName,
          decisionOrder = decisionOrder,
          decisionLabels = decisionLabels,
          criterionOrder = criterionOrder,
          criterionLabels = criterionLabels,
          alternativeLabels = alternativeLabels,
          parentCriterionIds_by_childId =
            criteria$convenience$parentCriterionIds_by_childId,
          parentCriterionOrder = parentCriterionOrder,
          scoreBarchart_criteria_args = list(yLab = "Total score per outcome\n(sum of weighted estimates)",
                                             fill = "black"),
          scoreBarchart_decisions_args = list(yLab = "Total score per instrument\n(sum of weighted estimates)",
                                              fill = "black")
        )
      );
    }
  );
names(scenarioScores) <- names(scenarioDefinitions);

### Dutch versions
scenarioScores_nl <-
  lapply(
    names(scenarioDefinitions),
    function(x) {
      return(
        mdmcda::scenario_overview(
          scenario = scenarioDefinitions[[x]],
          scenarioLabel = scenarioLabels_NL[x],
          multiEstimateDf = estimates$multiEstimateDf,
          estimateCol = weightedEstimateName,
          decisionOrder = decisionOrder,
          decisionLabels = decisionLabels_NL,
          criterionOrder = criterionOrder,
          criterionLabels = criterionLabels_NL,
          alternativeLabels = alternativeLabels_NL,
          parentCriterionIds_by_childId =
            criteria$convenience$parentCriterionIds_by_childId,
          parentCriterionOrder = parentCriterionOrder,
          scoreBarchart_criteria_args = list(yLab = "Score",
                                             fill = "white"),
          scoreBarchart_decisions_args = list(yLab = "Score",
                                              fill = "white")
        )
      );
    }
  );
names(scenarioScores_nl) <- names(scenarioDefinitions);

###~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
###
### Determine order for displaying the criteria within their clusters. This is
### based on the decreasingly sorted scores in the optimal model.
###
### This should not be deleted, but only needs to be run once and then the
### results are copy-pasted at the top.
###
###~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# criteriaOrder <-
#   unlist(
#     lapply(
#       parentCriterionOrder,
#       function(clusterName) {
#         res <-
#           scenarioScores$optimal_scenario$byCriterion[
#             criteria$convenience$childCriteriaIds[[clusterName]],
#             ,
#             drop=FALSE
#           ];
#         return(
#           res$criterion_id[
#             order(
#               res[, weightedEstimateName],
#               decreasing = TRUE
#             )
#           ]
#         );
#       }
#     )
#   );
# dput(criteriaOrder);

###~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
###
### End of commented out fragment that should be preserved
###
###~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

###-----------------------------------------------------------------------------
### Create performance tables for each scenario
###-----------------------------------------------------------------------------

performanceTables <-
  lapply(
    unique(weightedEstimates$scenario_id),
    function(currentScenario) {
      return(tidyr::pivot_wider(
        weightedEstimates[weightedEstimates$scenario_id==currentScenario,
                         c("decision_id",
                           "criterion_id",
                           weightedEstimateName)],
        id_cols="decision_id",
        names_from="criterion_id",
        values_from=weightedEstimateName
      ));
    }
  );
Note: Using an external vector in selections is ambiguous.
i Use `all_of(weightedEstimateName)` instead of `weightedEstimateName` to silence this message.
i See <https://tidyselect.r-lib.org/reference/faq-external-vector.html>.
This message is displayed once per session.
names(performanceTables) <- unique(weightedEstimates$scenario_id);

Input

Outcomes and anchors

Outcome tree

cat("\n\n<div style='width: 70%'>\n\n");
cat(
  mdmcda::plot_criteria(
    criteria,
    labels = criterionLabels,
    show_weights = FALSE,
    renderGraph = FALSE,
    returnSVG = TRUE,
    outputFile = file.path(workingPath,
                           "criteria-tree-without-weights.pdf")
  )
);
%0 1 Outcomes 2 Use (prevalence & patterns) 3 Use in vulnerable populations 4 Prevalence (general population) 5 Use frequency and intensity by users 6 User health 7 Health damage 8 Social damage 9 Health benefits 10 Social benefits 11 Shift to other drugs 12 Quality of information about MDMA 13 Stigmatization of users 14 Ideological values 15 Policy consistent with liberal values 16 Policy consistent with conservative values 17 Crime 18 Criminalisation of users 19 MDMA-related small crime 20 Organized crime (MDMA-related) 21 Organized crime (not MDMA-related) 22 International trafficking of MDMA 23 Criminals exploitating vulnerable groups 24 Financial costs and benefits 25 State revenue (VAT) 26 State revenue (other) 27 Costs related to pollution (dumpings) 28 Costs related to health damage 29 Costs related to crime 30 Environmental protection 31 Environmental damage 32 International politics 33 International image of the Netherlands 34 Economic boycotts 35 International legal countermeasures 1->2 1->6 1->14 1->17 1->24 1->30 1->32 2->3 2->4 2->5 6->7 6->8 6->9 6->10 6->11 6->12 6->13 14->15 14->16 17->18 17->19 17->20 17->21 17->22 17->23 24->25 24->26 24->27 24->28 24->29 30->31 32->33 32->34 32->35
cat("\n\n</div>\n\n");

Anchors of each outcome

for (currentCriterion in criterionOrder) {
  
  ufs::cat0("\n\n##### ", criterionLabels[currentCriterion], "\n\n");
  
  if (producePlots) {
  
    ufs::knitAndSave(criteria$anchoringGraphs[[currentCriterion]],
                     path = workingPath,
                     figWidth = 5,
                     figHeight = 5,
                     figCaption = paste0("Outcome anchors for outcome ",
                                         criterionLabels[currentCriterion]));
    
  }
  
}
Use frequency and intensity by users
Figure 1: Outcome anchors for outcome Use frequency and intensity by users.

Figure 1: Outcome anchors for outcome Use frequency and intensity by users.

Use in vulnerable populations