\documentclass[fleqn,10pt]{wlscirep}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{hyperref}

\graphicspath{{./assets/}} 
\usepackage{booktabs, longtable, adjustbox, rotating}
\usepackage{multirow, tabularx}

\usepackage{etoolbox}
\usepackage{xcolor}

\usepackage[sort, numbers]{natbib}	
\setcitestyle{square}


\usepackage{url}
\usepackage{bm, amsmath}
\usepackage{soul}
\usepackage{bbm}
\usepackage{hyperref}
\graphicspath{{./assets/}} 

\title{Supplemental Materials}

\begin{document}
The following supplemental sections primarily serve to provide a more detailed explanation of our methodology such that future researchers can replicate this work. We follow our methodology supplement with additional results.

\section*{Methods}
Our novel quantitative approach measures the causal impact between media outlets as they propagate quotations with specific sentiments on the topic of interest. Beyond the initial data curation process, this approach is automated with state-of-the-art text analytics and novel causal inference on networks. Our methodology has four main components, as shown in Figure \ref{fig:block_diagram}. First, we collect a dataset of quotes pulled from articles from prominent European digital media outlets and hand-label them with context-based sentiment and topic. Then, we extract matching quotes from the dataset using a novel and automated method by mapping each quote to a semantic space using the transformer model SBERT and then identifying matching quotes through clustering with HDBSCAN. Matching quotes are weighted by salience, such that quotes with better signal of influence are weighted more than those with lower signals. Next, we construct an influence network using each media outlet as a node and connecting edges based on historical \textit{quote following} (when an outlet publishes a quote identical to one from another outlet). Finally, we apply a novel network causal inference method to quantify causal impact between outlets.

\begin{figure}[ht]
	\centering
	\includegraphics[width=.75\linewidth]{{block_diagram.png}}
	\caption{Data-driven methodology for semi-automated media influence estimation}
	\label{fig:block_diagram}
\end{figure}

\subsection*{Suitability of Quotes as Instrument}
Our paper is unique among the literature for its use of quotations as the instrument for influence. This section expands on our justifications for using quotations. When an outlet features a quotation within an article, a decision is made by the authors and editors to feature this particular voice as a means to elucidate details of the article’s topic, or relevant opinions that provide context and insight. These quotations form the syntactical unit of the subsequent news coverage that readers consume \cite{krippendorf1980}. The authors have extracted quotations---to include paraphrased quotes---as a means to validly separate discrete units of content. Quotations are particularly well suited as valid syntactical units of traditional media with limited distortion when separated from the article’s context \cite{fico2008}. Examining the decisions of whom to quote highlights how outlets influence what issues news consumers pay attention to and what positions they consider, a process referred to as \textit{agenda setting} \cite{mccombs1972}. In topics with competing narratives, the choice of whom to quote, which quotations to highlight, and how many voices from each side of an argument to include will determine the media coverage to which the reader is exposed, thus justifying quotations as the basic unit of analysis in outlet coverage leaning and influence.  

Quotes are a suitable unit of analysis from a practical standpoint, as well. The diffusion of quotes can be reliably and automatically detected. While an outlet may excerpt or otherwise change a quote, the quote remains recognizable. 


\subsection*{Data Collection and Labeling}
For our corpus we use data collected from a wide selection of the most prominent paid and unpaid European and Russian traditional media outlets based on third-party services like Alexa.com and SimilarWeb that track the popularity of webpages.  The genre of the corpus is geopolitical news coverage. Content was aggregated using standardized search strings filtering for diplomatic, informational, military, and economic content.\footnote{One search string example is:  ("NATO" OR (( "USA" OR "America" OR "European Union" OR "Latvia" OR "Lithuania" OR "Estonia" OR "Baltics" OR "Ukraine" OR "Poland" OR "Balkans" OR "Turkey" OR "Georgia" OR "Greece" OR "Serbia" OR "Croatia" OR "Montenegro" OR "Bosnia" OR "Finland" OR "Azerbaijan" OR "Armenia") AND ("rocket" OR "missile" OR "missiles" OR "corruption" OR "nuclear" OR "gas" OR "gazprom" OR "humanitarian aid" OR "human rights activist" OR "human rights" OR "refugees" OR "protest" OR "demonstration" OR "freedom of the press" OR "terrorism" OR "ISIS" OR "Arctic" OR "intercept" OR "interception" OR "exercises" OR "elections" OR "aircraft carrier" OR "military")))} The data set draws more heavily from countries in Eastern Europe than Western Europe but includes the most prominent outlets in 24 countries. Though news consumption preferences vary considerably within every country in Europe, the dataset has selected the most prominent outlets in select countries to approximate the news coverage of geopolitical issues that an average European is exposed to on a regular basis. As such, we have included a small number of U.S.-based outlets to capture the borderless nature of media consumption in Europe. For those outlets based in Russia, the data set includes media affiliated with the Russian government as well as media that is not.

To understand the underlying content and sentiments in each news article, we extract all quotations and label each quotation with who the quoted speaker is, a label of the topic based on a discrete set of geopolitical issues, and a label of the sentiment of that quote toward a particular entity. For example, a quote might be labeled with speaker Vladimir Putin, topic ``discussing Economic Sanctions,'' and sentiment-to-entity ``negative to U.S.''  
The coding was conducted by a team of nine analysts, three of whom coded Russian-language content in natural language and six of whom coded machine-translated English content of all other ingested European languages. The analysts assigned metadata tags such as directional sentiment (e.g. Positive to the U.S., Neutral to Russia, Negative to NATO, etc.) and topic (e.g. NATO Exercises, Missile Defense/Deployment, Russian Military Deployments, etc.) to each extracted quote from the ingested articles using a codebook that provided guidelines for what topics to use in a given circumstance. For instance, the guidance for assigning the topic ``Nuclear Cooperation'' reads as follows: ``Use for statements regarding U.S.-Russia nonproliferation cooperation and general cooperation on nuclear security issues.'' Only one topic and one sentiment could be assigned to each quote. Sentiment was assigned based on the analysts' informed insight on the topic at hand and familiarity with the public and political discourse at the time to interpret whether the quote shared criticism of the entity in question or revealed information that is unfavorable to the goals, status, or conditions of the entity in question. The same quote used across different articles had the same sentiment assigned unless outlets truncated or elongated the quote they published in a way that could alter the sentiment.

Given the numerous topics (144), directional sentiment entities (165), and coders (9), reliably achieving agreement among the coders presented a challenge. We took a designated coder approach to inter-coder reliability in which all analysts' coding was measured against one individual standard for agreement. To aid in this process, a comprehensive selection and hiring process was conducted to ensure that analysts had an appropriate level of understanding of the geopolitical focus of the dataset. All analysts were college-educated primarily in the fields of Political Science, International Relations, and International Conflict. Some had specific expertise regarding the Russia-Ukraine conflict, the NATO relations, or the Balkan region. Several of the Russian-enabled analysts previously lived in Russia, bringing them closer to the topic of analysis and expediting the training process of understanding project methodology (topics and sentiments), identifying political media affiliation, rhetoric and narratives used by various actors, and deciphering the true meaning and sentiment of each quotation. All content enriched by new analysts during their first month was assessed by a quality assurance team (which represents the designated coder) during multi-hour meetings with the analyst to review, discuss, and if necessary correct metadata assignments according to the methodology. This probationary period lasted up to 5 weeks, and new analysts were instructed not to add coded content to the finalized dataset until it was reviewed by the quality assurance team. This process ensured a complete learning process for the analyst to achieve consistency and agreement among the coders with the designated coder according to the established methodology in the codebook. Following the probationary period, each week a 10--15\% sample of every coder's output quota was analyzed for agreement with the designated coder. Metrics of agreement included sources, sentiment, locations, and other metadata fields. The quality assurance team met with each analyst to inform them of the results of the review process, to discuss ways to improve efficiency and agreement, and to identify metadata assignments that stood in need of correction.

\subsection*{Quotation Matching}
We used HDBSCAN to cluster quotes such that a cluster indicated a matched quote. We optimized and tested HDBSCAN parameters using a hand-labeled truth set of 1,668 quotes. Results from our clustering optimization can be seen in Figure \ref{fig:pr_curve}, with specific examples provided in Table \ref{table:quote_types}. At the optimal HDBSCAN parameters, we achieved 94\% recall and 78\% precision on our task of clustering quotes. In general, false positives tend to occur with quotes that use jargon, such as the quote discussing S-400s and F-35s. They also occur with subjects where there are not many quotes; these quotes may end up evenly-spaced in the embedded space and the density-based clustering will group them together. False negatives tend to occur on difficult paraphrase matching, or in areas where the same quote is used in slightly different form many times and the clustering algorithm fails to merge two nearby clusters. 

While there are no directly comparable benchmarks in the literature, we can compare our results to ``paraphrase identification'' results. Wang et al. report 89\% basic accuracy on paraphrase identification, where accuracy is the sum of true positives and true negatives divided by all labels \cite{wang2017}. We demonstrate 99.4\% accuracy on our task. Admittedly, our quote matching task is less challenging than paraphrasing; while some of quote matching is paraphrase identification, many of the quotes are near-exact matches. Mellace et al. proposed a similar sentence matching pipeline using SBERT and DBSCAN; they analyze novels and cluster sentences that captured similar relations between characters, such as characters speaking to or smiling at other characters \cite{mellace2020}.

\begin{figure}
	\centering
	\includegraphics[width=.5\linewidth]{{pr_curve.png}}
	\caption{Precision recall curve for quote matching clustering. At the X (our optimum), we achieve 78\% precision and 94\% recall. That is, 78\% of identified quote matches are true matches and we capture approximately 94\% of all quote matches.}
	\label{fig:pr_curve}
\end{figure}

\begin{table}
	\scriptsize
	\centering
	\resizebox{\textwidth}{!}{
		\input{{./assets/quote_matches.tex}}
	}
	\caption{Examples of quote matches, successful and unsuccessful. True positives were quotes that matched correctly. False positives are quotes that matched, but should not have because they are actually different quotes. False negatives are quotes that did not match, but should have because they are the same quote. }
	\label{table:quote_types}
\end{table}

\subsection*{Saliency-weighted Quote Influence Heuristics}
We apply three heuristics to transform sets of matched quotes into saliency-weighted quote influence. This section explains our assumptions more precisely. First, \textit{potential influence should only be attributed for quotes used after exposure}. We apply this heuristic by scaling our value by the proportion of matched quotes that came after possible exposure. Second, \textit{several outlets using the same quote suggests that the quote may be general knowledge}. We assume that the more outlets there are using the quote, the likelier it is to be general knowledge and the less signal the instance of quote following contains. Thus, we discount by some function of the number of outlets that used the quote. We represent this heuristic by function $g_1(x)$, a monotonically increasing and concave function. The final heuristic is \textit{diminishing signal on the number of times an outlet uses a quote}. More plainly, the first time an outlet uses a quote is more informative than the tenth time. We model this heuristic as a monotonically increasing and concave function, $g_2(x)$. Let $Q$ be the set of quotes outlet $i$ used, $S_q$ be the number of outlets that used quote $q$ and let $T_{q,i}$ be the time outlet $j$ used quote $q$. The following equation incorporates the above heuristics to transform the sets of matched quotes into saliency-weighted quote influence of outlet $i$ on outlet $j$:

\begin{equation}
	\kappa_{ij} = \sum_{q \in Q_i \cap Q_j} \frac{1}{g_1(S_q)} g_2(|q \in Q_j|) \frac{\left| \{ q \in Q_j \ : \ t_{q,i} \geq \min (t_{q}) \} \right|}{\sqrt{|q \in Q_j|}}
\end{equation}

Inside the summation, the first term models the general knowledge discount, the second term models the diminishing signal on the number of times an outlet uses a quote, and the third term is the fraction of quotes after initial exposure. For this analysis, we substitute the square root function as our monotonically increasing and concave function ($g_1(x) = g_2(x) = \sqrt{x}$), which gives us the following equation:

\begin{equation}
	\kappa_{ij} = \sum_{q \in Q_i \cap Q_j} \frac{1}{\sqrt{S_q}} \sqrt{|q \in Q_j|} \frac{\left| \{ q \in Q_j \ : \ t_{q,i} \geq \min (t_{q}) \} \right|}{\sqrt{|q \in Q_j|}}
	% \caption{Saliency-Weighted Quote Following}
	\label{eq:saliency_outcome}
\end{equation}

Figure \ref{fig:quote_saliency_explainer} provides an example of how saliency-weighting is applied on a potential quote influence of \textit{AFP} on \textit{Sputnik}.

\begin{figure}
	\centering
	\includegraphics[width=\linewidth]{{quote_saliency_explainer.png}}
	\caption{Saliency-weighted quote influence example. \textit{Sputnik} first used this quote on July 18th. \textit{AFP} first used it the next day on July 19th. \textit{Sputnik} then used the quote three more times. So, three out of four instances of \textit{Sputnik} using the quote came after exposure to \textit{AFP}. A total of four outlets used the quote. For this specific quote, \textit{AFP} has a saliency-weighted potential quote influence of $\frac{3}{4}$ on \textit{Sputnik}.}
	\label{fig:quote_saliency_explainer}
\end{figure}

\subsection*{Potential Outcome Model for Causal Impact Estimation}
The next step is to pair the processed quotes with a network causal inference framework \cite{kao2017, smith2021}. The saliency-weighted quote influence only represents \textit{potential influence} between the outlets. Actual causal impact of outlet $i$ on outlet $j$ is explicitly captured in the causal estimand: 

\begin{equation}
	\label{eq:}
	\zeta_{i,j} \buildrel{\rm def}\over= Y_j (\bm{z}_{i+}, \bm{A}) - Y_j (\bm{z}_{i-}, \bm{A})
\end{equation}

where $Y_j (\bm{z}_{i+}, \bm{A})$ represents potential outcome on outlet $j$’s saliency-weighted quotes, with the binary vector $\bm{z}_{i+}$ denoting outlet $i$ being the quote source on the influence network represented by the directed and weighted adjacency matrix $\bm{A}$. Outlet $i$’s causal impact on outlet $j$ is established by subtracting the counterfactual outcome of outlet $j$’s saliency-weighted quotes in the absence of outlet $i$ as the source, $Y_j (\bm{z}_{i-}, \bm{A})$. In summary, the difference between these two potential outcomes represents outlet $i$’s causal impact on outlet $j$’s outcome. The first potential outcome $Y_j (\bm{z}_{i+}, \bm{A})$ is observed as outlet $i$’s saliency-weighted quote influence on outlet $j$, in Equation \ref{eq:saliency_outcome}. However, the counterfactual outcome is not directly observable. They are imputed using a Poisson generalized linear mixed model (GLMM) fitted to the observed outcome $Y_j (\bm{z}_{i+}, \bm{A})$ on each outlet $j$. 

This outcome model captures the effects of each $n$-hop exposure to the quote source via the influence network, network confounders such as node degrees and node community membership, and heterogeneity between the outlets. The Poisson distribution models the saliency-weighted quote outcomes as Poisson processes. Practically, the saliency-weighting makes the observed outcomes non-discrete so they need to be rounded before being fitted to the model. To reduce the rounding effect and increase the resolution of the Poisson-distributed outcome, the observed outcomes are multiplied by a scalar of ten before rounding. This makes a unit of causal impact in this paper equal to 0.1 saliency-weighted quote defined in Equation \ref{eq:saliency_outcome}. The Poisson GLMM uses the canonical log-link function and includes linear predictor coefficients ($\tau, \gamma, \beta, \mu$), corresponding to the source indicator $z_j$, n-hop exposures $s_j^{(n)}$, the covariate vector $\bm{x_j}$, and the baseline outcome:

\begin{equation}
\begin{gathered}
	Y_j(\bm{z}, \bm{A}) \sim \text{Poisson}(\lambda_j)\\
	\log(\lambda_j) = \tau z_j + \sum_{n=1}^{N_\text{hop}} s_j^{(n)} \tau \prod_{k=1}^{n} \gamma_k + \bm{\beta}^T \bm{x}_j + \mu + \epsilon_j
\end{gathered}
\end{equation}


In the five effect terms, $\tau z_j$  represents the primary effect of the quote source, $\sum_{n=1}^{N_\text{hop}} s_j^{(n)} \tau \prod_{k=1}^{n} \gamma_k$ represents the accumulative network influence effect from each n-hop exposures  $s_j^{(n)}$ to the source, $\gamma_k$ (between 0 and 1) represents how quickly the effect decays over each additional $k$th hop, $\bm{\beta}^T \bm{x}_j$ is the effect of the unit covariates $x_j$ of network confounders including node degrees and node community membership (fitting the influence network to a blockmodel with five communities). These confounders are accounted for through covariate adjustment to disentangle actual causal impact from effects of homophily (birds of a feather flock together) and vertex degrees. Lastly, $\mu$ is the baseline effect on each outlet, and $\epsilon_j \sim \text{Normal}(0, \sigma^2_\epsilon = .1)$ provides independent and identically distributed variation for heterogeneity between the outlets. The amounts of social exposure at the nth hop are determined by $\bm{A}^{T^n} \bm{z}$. This captures quote diffusion via all exposure paths to the quote source over the influence network. Diminishing return of additional exposures is modeled using (elementwise) log-exposure, $s^{(n)} = \log (\bm{A}^{T^n}\bm{z} + 1)$.To account for the uncertainty of the influence network $\bm{A}$, it is jointly estimated with the model parameters ($\tau, \gamma, \beta, \mu$) through Markov Chain Monte Carlo (MCMC) and Bayesian regression.


\section*{Results}
\subsection*{Directionality of Influence and Impact Differentials}

We can also examine the directionality of influence by looking at total impact differential (Figure \ref{fig:impact_differential}). An outlet $i$’s total impact differential with respect to outlet $j$ is defined by the impact of $i$ on $j$ minus the impact of $j$ on $i$. In general, impact is symmetric with outlet $i$ influencing outlet $j$ about as much as outlet $j$ influences outlet $i$; this leads to an impact differential near zero, as is the case between \textit{Sputnik} and \textit{TASS English}. However, there are some outlets that tend to have asymmetric influence. \textit{RT Russian} stands out as a follower, or an outlet that is influenced more than it influences. Looking at the row for \textit{RT Russian} in Figure \ref{fig:impact_differential}, many of the values in the Russian area are quite negative and brown, which indicates Russian outlets exert more influence on \textit{RT Russian} than \textit{RT Russian} exerts on them. Equivalently, the column for \textit{RT Russian} is positive and blue. In the other direction, some outlets are influencers and tend to influence more than they are influenced. One example of an influencer outlet might be \textit{Sputnik} or \textit{Anadolu News Agency}, who generally demonstrate net influence on other outlets in their media ecosystem. While \textit{Sputnik} is only an influencer in the Russian sphere, \textit{Anadolu News Agency}’s positive differential persists with non-Turkish outlets as well, although to a lesser extent. \textit{RFE/RL} is more often influenced by Western outlets such as \textit{Agence France Presse}, \textit{ABC News}, and \textit{Reuters News}. This tracks well with \textit{RFE/RL}'s mission of bringing Western-style reporting to other countries, not being a prominent outlet in the Western media environment. That Turkish outlet \textit{Daily Sabah} has high asymmetric influence over \textit{Agence France Presse} could suggest \textit{Agence France Presse} relies on \textit{Daily Sabah} in reporting on matters pertaining to Turkey.

\begin{figure}
	\centering
	\includegraphics[width=.7\linewidth]{{impact_differential_table.png}}
	\caption{Total impact differentials of the top 19 outlets. These values represent the net impact of the row outlet on the column outlet. Negative, brown values mean the column outlet had more impact on the row outlet, while positive, purple values mean the row outlet had more impact than the column outlet. White, small magnitude values suggest symmetrical impact.}
	\label{fig:impact_differential}
\end{figure}

\subsection*{INF Case Study Extended}
Our paper references \textit{Kommersant} as an outlet that surprisingly has 3x as much pro-U.S. outcome as pro-Russia outcome despite publishing roughly the same number of pro-U.S. quotes as pro-U.S. quotes. Upon closer examination, \textit{Kommersant}’s pro-Russia quotes were not picked up by other Russian outlets, which limited \textit{Kommersant}’s pro-Russia impact. Figure \ref{fig:kommersant} displays the top few quotes, along with other relevant quote following statistics.

\begin{figure}
	\centering
	\includegraphics[width=.75\linewidth]{{kommersant_quotes_2.png}}
	\caption{Examples of top followed quotes for \textit{Kommersant}. Many of the top quotes were negative to Russia, which explains why \textit{Kommersant} was pro-U.S. over the nuclear treaty topic.}
	\label{fig:kommersant}
\end{figure}

\subsection*{Relationship Between Volume and Impact}

\begin{figure}
	\centering
	\includegraphics[width=.45\linewidth]{{total_quotes_v_total_impact.png}}
	\includegraphics[width=.45\linewidth]{{quote_slant_v_impact_slant.png}}
	\caption{Relationship between quotes and impact. The left figure shows the relationship between total quotes and impact, while the right figure shows the relationship between quote slant and impact. Points are sized according to the x axis and colored according to the y axis. Pro-Russia slant is indicated as negative values and pro-U.S. positive.}
	\label{fig:quotes_and_impact}
\end{figure}

One relevant question considers the relationship between quotes and impact: do outlets that publish frequently have the most impact? Moreover, what is the connection between quote slant (how many pro-U.S. quotes you publish versus pro-Russia quotes) and impact slant? Figure \ref{fig:quotes_and_impact} examines this relationship. The left plot demonstrates that outlets that published more quotes were generally more impactful. \textit{Sputnik News Service}, the most impact outlet, also published the most quotes. However, the correlation is not perfect. Consider \textit{RT Russian}, which had over twice as many quotes as \textit{Reuters News} and \textit{Agence France Presse}, yet had less impact. There is considerable variation in total impact among outlets with the same number of quotes, demonstrating the role that causal inference on networks play to more accurately quantify actual impact from simple activity count statistics. 

A similar story holds when we compare impact slant to quote slant. There is a linear relationship between the two, but there is significant variation in impact slant among outlets with similar quote slants. \textit{Sputnik News Service}, for example, is fairly neutral despite very pro-Russia quote slant, which can be explained by its pro-U.S. quotes being generally more impactful and propagated by Western media outlets, shown in Figure 6 of the main paper. 



\bibliography{main.bib}

\end{document}
