Content

.gitignore

@@ -7,6 +7,7 @@
 *.fls
 *.lof
 *.log
+*.lol
 *.lot
 *.out
 *.pdf

acronyms.tex

@@ -67,12 +67,12 @@
 
 \DeclareAcronym{wcc}{
   short = {WCC},
-  long  = {weakly connected component}
+  long  = {Weakly Connected Component}
 }
 
 \DeclareAcronym{as}{
   short = {AS},
-  long  = {autonomous system}
+  long  = {Autonomous System}
 }
 
 \DeclareAcronym{grpc}{
@@ -82,7 +82,7 @@
 
 \DeclareAcronym{nat}{
   short = {NAT},
-  long  = {network access translation}
+  long  = {Network Access Translation}
 }
 
 \DeclareAcronym{md5}{
@@ -95,4 +95,9 @@
   long  = {Membership Management},
 }
 
+\DeclareAcronym{dht}{
+  short = {DHT},
+  long  = {Distributed Hash Table},
+}
+
 % vim: set filetype=tex ts=2 sw=2 tw=0 et :

appendix.tex

@@ -10,6 +10,11 @@
 \addcontentsline{toc}{section}{List of Tables}
 \listoftables
 
+\clearpage{}
+
+\addcontentsline{toc}{section}{List of Listings}
+\listoflistings{}
+
 \clearpage
 
 % TODO: add to table of contents?

bibliography.bib

@@ -392,4 +392,23 @@
     note         = {\url{https://ia.cr/2006/104}},
 }
 
+@incollection{bib:baileyNextGen,
+  location = {Cham},
+  title = {Next Generation P2P Botnets: Monitoring Under Adverse Conditions},
+  volume = {11050},
+  isbn = {978-3-030-00469-9 978-3-030-00470-5},
+  url = {http://link.springer.com/10.1007/978-3-030-00470-5\_24},
+  shorttitle = {Next Generation P2P Botnets},
+  pages = {511--531},
+  booktitle = {Research in Attacks, Intrusions, and Defenses},
+  publisher = {Springer International Publishing},
+  author = {Böck, Leon and Vasilomanolakis, Emmanouil and Mühlhäuser, Max and Karuppayah, Shankar},
+  editor = {Bailey, Michael and Holz, Thorsten and Stamatogiannakis, Manolis and Ioannidis, Sotiris},
+  urldate = {2022-04-08},
+  date = {2018},
+  doi = {10.1007/978-3-030-00470-5_24},
+  note = {Series Title: Lecture Notes in Computer Science},
+  file = {Full Text:/home/me/Zotero/storage/UGX3MEA7/Böck et al. - 2018 - Next Generation P2P Botnets Monitoring Under Adve.pdf:application/pdf}
+}
+
 /* vim: set filetype=bib ts=2 sw=2 tw=0 et :*/

commands.tex

@@ -9,3 +9,21 @@
 
 % absolute value
 \DeclarePairedDelimiter\abs{\lvert}{\rvert}%
+
+% fancyref for listings
+\newcommand*{\fancyreflstlabelprefix}{lst}
+
+\fancyrefaddcaptions{english}{%
+  \providecommand*{\freflstname}{listing}%
+  \providecommand*{\Freflstname}{Listing}%
+}
+
+\frefformat{plain}{\fancyreflstlabelprefix}{\freflstname\fancyrefdefaultspacing#1}
+\Frefformat{plain}{\fancyreflstlabelprefix}{\Freflstname\fancyrefdefaultspacing#1}
+
+\frefformat{vario}{\fancyreflstlabelprefix}{%
+  \freflstname\fancyrefdefaultspacing#1#3%
+}
+\Frefformat{vario}{\fancyreflstlabelprefix}{%
+  \Freflstname\fancyrefdefaultspacing#1#3%
+}

content.tex

@@ -15,12 +15,17 @@ While in 2016 only \SI{77}{\percent} of German households had a broadband connec
 Their nature as small, always online devices---often without any direct user interaction---behind internet connections that are getting faster and faster makes them a desirable target for botnet operators.
 In recent years, \ac{iot} botnets have been responsible for some of the biggest \ac{ddos} attacks ever recorded---creating up to \SI{1}{\tera\bit\per\second} of traffic~\cite{bib:ars_ddos_2016}.
 
-A botnet is a network of infected computers with some means of communication to control the infected systems.
+%}}} motivation
-Centralized botnets use one or more coordinating hosts called \ac{c2} servers.
+
+\clearpage{}
+\section{Background}
+
+Botnets consist of infected computers, so called \textit{bots}, controlled by a \textit{botmaster}.
+\textit{Centralized} and \textit{decentralized botnets} use one or more coordinating hosts called \textit{\ac{c2} servers} respectively.
 These \ac{c2} servers can use any protocol from \ac{irc} over \ac{http} to Twitter~\cite{bib:pantic_covert_2015} as communication channel with the infected hosts.
 The abuse of infected systems includes several activities---\ac{ddos} attacks, banking fraud, as proxies to hide the attacker's identity, send spam emails\dots{}
 
-Analyzing and shutting down a centralized botnet is comparatively easy since the central means of communication (the \ac{c2} IP address or domain name, Twitter handle or \ac{irc} channel), can be extracted from the malicious binaries and are therefore publicly known.
+Analyzing and shutting down a centralized or decentralized botnet is comparatively easy since the central means of communication (the \ac{c2} IP addresses or domain names, Twitter handles or \ac{irc} channels), can be extracted from the malicious binaries or determined by analyzing network traffic and can therefore be considered publicly known.
 
 A coordinated operation with help from law enforcement, hosting providers, domain registrars, and platform providers could shut down or take over the operation by changing how requests are routed or simply shutting down the controlling servers/accounts.
 
@@ -44,25 +49,18 @@ To complicate take-down attempts, botnet operators came up with a number of idea
 %}}}fig:c2vsp2p
 
 A number of botnet operations were shut down like this~\cite{bib:nadji_beheading_2013} and as the defenders upped their game, so did attackers---the concept of \ac{p2p} botnets emerged.
-The idea is to build a distributed network without \acp{spof} in the form of \ac{c2} servers as shown in \autoref{fig:p2p}.
+The idea is to build a distributed network without \acp{spof} in the form of \ac{c2} servers as shown in \Fref{fig:p2p}.
 In a \ac{p2p} botnet, each node in the network knows a number of its neighbors and connects to those, each of these neighbors has a list of neighbors on its own, and so on.
 The botmaster only needs to join the network to send new commands or receive stolen data.
-Any of the nodes in \autoref{fig:p2p} could be the bot master but they don't even have to be online all the time since the peers will stay connected autonomously.
+Any of the nodes in \Fref{fig:p2p} could be the botmaster but they don't even have to be online all the time since the peers will stay connected autonomously.
 In fact there have been arrests of operators of \ac{p2p} botnets but due to the autonomy offered by the distributed approach, the botnet keeps communicating~\cite{bib:netlab_mozi}.
-Especially worm-like botnets, where each peer tries to find and infect other systems, the network can keep lingering for many years.
+Especially worm-like botnets, where each peer tries to find and infect other systems, can keep lingering for many years.
 
 This lack of a \ac{spof} makes \ac{p2p} botnets more resilient to take-down attempts since the communication is not stopped and botmasters can easily rejoin the network and send commands.
 
-%}}} motivation
 
-\section{Background}
+Bots in a \ac{p2p} botnet can be split into two distinct groups according to their reachability: publicly reachable peers, also known as \textit{superpeers}, and those, that are not (\eg{} because they are behind a \ac{nat} router or firewall).
-
+In contrast to centralized botnets with a fixed set of \ac{c2} servers, in a \ac{p2p} botnet, every superpeer might take the roll of a \ac{c2} server and \textit{non-superpeers} will connect to those superpeers when joining the network.
-%{{{ formal model
-\subsection{Definition and Formal Model of \Acs*{p2p} Botnets}
-
-Botnets consist of infected computers, so called \textit{bots}, controlled by a \textit{botmaster}.
-Bots can be split into two distinct groups according to their reachability: publicly reachable peers, also known as \textit{superpeers}, and those, that are not (\eg{} because they are behind a \ac{nat} router or firewall).
-In contrast to centralized botnets with a fixed set of \ac{c2} servers, in a \ac{p2p} botnet, every superpeer might take the roll of a \ac{c2} server.
 
 As there is no well known server in a \ac{p2p} botnet, they have to coordinate autonomously.
 This is achieved by connecting the bots among each other.
@@ -71,6 +69,10 @@ Since bots can become unavailable, they have to permanently update their neighbo
 This is achieved by periodically querying their neighbor's neighbors.
 This process is known as \textit{\ac{mm}}.
 
+\Ac{mm} comes in two forms: structured and unstructured~\cite{bib:baileyNextGen}\todo{explain structured and unstructured}.
+Structured \ac{p2p} botnets often use a \ac{dht} and strict rules for a bot's neighbors based on its unique ID.
+In unstructured botnets on the other hand, bots ask any peer they know for new peers to connect to, in a process called \textit{peer discovery}.
+
 The concept of \textit{churn} describes when a bot becomes unavailable.
 There are two types of churn:
 
@@ -82,6 +84,10 @@ There are two types of churn:
 
 \end{itemize}
 
+
+%{{{ formal model
+\subsection{Formal Model of \Acs*{p2p} Botnets}
+
 A \ac{p2p} botnet can be modelled as a digraph
 
 \begin{align*}
@@ -126,7 +132,7 @@ This has some advantages in that it is not possible for bot masters to detect or
 \citeauthor{bib:zhang_building_2014} propose a system of statistical analysis to solve some of these problems in~\cite{bib:zhang_building_2014}.
 Also getting access to the required datasets might not be possible for everyone.
 
-As most detection botnet mechanisms, also the passive ones work by building communication graphs and finding tightly coupled subgraphs that might be indicative of a botnet~\cite{bib:botgrep2010}. An advantage of passive detection is, that it is independent of protocol details, specific binaries or the structure of the network (\ac{p2p} vs.\ centralized)~\cite{bib:botminer2008}.
+As most botnet detection mechanisms, also the passive ones work by building communication graphs and finding tightly coupled subgraphs that might be indicative of a botnet~\cite{bib:botgrep2010}. An advantage of passive detection is, that it is independent of protocol details, specific binaries or the structure of the network (\ac{p2p} vs.\ centralized/decentralized)~\cite{bib:botminer2008}.
 
 \begin{itemize}
 
@@ -162,8 +168,8 @@ They cannot be used to create the botnet graph (only edges into the sensor node)
 
 %}}} active detection
 
-%{{{ anti-monitoring
+%{{{ monitoring prevention
-\subsubsection{Anti-Monitoring}
+\subsubsection{Monitoring Prevention Techniques}
 \todo{good title}
 
 The constantly growing damage produced by botnets has many researchers and law enforcement agencies trying to shut down these operations~\cite{bib:nadji_beheading_2013, bib:nadji_still_2017, bib:dittrich_takeover_2012, bib:fbiTakedown2014}.
@@ -174,7 +180,7 @@ Some of these countermeasures are explored by \citeauthor{bib:andriesse_reliable
 Successful take-downs of a \ac{p2p} botnet requires intricate knowledge over the network topology, protocol characteristics and participating peers.
 In this work we try to find ways to make the monitoring and information gathering phase more efficient and resilient to detection.
 
-%}}} anti-monitoring
+%}}} monitoring prevention
 
 %}}} detection techniques
 
@@ -196,6 +202,7 @@ In this work we try to find ways to make the monitoring and information gatherin
 
 
 %{{{ methodology
+\clearpage{}
 \section{Methodology}
 
 The implementation of the concepts of this work will be done as part of \ac{bms}\footnotemark, a monitoring platform for \ac{p2p} botnets described by \citeauthor{bib:bock_poster_2019} in \citetitle{bib:bock_poster_2019}.
@@ -205,11 +212,11 @@ The implementation of the concepts of this work will be done as part of \ac{bms}
 In an earlier project, I implemented different node ranking algorithms (among others \enquote{PageRank}~\cite{bib:page_pagerank_1998}) to detect sensors and crawlers in a botnet, as described in \citetitle{bib:karuppayah_sensorbuster_2017}.
 Both ranking algorithms use the \(\deg^+\) and \(\deg^-\) to weight the nodes.
 Another way to enumerate candidates for sensors in a \ac{p2p} botnet is to find \acp{wcc} in the graph.
-Sensors will have few to none outgoing edges, since they don't participate actively in the botnet.
+Sensors will have few to none outgoing edges, since they don't participate actively in the botnet, while crawlers have only outgoing edges.
 
 The goal of this work is to complicate detection mechanisms like this for botmasters by centralizing the coordination of the system's crawlers and sensors, thereby reducing the node's rank for specific graph metrics.
-The coordinated work distribution also helps in efficiently monitoring large botnets where one sensor is not enough to track all peers.
+The coordinated work distribution also helps in efficiently monitoring large botnets where one crawler is not enough to track all peers.
-The changes should allow the current sensors to use the new abstraction with as few changes as possible to the existing code.
+The changes should allow the current crawlers and sensors to use the new abstraction with as few changes as possible to the existing code.
 
 The final results should be as general as possible and not depend on any botnet's specific behaviour, but it assumes, that every \ac{p2p} botnet has some kind of \enquote{getPeerList} method in the protocol, that allows other peers to request a list of active nodes to connect to.
 
@@ -225,7 +232,7 @@ Further work might even consider autoscaling the monitoring activity using some
 %}}} methodology
 
 %{{{ primitives
-\subsection{Protocol Primitives}
+\subsection{Protocol Primitives}\label{sec:protPrim}
 
 The coordination protocol must allow the following operations:
 
@@ -245,6 +252,7 @@ The coordination protocol must allow the following operations:
 
 \mintinline{go}{requestTasks() []PeerTask}: Receive a batch of crawl tasks from the coordinator. The tasks consist of the target peer, if the crawler should start or stop the operation, when it should start and stop monitoring and the frequency.
 
+\begin{listing}
 \begin{minted}{go}
 type Peer struct {
   BotID string
@@ -259,52 +267,57 @@ type PeerTask struct {
   StopCrawling bool
 }
 \end{minted}
+\caption{Relevant Fields for Peers and Tasks}\label{lst:peerFields}
+\end{listing}
+\todo{caption not shown, link in list of listings is broken}
 
 %}}} primitives
 
 %}}} methodology
 
 %{{{ strategies
+\clearpage{}
 \section{Coordination Strategies}
 
 Let \(C\) be the set of available crawlers.
-Without loss of generality, if not stated otherwise, I assume that \(C\) is known when \ac{bms} is started and will not change afterward.
+Without loss of generality, if not stated otherwise, we assume that \(C\) is known when \ac{bms} is started and will not change afterward.
 There will be no joining or leaving crawlers.
 This assumption greatly simplifies the implementation due to the lack of changing state that has to be tracked while still exploring the described strategies.
 A production-ready implementation of the described techniques can drop this assumption but might have to recalculate the work distribution once a crawler joins or leaves.
+The protocol primitives described in \Fref{sec:protPrim} already allow for this to be implemented by first creating tasks with the \mintinline{go}{StopCrawling} flag set to true for all active tasks, run the strategy again and create the according tasks to start crawling again.
 
 %{{{ load balancing
 \subsection{Load Balancing}\label{sec:loadBalancing}
 
 This strategy simply splits the work into chunks and distributes the work between the available crawlers.
-The following sharding strategy will be investigated:
+The following load balancing strategies will be investigated:
 
 \begin{itemize}
-  \item Round Robin. See~\autoref{sec:rr}
+  \item Round Robin. See~\Fref{sec:rr}
 
-  \item Assuming IP addresses are evenly distributed and so are infections, take the IP address as an \SI{32}{\bit} integer modulo \(\abs{C}\). See~\autoref{sec:ipPart}
+  \item Assuming IP addresses are evenly distributed and so are infections, take the IP address as an \SI{32}{\bit} integer modulo \(\abs{C}\). See~\Fref{sec:ipPart}
     Problem: reassignment if a crawler joins or leaves
 \end{itemize}
+\todo{remove?}
 
 Load balancing in itself does not help prevent the detection of crawlers but it allows better usage of available resources.
-No peer will be crawled by more than one crawler and it allows crawling of bigger botnets where the current approach would reach its limit and could also be worked around with scaling up the machine where the crawler is executed.
+It prevents unintentionally crawling the same peer with multiple crawlers and allows crawling of bigger botnets where the uncoordinated approach would reach its limit and could only be worked around by scaling up the machine where the crawler is executed.
 Load balancing allows scaling out, which can be more cost-effective.
 
 \subsubsection{Round Robin Distribution}\label{sec:rr}
 
-\subsubsection{Even Work Distribution}\label{sec:ewd}
+This strategy distributes work evenly among crawlers by either naively assigning tasks to the crawlers rotationally or weighted according to their capabilities\todo{1 -- 2 sentences about naive rr?}.
-\todo{weighted round robin}
+To keep the distribution as even as possible, we keep track of the last crawler a task was assigned to and start with the next in line in the subsequent round of assignments.
-
+For the sake of simplicity, only the bandwidth will be considered as capability but it can be extended by any shared property between the crawlers, \eg{} available memory or processing power.
-Work is evenly distributed between crawlers according to their capabilities.
+For a given crawler \(c_i \in C\) let \(cap(c_i)\) be the capability of the crawler.
-For the sake of simplicity, only the bandwidth will be considered as capability but it can be extended by any shared property between the crawlers, \eg{} available memory, CPU speed.
+The total available capability is \(B = \sum\limits_{c \in C} cap(c)\).
-For a given crawler \(c_i \in C\) let \(B(c_i)\) be the total bandwidth of the crawler.
+With \(G\) being the greatest common divisor of all the crawler's capabilities, the weight \(W(c_i) = \frac{cap(c_i)}{G}\).
-The total available bandwidth is \(b = \sum\limits_{c \in C} B(c_i)\).
+\(\frac{cap(c_i)}{B}\) gives us the percentage of the work a crawler is assigned.
-The weight \(W(c_i) = \frac{B}{B(c_i)}\)\todo{proper def for weight} defines which percentage of the work gets assigned to \(c_i\).
+% The set of target peers \(P = <p_0, p_1, \ldots, p_{n-1}>\), is partitioned into \(|C|\) subsets according to \(W(c_i)\) and each subset is assigned to its crawler \(c_i\).
-The set of target peers \(P = <p_0, p_1, \ldots, p_{n-1}>\), is partitioned into \(|C|\) subsets according to \(W(c_i)\) and each subset is assigned to its crawler \(c_i\).
+% The mapping \mintinline{go}{gcd(C)} is the greatest common divisor of all peers in \mintinline{go}{C}, \(\text{maxWeight}(C) = \max \{ \forall c \in C : W(c) \}\).
-The mapping \mintinline{go}{gcd(C)} is the greatest common divisor of all peers in \mintinline{go}{C}, \(\text{maxWeight}(C) = \max \{ \forall c \in C : W(c) \}\).
+The algorithm in \Fref{lst:wrr}\todo{page numbers for forward refs?} distributes the work according to the crawler's capabilities.
-
-The following algorithm distributes the work according to the crawler's capabilities:
 
+\begin{listing}
 \begin{minted}{go}
 func WeightCrawlers(crawlers ...Crawler) map[string]uint {
   weights := []int{}
@@ -338,43 +351,16 @@ func WeightedCrawlerList(crawlers ...Crawler) []string {
   return crawlerIds
 }
 \end{minted}
+\caption{Pseudocode for weighted round robin}\label{lst:wrr}
+\end{listing}
 
 This creates a list of crawlers where a crawler can occur more than once, depending on its capabilities.
-The set of crawlers \(\{a, b, c\}\) with capabilities \(cap(a) = 3, cap(b) = 2, cap(c) = 1\) would produce \(<a, b, c, a, b, a>\), allocating two and three times the work to crawlers \(b\) and \(a\) respectively.
+To ensure better distribution, first every crawler is assigned one task, then, according to the capabilities, every crawler with a weight of 2 or more is assigned a task, and so on.\todo{better wording}
-
+The set of crawlers \(\{a, b, c\}\) with the capabilities \(cap(a) = 3\), \(cap(b) = 2\), \(cap(c) = 1\) would produce \(<a, b, c, a, b, a>\), allocating two and three times the work to crawlers \(b\) and \(a\) respectively.
-The following weighted round-robin algorithm distributes the work according to the crawlers' capabilities:
-
-\begin{minted}{go}
-work := make(map[string][]strategy.Peer)
-commonWeight := 0
-counter := -1
-for _, peer := range peers {
-  for {
-    counter += 1
-    if counter <= mod {
-      counter = 0
-    }
-    crawler := crawlers[counter]
-    if counter == 0 {
-      commonWeight = commonWeight - gcd(weightList...)
-      if commonWeight <= 0 {
-        commonWeight = max(weightList...)
-        if commonWeight == 0 {
-          return nil, errors.New("invalid common weight")
-        }
-      }
-    }
-    if weights[crawler] >= commonWeight {
-      work[crawler] = append(work[crawler], peer)
-      break
-    }
-  }
-}
-\end{minted}
-\todo{reference for wrr}
 
 
 \subsubsection{IP-based Partitioning}\label{sec:ipPart}
+\todo{don't use substrings, bit.int for 128 bit modulo, argumentation why this works}
 
 The output of cryptographic hash functions is uniformly distributed---even substrings of the calculated hash hold this property.
 Calculating the hash of an IP address and distributing the work with regard to \(H(\text{IP}) \mod \abs{C}\) creates about evenly sized buckets for each worker to handle.
@@ -385,6 +371,7 @@ While the \ac{md5} hash function must be considered broken for cryptographic use
 For the use case at hand, only the uniform distribution property is required so \ac{md5} can be used without scarifying any kind of security.
 
 This strategy can also be weighted using the crawlers capabilities by modifying the list of available workers so that a worker can appear multiple times according to its weight.
+The weighting algorithm from \Fref{lst:wrr} is used to create the weighted multiset of crawlers \(C_W\) and the mapping changes to \(m(i) = H(i) \mod \abs{C_W}\).
 
 \begin{figure}[H]
   \centering
@@ -399,6 +386,7 @@ This helps us in implementing the mapping \(m\) from above.
 
 By exploiting the even distribution offered by hashing, the work of each crawler is also evenly distributed over all IP subnets, \ac{as} and geolocations.
 This ensures neighboring peers (\eg{} in the same \ac{as}, geolocation or IP subnet) get visited by different crawlers.
+It also allows us to get rid of the state in our strategy since we don't have to keep track of the last crawler we assigned a task to, making it easier to implement and reason about.
 
 %}}} load balancing
 
@@ -406,7 +394,7 @@ This ensures neighboring peers (\eg{} in the same \ac{as}, geolocation or IP sub
 \subsection{Reduction of Request Frequency}
 
 The GameOver Zeus botnet deployed a blacklisting mechanism, where crawlers are blocked based in their request frequency~\cite{bib:andriesse_goz_2013}.
-In a single crawler approach, the crawler frequency has to be limited to prevent being hitting the request limit.
+In a single crawler approach, the crawler frequency has to be limited to prevent hitting the request limit.
 
 %{{{ fig:old_crawler_timeline
 \begin{figure}[h]
@@ -434,7 +422,7 @@ The amount of crawlers \(C\) required to achieve the frequency \(F\) without bei
 
 Taking advantage of the \mintinline{go}{StartAt} field from the \mintinline{go}{PeerTask} returned by the \mintinline{go}{requestTasks} primitive above, the crawlers can be scheduled offset by \(O\) at a frequency \(L\) to ensure, the overall requests to each peer are evenly distributed over time.
 
-Given a limit \(L = \SI{5}{\request\per 100\second}\), crawling a botnet at \(F = \SI{20}{\request\per 100\second}\) requires \(C = \left\lceil \frac{\SI{20}{\request\per 100\second}}{\SI{5}{\request\per 100\second}} \right\rceil = 4\) crawlers.
+Given a limit \(L = \SI{5}{\request\per 100\second}\)\todo{better numbers for example?}, crawling a botnet at \(F = \SI{20}{\request\per 100\second}\) requires \(C = \left\lceil \frac{\SI{20}{\request\per 100\second}}{\SI{5}{\request\per 100\second}} \right\rceil = 4\) crawlers.
 Those crawlers must be scheduled \(O = \frac{\SI{1}{\request}}{\SI{20}{\request\per 100\second}} = \SI{5}{\second}\) apart at a frequency of \(L\) for an even request distribution.
 
 
@@ -471,7 +459,7 @@ Those crawlers must be scheduled \(O = \frac{\SI{1}{\request}}{\SI{20}{\request\
 \end{figure}
 %}}} fig:crawler_timeline
 
-As can be seen in~\autoref{fig:crawler_timeline}, each crawler \(C_0\) to \(C_3\) performs only \SI{5}{\request\per 100\second} while overall achieving \(\SI{20}{\request\per 100\second}\).
+As can be seen in~\Fref{fig:crawler_timeline}, each crawler \(C_0\) to \(C_3\) performs only \SI{5}{\request\per 100\second} while overall achieving \(\SI{20}{\request\per 100\second}\).
 
 Vice versa given an amount of crawlers \(C\) and a request limit \(L\), the effective frequency \(F\) can be maximized to \(F = C \times L\) without hitting the limit \(L\) and being blocked.
 
@@ -500,18 +488,18 @@ Using the example from above with \(L = \SI{5}{\request\per 100\second}\) but no
 \end{figure}
 %}}} fig:crawler_timeline
 
-While the effective frequency of the whole system is halved compared to~\autoref{fig:crawler_timeline}, it is still possible to double the frequency over the limit.
+While the effective frequency of the whole system is halved compared to~\Fref{fig:crawler_timeline}, it is still possible to double the frequency over the limit.
 
 %}}} frequency reduction
 
 %{{{ against graph metrics
-\subsection{Creating Outgoing Edges for Crawlers and Sensors}
+\subsection{Creating Edges for Crawlers and Sensors}
 
 \citetitle*{bib:karuppayah_sensorbuster_2017} describes different graph metrics to find sensors in \ac{p2p} botnets.
 These metrics depend on the uneven ratio between incoming and outgoing edges for crawlers.
 One of those, \enquote{SensorBuster} uses \acp{wcc} since crawlers don't have any edges back to the main network in the graph.
 
-Building a complete graph \(G_C = K_{\abs{C}}\) between the crawlers by making them return the other crawlers on peer list requests would still produce a disconnected component and while being bigger and maybe not as obvious at first glance, it is still easily detectable since there is no path from \(G_C\) back to the main network (see~\autoref{fig:sensorbuster2} and~\autoref{fig:metrics_table}).
+Building a complete graph \(G_C = K_{\abs{C}}\) between the crawlers by making them return the other crawlers on peer list requests would still produce a disconnected component and while being bigger and maybe not as obvious at first glance, it is still easily detectable since there is no path from \(G_C\) back to the main network (see~\Fref{fig:sensorbuster2} and~\Fref{tab:metricsTable}).
 
 With \(v \in V\), \(\text{succ}(v)\) being the set of successors of \(v\) and \(\text{pred}(v)\) being the set of predecessors of \(v\), PageRank is recursively defined as~\cite{bib:page_pagerank_1998}:
 
@@ -520,7 +508,7 @@ With \(v \in V\), \(\text{succ}(v)\) being the set of successors of \(v\) and \(
   \text{PR}_{n+1}(v) &= \text{dampingFactor} \times \sum\limits_{p \in \text{pred}(v)} \frac{\text{PR}_n(p)}{\abs{\text{succ}(p)}} + \frac{1 - \text{dampingFactor}}{\abs{V}}
 \end{align*}
 
-For the first iteration, the PageRank of all nodes is set to the same initial value. When iterating often enough, any value can be chosen~\cite{bib:page_pagerank_1998}.
+For the first iteration, the PageRank of all nodes is set to the same initial value. \citeauthor{bib:page_pagerank_1998} argue that when iterating often enough, any value can be chosen~\cite{bib:page_pagerank_1998}.
 
 The dampingFactor describes the probability of a person visiting links on the web to continue doing so, when using PageRank to rank websites in search results.
 For simplicity---and since it is not required to model human behaviour for automated crawling and ranking---a dampingFactor of \(1.0\) will be used, which simplifies the formula to
@@ -535,28 +523,6 @@ Based on this, SensorRank is defined as
   \text{SR}(v) = \frac{\text{PR}(v)}{\abs{\text{succ}(v)}} \times \frac{\abs{\text{pred}(v)}}{|V|}
 \]
 
-\todo{percentage of botnet must be crawlers to make a significant change}
-
-Applying PageRank once with an initial rank of \(0.25\) once on the example graphs above results in:
-
-\todo{pagerank, sensorrank calculations, proper example graphs, proper table formatting}
-\begin{table}[H]
-  \centering
-\begin{tabular}{llllll}
-  Node & \(\deg^{+}\) & \(\deg^{-}\) & In \ac{wcc}? & PageRank & SensorRank \\
-  n0 & 0/0 & 4/4 & no & 0.75/0.5625 & 0.3125/0.2344 \\
-  n1 & 1/1 & 3/3 & no & 0.25/0.1875 & 0.0417/0.0313 \\
-  n2 & 2/2 & 2/2 & no & 0.5/0.375 & 0.3333/0.25 \\
-  c0 & 3/5 & 0/2 & yes (1/3) & 0.0/0.125 & 0.0/0.0104 \\
-  c1 & 1/3 & 0/2 & yes (1/3) & 0.0/0.125 & 0.0/0.0104 \\
-  c2 & 2/4 & 0/2 & yes (1/3) & 0.0/0.125 & 0.0/0.0104 \\
-\end{tabular}
-\caption{Values for metrics from~\autoref{fig:sensorbuster} (a/b)}\label{fig:metrics_table}
-\end{table}
-
-While this works for small networks, the crawlers must account for a significant amount of peers in the network for this change to be noticeable.
-The generated \(K_n\) needs to be at least as big as the smallest regular component in the botnet, which is not feasible.
-
 In our experiments on a snapshot of the Sality~\cite{bib:falliere_sality_2011} botnet obtained from \ac{bms} over the span of \daterange{2021-04-21}{2021-04-28} even 1 iteration were enough to get distinct enough values to detect sensors and crawlers.
 
 \begin{table}[H]
@@ -569,7 +535,7 @@ In our experiments on a snapshot of the Sality~\cite{bib:falliere_sality_2011} b
   4 & 0.24501068 & 0.46486353 & 0.13810930 & 0.41540997 \\
   5 & 0.24233737 & 0.50602884 & 0.14101354 & 0.45219598 \\
 \end{tabular}
-  \caption{Values for PageRank iterations with initial rank \(\forall v \in V : \text{PR}(v) = 0.25\)}\label{fig:pr_iter_table_25}
+  \caption{Values for PageRank iterations with initial rank \(\forall v \in V : \text{PR}(v) = 0.25\)}\label{tab:pr_iter_table_25}
 \end{table}
 
 \begin{figure}[H]
@@ -597,7 +563,7 @@ In our experiments on a snapshot of the Sality~\cite{bib:falliere_sality_2011} b
   4 & 0.49002136 & 0.92972707 & 0.27621861 & 0.83081993 \\
   5 & 0.48467474 & 1.01205767 & 0.28202708 & 0.90439196 \\
 \end{tabular}
-  \caption{Values for PageRank iterations with initial rank \(\forall v \in V : \text{PR}(v) = 0.5\)}\label{fig:pr_iter_table_5}
+  \caption{Values for PageRank iterations with initial rank \(\forall v \in V : \text{PR}(v) = 0.5\)}\label{tab:pr_iter_table_5}
 \end{table}
 
 \begin{figure}[H]
@@ -625,7 +591,7 @@ In our experiments on a snapshot of the Sality~\cite{bib:falliere_sality_2011} b
   4 & 0.73503203 & 1.39459060 & 0.41432791 & 1.24622990 \\
   5 & 0.72701212 & 1.51808651 & 0.42304062 & 1.35658794 \\
 \end{tabular}
-  \caption{Values for PageRank iterations with initial rank \(\forall v \in V : \text{PR}(v) = 0.75\)}\label{fig:pr_iter_table_75}
+  \caption{Values for PageRank iterations with initial rank \(\forall v \in V : \text{PR}(v) = 0.75\)}\label{tab:pr_iter_table_75}
 \end{table}
 
 \begin{figure}[H]
@@ -643,7 +609,7 @@ In our experiments on a snapshot of the Sality~\cite{bib:falliere_sality_2011} b
   \caption{SensorRank distribution with initial rank \(\forall v \in V : \text{PR}(v) = 0.75\)}\label{fig:dist_sr_75}
 \end{figure}
 
-The distribution graphs in \autoref{fig:dist_sr_25}, \autoref{fig:dist_sr_50} and \autoref{fig:dist_sr_75} show that the initial rank has no effect on the distribution, only on the actual numeric rank values.
+The distribution graphs in \Fref{fig:dist_sr_25}, \Fref{fig:dist_sr_50} and \Fref{fig:dist_sr_75} show that the initial rank has no effect on the distribution, only on the actual numeric rank values.
 
 For all combinations of initial value and PageRank iterations, the rank for a well known crawler is in the \nth{95} percentile, so for our use case, those parameters do not matter.
 
@@ -661,9 +627,9 @@ Looking at the data in smaller buckets of one hour each, the average number of s
 
 Since crawlers never respond to peer list requests, they will always be detectable by the described approach but sensors might benefit from the following technique.
 
-By responding to peer list requests with plausible data, one can move make those metrics less suspicious, because it produces valid outgoing edges from the sensors.
+By responding to peer list requests with plausible data, one can make those metrics less suspicious, because it produces valid outgoing edges from the sensors.
 The hard part is deciding which peers can be returned without actually supporting the network.
-The following candidates to place into the NL will be investigated:
+The following candidates to place into the neighbor list will be investigated:
 
 \begin{itemize}
 
@@ -675,10 +641,8 @@ The following candidates to place into the NL will be investigated:
 
 \end{itemize}
 
-Knowledge of only \num{90} peers leaving due to IP rotation would be enough to make a crawler look average in Sality.
+Knowledge of only \num{90} peers leaving due to IP rotation would be enough to make a crawler look average in Sality\todo{repeat analysis, actual number}.
-This number will differ between different botnets, depending on implementation details and size of the network.
+This number will differ between different botnets, depending on implementation details and size of the network\todo{upper limit for NL size as impl detail}.
-
-Adding edges from the known crawler to \num{90} random peers to simulate the described strategy gives the following rankings:\todo{table, distribution with random edges}
 
 
 %{{{ other sensors
@@ -704,6 +668,25 @@ Also this does not help against the \ac{wcc} metric since this would create a bi
   \caption{Differences in graph metrics}\label{fig:sensorbuster}
 \end{figure}
 
+Applying PageRank once with an initial rank of \(0.25\) once on the example graphs in \Fref{fig:sensorbuster} results in:
+
+\begin{table}[H]
+  \centering
+\begin{tabular}{llllll}
+  Node & \(\deg^{+}\) & \(\deg^{-}\) & In \ac{wcc}? & PageRank & SensorRank \\
+  n0 & 0/0 & 4/4 & no & 0.75/0.5625 & 0.3125/0.2344 \\
+  n1 & 1/1 & 3/3 & no & 0.25/0.1875 & 0.0417/0.0313 \\
+  n2 & 2/2 & 2/2 & no & 0.5/0.375 & 0.3333/0.25 \\
+  c0 & 3/5 & 0/2 & yes (1/3) & 0.0/0.125 & 0.0/0.0104 \\
+  c1 & 1/3 & 0/2 & yes (1/3) & 0.0/0.125 & 0.0/0.0104 \\
+  c2 & 2/4 & 0/2 & yes (1/3) & 0.0/0.125 & 0.0/0.0104 \\
+\end{tabular}
+\caption{Values for metrics from~\Fref{fig:sensorbuster} (a/b)}\label{tab:metricsTable}
+\end{table}
+
+While this works for small networks, the crawlers must account for a significant amount of peers in the network for this change to be noticeable.
+The generated \(K_n\) needs to be at least as big as the smallest regular component in the botnet, which is not feasible.
+
 
 %}}} other sensors
 
@@ -778,6 +761,7 @@ Experiments were performed, in which a fixed amount of random outgoing edges wer
 %}}} strategies
 
 %{{{ implementation
+\clearpage{}
 \section{Implementation}
 
 Crawlers in \ac{bms} report to the backend using \acp{grpc}\footnote{\url{https://www.grpc.io}}.
@@ -835,6 +819,7 @@ The server-side part of the system consists of a \ac{grpc} server to handle the
 %}}} implementation
 
 %{{{ conclusion
+\clearpage{}
 \section{Conclusion, Lessons Learned}\todo{decide}
 
 Collaborative monitoring of \ac{p2p} botnets allows circumventing some anti-monitoring efforts.
@@ -845,17 +830,22 @@ The current concept of independent crawlers in \ac{bms} can also use multiple wo
 %}}} conclusion
 
 %{{{ further work
+\clearpage{}
 \section{Further Work}
 
-Following this work, it should be possible to rewrite the existing crawlers to use the new abstraction.
+Following this work, it should be possible to rewrite the existing crawlers using the new abstraction.
 This might bring some performance issues to light which can be solved by investigating the optimizations from the old implementation and applying them to the new one.
 
 Another way to expand on this work is automatically scaling the available crawlers up and down, depending on the botnet size and the number of concurrently online peers.
 Doing so would allow a constant crawl interval for even highly volatile botnets.
 
+Placing churned peers or peers with suspicious network activity (those behind carrier-grade \acp{nat}) might just offer another characteristic to flag sensors in a botnet.
+This should be investigated and maybe there are ways to mitigate this problem.
+
 %}}} further work
 
 %{{{ acknowledgments
+\clearpage{}
 \section*{Acknowledgments}
 
 In the end, I would like to thank

report.tex

@@ -46,6 +46,7 @@ headsepline,
 \usepackage{chronology}
 
 % code listings
+\usepackage{listings}
 \usepackage{minted}
 
 % UTF-8 encoding
@@ -92,8 +93,10 @@ headsepline,
 % line spacing
 \usepackage[onehalfspacing]{setspace}
 
+\usepackage[plain]{fancyref}
+
 % TODO: start new page with each new section
-\AddToHook{cmd/section/before}{\clearpage}
+% \AddToHook{cmd/section/before}{\clearpage}
 % hyperlinks
 \usepackage[pdftex,colorlinks=false]{hyperref}