• We first consider the static problem of allocating resources to ( i.e. , scheduling) multiple distributed application framework s, possibly with different priorities and server preferences , in a private cloud with heterogeneous servers. Several fai r scheduling mechanisms have been proposed for this purpose. We extend pr ior results on max-min and proportional fair scheduling to t his constrained multiresource and multiserver case for generi c fair scheduling criteria. The task efficiencies (a metric r elated to proportional fairness) of max-min fair allocations found b y progressive filling are compared by illustrative examples . They show that "server specific" fairness criteria and those that are b ased on residual (unreserved) resources are more efficient.
  • We consider a cloud based multiserver system consisting of a set of replica application servers behind a set of proxy (indirection) servers which interact directly with clients over the Internet. We study a proactive moving-target defense to thwart a DDoS attacker's reconnaissance phase and consequently reduce the attack's impact. The defense is effectively a moving-target (motag) technique in which the proxies dynamically change. The system is evaluated using an AWS prototype of HTTP redirection and by numerical evaluations of an adversarial coupon-collector mathematical model, the latter allowing larger-scale extrapolations.
  • In the following, we present example illustrative and experimental results comparing fair schedulers allocating resources from multiple servers to distributed application frameworks. Resources are allocated so that at least one resource is exhausted in every server. Schedulers considered include DRF (DRFH) and Best-Fit DRF (BF-DRF), TSF, and PS-DSF. We also consider server selection under Randomized Round Robin (RRR) and based on their residual (unreserved) resources. In the following, we consider cases with frameworks of equal priority and without server-preference constraints. We first give typical results of a illustrative numerical study and then give typical results of a study involving Spark workloads on Mesos which we have modified and open-sourced to prototype different schedulers.
  • Efficient and fair allocation of multiple types of resources is a crucial objective in a cloud/distributed computing cluster. Users may have diverse resource needs. Furthermore, diversity in server properties/ capabilities may mean that only a subset of servers may be usable by a given user. In platforms with such heterogeneity, we identify important limitations in existing multi-resource fair allocation mechanisms, notably Dominant Resource Fairness (DRF) and its follow-up work. To overcome such limitations, we propose a new server-based approach; each server allocates resources by maximizing a per-server utility function. We propose a specific class of utility functions which, when appropriately parameterized, adjusts the trade-off between efficiency and fairness, and captures a variety of fairness measures (such as our recently proposed Per-Server Dominant Share Fairness). We establish conditions for the proposed mechanism to satisfy certain properties that are generally deemed desirable, e.g., envy-freeness, sharing incentive, bottleneck fairness, and Pareto optimality. To implement our resource allocation mechanism, we develop an iterative algorithm which is shown to be globally convergent. Finally, we show how the proposed mechanism could be implemented in a distributed fashion. We carry out extensive trace-driven simulations to show the enhanced performance of our proposed mechanism over the existing ones.
  • We consider a cloud based multiserver system, that may be cloud based, consisting of a set of replica application servers behind a set of proxy (indirection) servers which interact directly with clients over the Internet. We address cloud-side proactive and reactive defenses to combat DDoS attacks that may target this system. DDoS attacks are endemic with some notable attacks occurring just this past fall. Volumetric attacks may target proxies while "low volume" attacks may target replicas. After reviewing existing and proposed defenses, such as changing proxy IP addresses (a "moving target" technique to combat the reconnaissance phase of the botnet) and fission of overloaded servers, we focus on evaluation of defenses based on shuffling client-to-server assignments that can be both proactive and reactive to a DDoS attack. Our evaluations are based on a binomial distribution model that well agrees with simulations and preliminary experiments on a prototype that is also described.
  • Many different caching mechanisms have been previously proposed, exploring different insertion and eviction policies and their performance individually and as part of caching networks. We obtain a novel closed-form stationary invariant distribution for a generalization of LRU and MRU caching nodes under a reference Markov model. Numerical comparisons are made with an "Incremental Rank Progress" (IRP a.k.a. CLIMB) and random eviction (a.k.a. random replacement) methods under a steady-state Zipf popularity distribution. The range of cache hit probabilities is smaller under MRU and larger under IRP compared to LRU. We conclude with the invariant distribution for a special case of a random-eviction caching tree-network and associated discussion.
  • This papers consists of two parts. The first is a critical review of prior art on adversarial learning, identifying some significant limitations of previous works. The second part is an experimental study considering adversarial active learning and an investigation of the efficacy of a mixed sample selection strategy for combating an adversary who attempts to disrupt the classifier learning.
  • Users of cloud computing platforms pose different types of demands for multiple resources on servers (physical or virtual machines). Besides differences in their resource capacities, servers may be additionally heterogeneous in their ability to service users - certain users' tasks may only be serviced by a subset of the servers. We identify important shortcomings in existing multi-resource fair allocation mechanisms - Dominant Resource Fairness (DRF) and its follow up work - when used in such environments. We develop a new fair allocation mechanism called Per-Server Dominant-Share Fairness (PS-DSF) which we show offers all desirable sharing properties that DRF is able to offer in the case of a single "resource pool" (i.e., if the resources of all servers were pooled together into one hypothetical server). We evaluate the performance of PS-DSF through simulations. Our evaluation shows the enhanced efficiency of PS-DSF compared to the existing allocation mechanisms. We argue how our proposed allocation mechanism is applicable in cloud computing networks and especially large scale data-centers.
  • Net neutrality on the Internet is perceived as the policy that mandates Internet Service Providers (ISPs) to treat all data equally, regardless of the source, destination, or type of transmitted data. In this work, we consider a scheme in which the decision makers of the market are two ISPs, one "big" Content Provider (CP), and a continuum of end-users. One of the ISPs is neutral and the other is non-neutral, i.e. she offers a premium quality to a CP in exchange for a side-payment. In addition, we assume that the CP can differentiate between ISPs by controlling the quality of the content she is offering on each one. In this part of the paper, we consider a scenario in which end-users are not locked in with the ISPs and can switch between ISPs easily. We formulate a sequential game, and show that there exists a unique Sub-game Perfect Nash Equilibrium (SPNE) for the game, where the CP pays the side-payment to the non-neutral ISP and offers her content with the premium quality. In addition, the CP does not offer her content on the neutral ISP. Thus, driving this ISP out of the market.
  • In a variety of applications, one desires to detect groups of anomalous data samples, with a group potentially manifesting its atypicality (relative to a reference model) on a low-dimensional subset of the full measured set of features. Samples may only be weakly atypical individually, whereas they may be strongly atypical when considered jointly. What makes this group anomaly detection problem quite challenging is that it is a priori unknown which subset of features jointly manifests a particular group of anomalies. Moreover, it is unknown how many anomalous groups are present in a given data batch. In this work, we develop a group anomaly detection (GAD) scheme to identify the subset of samples and subset of features that jointly specify an anomalous cluster. We apply our approach to network intrusion detection to detect BotNet and peer-to-peer flow clusters. Unlike previous studies, our approach captures and exploits statistical dependencies that may exist between the measured features. Experiments on real world network traffic data demonstrate the advantage of our proposed system, and highlight the importance of exploiting feature dependency structure, compared to the feature (or test) independence assumption made in previous studies.
  • In this paper, we study the problem of differential pricing and QoS assignment by a broadband data provider. In our model, the broadband data provider decides on the power allocated to an end-user not only based on parameters of the transmission medium, but also based on the price the user is willing to pay. In addition, end-users bid the price that they are willing to pay to the BTS based on their channel condition, the throughput they require, and their belief about other users' parameters. We will characterize the optimum power allocation by the BTS which turns out to be a modification of the solution to the well-known water-filling problem. We also characterize the optimum bidding strategy of end-users using the belief of each user about the cell condition.
  • We consider a simple two-player game involving a large incumbent and small entrant into a cellular wireless access provider marketplace. The entrant's customers must pay roaming charges. We assume that the roaming charges are regulated, because if they are dictated by the incumbent then they could be set so high so as to be a barrier to entry in the marketplace. The game is studied at its Nash equilibrium. A roaming charge is identified that is arguably fair in the sense that revenues for the access providers are proportionate to their infrastructure costs.
  • For a simple model of price-responsive demand, we consider a deregulated electricity marketplace wherein the grid (ISO, retailer-distributor) accepts bids per-unit supply from generators (simplified herein neither to consider start-up/ramp-up expenses nor day-ahead or shorter-term load following) which are then averaged (by supply allocations via an economic dispatch) to a common "clearing" price borne by customers (irrespective of variations in transmission/distribution or generation prices), i.e., the ISO does not compensate generators based on their marginal costs. Rather, the ISO provides sufficient information for generators to sensibly adjust their bids. Notwithstanding our idealizations, the dispatch dynamics are complex. For a simple benchmark power system, we find a price-symmetric Nash equilibrium through numerical experiments.
  • This is a variation of the two-sided market model of [10]: Demand D is concave in \tilde{D} in (16) of [10] So, in (5) of [10] and after Theorem 2, take the parametric case 0 < a <1. Thus, demand D is both decreasing and concave in price p, and so the utilities (U=pD) are also concave in price. Also, herein a simpler illustrative demand-response model is used in Appendix A and B.
  • The goal of this paper is to provide an insight into the equilibrium of the Internet market, when the current balance of the market is disrupted, and one of the ISPs switches to a non-neutral regime. We consider a content provider with a subscription revenue model and a continuum of end-users. The CP is also non-neutral, in the sense that she can charge users of different ISPs different subscription fees, and use this "leverage" to control the equilibrium outcome. Results reveal that the CP is able to control the non-neutral ISP to some extend. However, switching to a non-neutral regime by an ISP tips the balance of the market in favor of this ISP.
  • Flexibility in electric power consumption can be leveraged by Demand Response (DR) programs. The goal of this paper is to systematically capture the inherent aggregate flexibility of a population of appliances. We do so by clustering individual loads based on their characteristics and service constraints. We highlight the challenges associated with learning the customer response to economic incentives while applying demand side management to heterogeneous appliances. We also develop a framework to quantify customer privacy in direct load scheduling programs.
  • To respond to volatility and congestion in the power grid, demand response (DR) mechanisms allow for shaping the load compared to a base load profile. When tapping on a large population of heterogeneous appliances as a DR resource, the challenge is in modeling the dimensions available for control. Such models need to strike the right balance between accuracy of the model and tractability. The goal of this paper is to provide a medium-grained stochastic hybrid model to represent a population of appliances that belong to two classes: deferrable or thermostatically controlled loads. We preserve quantized information regarding individual load constraints, while discarding information about the identity of appliance owners. The advantages of our proposed population model are 1) it allows us to model and control load in a scalable fashion, useful for ex-ante planning by an aggregator or for real-time load control; 2) it allows for the preservation of the privacy of end-use customers that own submetered or directly controlled appliances.
  • We describe a distributed framework for resources management in peer-to-peer networks leading to golden-rule reciprocity, a kind of one-versus-rest tit-for-tat, so that the delays experienced by any given peer's messages in the rest of the network are proportional to those experienced by others' messages at that peer.
  • We formulate optimization problems to study how data centers might modulate their power demands for cost-effective operation taking into account three key complex features exhibited by real-world electricity pricing schemes: (i) time-varying prices (e.g., time-of-day pricing, spot pricing, or higher energy prices during coincident peaks) and (ii) separate charge for peak power consumption. Our focus is on demand modulation at the granularity of an entire data center or a large part of it. For computational tractability reasons, we work with a fluid model for power demands which we imagine can be modulated using two abstract knobs of demand dropping and demand delaying (each with its associated penalties or costs). Given many data center workloads and electric prices can be effectively predicted using statistical modeling techniques, we devise a stochastic dynamic program (SDP) that can leverage such predictive models. Since the SDP can be computationally infeasible in many real platforms, we devise approximations for it. We also devise fully online algorithms that might be useful for scenarios with poor power demand or utility price predictability. For one of our online algorithms, we prove a competitive ratio of 2-1/n. Finally, using empirical evaluation with both real-world and synthetic power demands and real-world prices, we demonstrate the efficacy of our techniques. As two salient empirically-gained insights: (i) demand delaying is more effective than demand dropping regarding to peak shaving (e.g., 10.74% cost saving with only delaying vs. 1.45% with only dropping for Google workload) and (ii) workloads tend to have different cost saving potential under various electricity tariffs (e.g., 16.97% cost saving under peak-based tariff vs. 1.55% under time-varying pricing tariff for Facebook workload).
  • Crowdsourcing allows to instantly recruit workers on the web to annotate image, web page, or document databases. However, worker unreliability prevents taking a workers responses at face value. Thus, responses from multiple workers are typically aggregated to more reliably infer ground-truth answers. We study two approaches for crowd aggregation on multicategory answer spaces stochastic modeling based and deterministic objective function based. Our stochastic model for answer generation plausibly captures the interplay between worker skills, intentions, and task difficulties and allows us to model a broad range of worker types. Our deterministic objective based approach does not assume a model for worker response generation. Instead, it aims to maximize the average aggregate confidence of weighted plurality crowd decision making. In both approaches, we explicitly model the skill and intention of individual workers, which is exploited for improved crowd aggregation. Our methods are applicable in both unsupervised and semisupervised settings, and also when the batch of tasks is heterogeneous. As observed experimentally, the proposed methods can defeat tyranny of the masses, they are especially advantageous when there is a minority of skilled workers amongst a large crowd of unskilled and malicious workers.
  • We study a problem of trust in a distributed system in which a common resource is shared by multiple parties. In such naturally information-limited settings, parties abide by a behavioral protocol that leads to fair sharing of the resource. However, greedy players may defect from a cooperative protocol and achieve a greater than fair share of resources, often without significant adverse consequences to themselves. In this paper, we study the role of a few vigilante players who also defect from a cooperative resource-sharing protocol but only in response to perceived greedy behavior. For a simple model of engagement, we demonstrate surprisingly complex dynamics among greedy and vigilante players. We show that the best response function for the greedy-player under our formulation has a jump discontinuity, which leads to conditions under which there is no Nash equilibrium. To study this property, we formulate an exact representation for the greedy player best response function in the case when there is one greedy player, one vigilante player and $N-2$ cooperative players. We use this formulation to show conditions under which a Nash equilibrium exists. We also illustrate that in the case when there is no Nash equilibrium, then the discrete dynamic system generated from fictitious play will not converge, but will oscillate indefinitely as a result of the jump discontinuity. The case of multiple vigilante and greedy players is studied numerically. Finally, we explore the relationship between fictitious play and the better response dynamics (gradient descent) and illustrate that this dynamical system can have a fixed point even when the discrete dynamical system arising from fictitious play does not.
  • Despite its existing incentives for leecher cooperation, BitTorrent file sharing fundamentally relies on the presence of seeder peers. Seeder peers essentially operate outside the BitTorrent incentives, with two caveats: slow downlinks lead to increased numbers of "temporary" seeders (who left their console, but will terminate their seeder role when they return), and the copyright liability boon that file segmentation offers for permanent seeders. Using a simple epidemic model for a two-segment BitTorrent swarm, we focus on the BitTorrent rule to disseminate the (locally) rarest segments first. With our model, we show that the rarest-segment first rule minimizes transition time to seeder (complete file acquisition) and equalizes the segment populations in steady-state. We discuss how alternative dissemination rules may {\em beneficially increase} file acquisition times causing leechers to remain in the system longer (particularly as temporary seeders). The result is that leechers are further enticed to cooperate. This eliminates the threat of extinction of rare segments which is prevented by the needed presence of permanent seeders. Our model allows us to study the corresponding trade-offs between performance improvement, load on permanent seeders, and content availability, which we leave for future work. Finally, interpreting the two-segment model as one involving a rare segment and a "lumped" segment representing the rest, we study a model that jointly considers control of rare segments and different uplinks causing "choking," where high-uplink peers will not engage in certain transactions with low-uplink peers.
  • In this paper, we introduce a scalable model for the aggregate electricity demand of a fleet of electric vehicles, which can provide the right balance between model simplicity and accuracy. The model is based on classification of tasks with similar energy consumption characteristics into a finite number of clusters. The aggregator responsible for scheduling the charge of the vehicles has two goals: 1) to provide a hard QoS guarantee to the vehicles at the lowest possible cost; 2) to offer load or generation following services to the wholesale market. In order to achieve these goals, we combine the scalable demand model we propose with two scheduling mechanisms, a near-optimal and a heuristic technique. The performance of the two mechanisms is compared under a realistic setting in our numerical experiments.
  • High fidelity simulation of large-sized complex networks can be realized on a distributed computing platform that leverages the combined resources of multiple processors or machines. In a discrete event driven simulation, the assignment of logical processes (LPs) to machines is a critical step that affects the computational and communication burden on the machines, which in turn affects the simulation execution time of the experiment. We study a network partitioning game wherein each node (LP) acts as a selfish player. We derive two local node-level cost frameworks which are feasible in the sense that the aggregate state information required to be exchanged between the machines is independent of the size of the simulated network model. For both cost frameworks, we prove the existence of stable Nash equilibria in pure strategies. Using iterative partition improvements, we propose game theoretic partitioning algorithms based on the two cost criteria and show that each descends in a global cost. To exploit the distributed nature of the system, the algorithm is distributed, with each node's decision based on its local information and on a few global quantities which can be communicated machine-to-machine. We demonstrate the performance of our partitioning algorithm on an optimistic discrete event driven simulation platform that models an actual parallel simulator.
  • In this paper, we consider medium access control of local area networks (LANs) under limited-information conditions as befits a distributed system. Rather than assuming "by rule" conformance to a protocol designed to regulate packet-flow rates (e.g., CSMA windowing), we begin with a non-cooperative game framework and build a dynamic altruism term into the net utility. The effects of altruism are analyzed at Nash equilibrium for both the ALOHA and CSMA frameworks in the quasistationary (fictitious play) regime. We consider either power or throughput based costs of networking, and the cases of identical or heterogeneous (independent) users/players. In a numerical study we consider diverse players, and we see that the effects of altruism for similar players can be beneficial in the presence of significant congestion, but excessive altruism may lead to underuse of the channel when demand is low.