This paper explores the sum rate performance by optimizing spectrum allocation in an unmanned aerial vehicle (UAV)-to-UAV communication underlaid cellular network consisting of a BS and a UAV swarm. Each transmitting UAV selects to communicate with the BS or its nearest UAV according to a received signal strength (RSS)-based mode selection scheme. With the mode selection scheme, if the RSS at the BS is greater than a threshold 'theta', it will select cellular communication mode with the BS; otherwise, it will select UAV-to-UAV communication mode with its nearest UAV. Our mode selection scheme is general in the sense that it can cover the following networks as special cases: cellular networks when 'theta' = 0, and ad hoc networks when 'theta'' = +œ. We utilize the coalition formation game theory to model the sum rate maximization problem. To this end, we first formulate the sum rate maximization as a non-linear and non-convex optimization problem, which is generally difficult to solve. Then, we propose a coalition formation algorithm to solve the optimization problem by optimizing the spectrum allocation among UAV-to-UAV links. The algorithm is further proved to converge to a Nash-stable equilibrium. Finally, simulation results are provided to indicate the impact of critical system parameters on the sum rate performance.