Jet substructure techniques are playing an essential role in exploring the
TeV scale at the Large Hadron Collider (LHC), since they facilitate the
efficient reconstruction and identification of highly-boosted objects. Both for
the LHC and for future colliders, there is a growing interest in using jet
substructure methods based only on charged-particle information. The reason is
that silicon-based tracking detectors offer excellent granularity and precise
vertexing, which can improve the angular resolution on highly-collimated jets
and mitigate the impact of pileup. In this paper, we assess how much jet
substructure performance degrades by using track-only information, and we
demonstrate physics contexts in which calorimetry is most beneficial.
Specifically, we consider five different hadronic final states - W bosons, Z
bosons, top quarks, light quarks, gluons - and test the pairwise discrimination
power with a multi-variate combination of substructure observables. In the
idealized case of perfect reconstruction, we quantify the loss in
discrimination performance when using just charged particles compared to using
all detected particles. We also consider the intermediate case of using charged
particles plus photons, which provides valuable information about neutral
pions. In the more realistic case of a segmented calorimeter, we assess the
potential performance gains from improving calorimeter granularity and
resolution, comparing a CMS-like detector to more ambitious future detector
concepts. Broadly speaking, we find large performance gains from
neutral-particle information and from improved calorimetry in cases where jet
mass resolution drives the discrimination power, whereas the gains are more
modest if an absolute mass scale calibration is not required.