Nanosized polymeric vesicles (polymersomes) assembled from ABA triblock copolymers of poly(N-vinylcaprolactam)-poly(dimethylsiloxane)-poly(N-vinylcaprolactam) (PVCL–PDMS–PVCL) are a promising platform for biomedical applications, as the temperature-responsiveness of the PVCL blocks enables reversible vesicle shrinkage and permeability of the polymersome shell at elevated temperatures. Herein, we explore the effects of molecular weight, polymer block weight ratios, and temperature on the structure of these polymersomes via electron microscopy, dynamic light scattering, small angle neutron scattering (SANS), and all-atom molecular dynamics methods. We show that the shell structure and overall size of the polymersome can be tuned by varying the hydrophilic (PVCL) weight fraction of the polymer: at room temperature, polymers of smaller hydrophilic ratios form larger vesicles that have thinner shells, whereas polymers with higher PVCL content exhibit interchain aggregation of PVCL blocks within the polymersome shell above 50 °C. Model fitting and model-free analysis of the SANS data reveals that increasing the mass ratio of PVCL to the total copolymer weight from 0.3 to 0.56 reduces the temperature-induced change in vesicle diameter by a factor of 3, while simultaneously increasing the change in shell thickness by a factor of 1.5. Finally, by analysis of the shell structures and overall size of polymersomes with various PVCL weight ratios and those without temperature-dependent polymer components, we bring into focus the mechanism of temperature-triggered drug release reported in a previous study. This work provides new fundamental perspectives on temperature-responsive polymersomes and elucidates important structure–property relationships of their constituent polymers.