The Terahertz Intensity Mapper (TIM) is designed to probe the star formation history in dust-obscured star-forming galaxies around the peak of cosmic star formation. This will be done via measurements of the redshifted 157.7 μm line of singly ionized carbon ([CII]). TIM employs two R∼250 long-slit grating spectrometers covering 240 to 420 μm. Each is equipped with a focal plane unit containing four wafer-sized subarrays of horn-coupled aluminum kinetic inductance detectors (KIDs). We present the design and performance of a prototype focal plane assembly for one of TIM's KID-based subarrays. The overall detector package must satisfy thermal and mechanical requirements, while maintaining high optical efficiency and a suitable electromagnetic environment for the KIDs. In particular, our design manages to strictly maintain a 50 μm air gap between the array and the horn block. The prototype detector housing in combination with the first flight-like quadrant were tested at 250 mK. A frequency scan using a vector network analyzer shows 823 resonance features, which represents 90% yield, indicating a good performance of our TIM detector wafer and the whole focal plane unit. Initial measurements also showed that many resonances were affected by collisions and/or very shallow transmission dips as a result of a degraded internal quality factor. This is attributed to the presence of an external magnetic field during cooldown. We report on a study of magnetic field dependence of the quality factor of our quadrant array. We implemented a Helmholtz coil to vary the magnetic field at the detectors by (partially) nulling earth's. Our investigation shows that the earth magnetic field can significantly affect our KIDs' performance by degrading the quality factor by a factor of two to five, well below those expected from the operational temperature or optical loading. We find that we can sufficiently recover our detectors' quality factor by tuning the current in the coils to generate a field that matches earth's magnetic field in magnitude to within a few μT. We emphasize that it is impractical to fly a Helmholtz coil on TIM and dynamically "null"earth's. Therefore, it is necessary to employ a properly designed magnetic shield enclosing the TIM focal plane unit. Based on the results presented in this paper, we set a shielding requirement of |B| 3 μT.