No. He's been in the TARDIS several times.

He went into the TARDIS in "The Three Doctors", and in "Mawdryn Undead" a Brigadier from 1977 and one from 1983 entered the TARDIS.

He also went into the TARDIS in "The Five Doctors" at the end so Rassilon could send him and the Second Doctor back to their original place in space and time.

In "The Paradise of Death" (audio story), the Brigadier again travels in the TARDIS.