I have the same problem on my dedicated server when I join it with a fresh, new install of gmod. It downloads the map and a few files (materials and something like 1 lua, 10 text files and 10 "other")
However, it works perfectly if the player has already installed TTT. (except for the scoring screen not showing up BUT that's another story)