It should be possible to build in a gitian VM without access the internet. At the least there needs to be a documented workaround for this (maybe this exists but we just didn't find it). @sipa tried this yesterday and didn't succeed in building, even after manually triggering download and copying the cache, ie
cd bitcoin/depends
make download
cp sources/* $GITIAN_PATH/cache/common
... but the builder still tried to download e.g. the .jar files..