We have deployed a cloud-hosted environment in our Azure subscription which is of type Build/Test. When we wanted to remote in and use Visual Studio Team Explorer, we have noticed that the self-service VM is slow when it tries to fetch changes from the server.
It was because the Dynamics 365 VSOAgent had already fetched our source code. When we tried to get the latest changes, VS tried to move the files from the VSOAgent folder to our target location under K:\AosService\PackagesLocalDirectory, but have been denied acces.
The solution was to purge the C:\DynamicsSDK\VSOAgent\_work\ folder contents, which has increased the server fetch speed dramatically.