I tried to find information about the "VisualStudio install cache" folder that Visual Studio installer creates, but was unable to find much. The folder takes a healthy couple of gigabytes of disk storage and, having the "cache" keyword in it, seems to be safe to delete. But deleting it will cripple Visual Studio updates. My Visual Studio 2017 started saying that its version is 18.104.22.168 and that everything is up to date, while for Visual Studio 2019 the update tool reported missing arguments. (Also, trying to start the same tool from command prompt with different arguments resulted in the maddening "Sorry, something went wrong" error message).
The main reason for this is the most important part of that folder, and that is Packages/Instances. This is where the installer seems to keep the information about your Visual Studio's current state. There's a folder for each installed Visual Studio instance - and the number seems to be random which makes sense because you may have multiple instances of the same (or same-ish) version of Visual Studio: you can have, for example, a release and a preview version of VS 2019. If this folder is lost, the installer won't know which versions you're on, your updates won't work, and the current installer refuses to install everything anew on top of an existing installation, so there's no easy way out. You will be forced to use install cleaner tools (probably without success), then delete the whole Visual Studio folder and install everything from scratch.
But Packages/Instances is not the only problem: when starting, the installer also seems to check for installation packages of installed extensions and won't work without them. These have semi-intelligible names but there's more than one folder for each extension, and some depend on external packages which have their own folders... Therefore, it's not easy to figure out which of the packages are important and which aren't. The unimportant ones can be deleted, the installer will complain about some but will know how to download and restore them so that everything is back to normal.
If you want to get rid of the excess packages, the regular way seems to be to run the installer with "--nocache" argument. (Or, use a more elaborate script from Microsoft). If that fails, you can make a backup of the existing content, remove everything you think you can do without (leaving the Instances folder is a must!) and then run the installer. It will complain that there was an error, and when you click Retry it will give the exact folder name for the first package it was unable to find so that you can restore it by copying the folder back. But you must repeat this for all missing packages - and I had more than ten, I think. At one point, clicking Retry will stop giving specific error messages (even if the installer reported an error), but the "install" button will become available and you will be able to click it to automatically download the rest of the missing stuff. I managed to shave off two thirds of my cache folder, reducing it from 3-4 gigs to slightly over 1 GB this way.