You should reinstall elementary to ensure that you get a fully functional system.
However, you can keep a list of the packages installed with
dpkg --get-selections > package-list-file
and backup this file and the packages in
/var/cache/apt/archives so that the packages not specific to elementary OS 0.4 would not be downloaded again. This will only work if you have not done any
apt-get clean /
apt-get autoclean since you have installed the packages.
After the installation of elementary OS 0.4 you can copy every
.deb file from you backup medium back to
/var/cache/apt/archives. Do not forget to add each PPA that you have used on the original system to the new one before as well. After these steps you restore packages with
sudo dpkg --set-selections < package-list-file
There might be some packages that you have to install/fix a little but it should work for most of the software installed.