Visual Studio appears to use the assembly name, not the file name, as the identity of references in the DTE. If a package includes an assembly whose file name is not the same as the assembly name then the reference will not be removed when uninstalling the nuget package.
A better approach would be to use the InternalName from System.Diagnostics.FileVersionInfo to locate the reference in the DTE and trying removing that.
A better approach would be to use the InternalName from System.Diagnostics.FileVersionInfo to locate the reference in the DTE and trying removing that.