We study the design of minimum variance portfolio when asset returns follow a low rank factor model. Using results from random matrix theory, an optimal shrinkage approach for the isolated eigenvalues of the covariance matrix is developed. The proposed portfolio optimization strategy is shown to have good performance on synthetic data but not always on real data sets. This leads us to refine the data model by considering time correlation between samples. By updating the shrinkage of the isolated eigenvalues accounting for the unknown time correlation, our portfolio optimization method is shown to have improved performance and achieves lower risk values than competing methods on real financial data sets.