Fourier Ptychography (FP) is a computational imaging technique which artificially increases the effective numerical aperture of an imaging system. In FP, the object is imaged using an array of Light Emitting Diodes (LEDs), each from a different illumination angle. A high resolution image is synthesized from this low resolution stack, typically using iterative phase retrieval algorithms. However, such algorithms are time consuming and fail when the overlap between the spectra of images is low, leading to high data requirements. At the crux of FP lies a phase retrieval problem. In this paper, we propose a Deep Learning (DL) algorithm to perform this synthesis under low spectral overlap between samples, and show a significant improvement in phase reconstruction over existing DL algorithms.