Born 10 September 1955 in the Kaduna State of Nigeria, Yakubu attended the Government College in Keffi from 1969 to 1973, obtaining a West African School Certificate. Thereafter, he attended the College of Arts and Science in Zaria until 1975, when he entered the Ahmadu Bello University there, studying chemical engineering. After graduating with a BA in 1979, he served various roles in refineries before becoming executive director of operations at NETCO in March 2005. He left that role in December 2006 to work at the Nigerian National Petroleum Corporation, first as general manager and ultimately as managing director and CEO. He assumed the role several months after a national fuel crisis in Nigeria, when the NNPC was widely perceived as a corrupt organization, and oversaw the resultant changes within the national institution.
In addition to his education in chemistry, he received a certificate in Strategy and Organization from Stanford University's Graduate School of Business Studies.