Do I have to use the lemon olive oil to cook the brioche french toast?

No, but it certainly gives the toast a nice lemony flavor. You can subsitute regular olive oil for the lemon flavored.