if the edge of ceiling fan to the L-box edging is about 2feet distance, that would be fine no shadow.
check distance between the downlight, worry the light not bright enough for both side without L-box light. Kelvyn is correct, you should use 6" downlight instead and both top and bottom too. try to use 28W, else i think will not bright enough.