Depends on your age, you may be loosing a few years of career progression as opportunity cost. Like for example you already have 5 years work experience and in that 2 years you could've progressed into a manager but you went to pick fruits instead.

Also many employers will see that as a gap in your resume and it will be challenging to go back to your previous career after youve done picking fruits.