He has a chance to 'kick you out' once a day each time you visit.
Once you've been kicked out once for the day you remain 'kicked out' until reset I'm afraid ;(
Sometime he kicks you out on your first visit of the day, sometime it can be after your 50th, it's completely random.