Popularity is the determining factor on how fast peasants arrive/leave  your village.  

The number displayed at the top of the Popularity Information panel shows the overall popularity.  

This represents the total of all the popularity factors listed below it. By increasing or decreasing the individual  popularity factors you can determine the exact speed in which peasants arrive at or leave the castle.  

This can change if you run out of gold (for bribing), rice, vegetables, meat or tea, or if your population changes or if an  event is taking place. 

Peasants will stop arriving at your castle  if you do not have enough houses to house them. 

Tax - Taxing your peasants is one of the best ways to generate  gold. The more peasants you have working in your village the  more gold you will gain from taxes. However raising taxes makes you  unpopular. There may be times when you need to switch from taxing  your peasants to bribing your peasants. For example, if you need  to increase your immigration rate to quickly raise an army or if you  have run out of food and need an emergency popularity boost. It’s an  expensive move but your peasants will love you for it.

Rice Rations - Rice is the staple food enjoyed by your  peasants. Just by feeding your peasants normal rations you will  gain a popularity boost but increasing this to double or triple rations  will make you even more popular. If however you are running out of  rice you may prefer to opt for half rations, cutting the amount of rice  being consumed but also incurring the wrath of the peasants. Be  careful as running out of rice (or setting no rice rations) will make your  peasants extremely unhappy

Vegetable Rations/Meat Rations/Tea Rations - The peasants  see vegetables, meat and tea as luxurious additions to their rice  diet. The more vegetables, meat and tea you feed your peasants the  more popular you will be, just the same as rice. However, unlike rice, if you run out of vegetables, meat and tea, you will not incur negative  popularity.

Clothes Rations - Not only do your peasants like to eat well but  they also like to wear good quality clothes. Once your silkworm  economy is up and running you can increase your popularity by  increasing your clothes rations.

Housing Bonus - Simply put, the higher the quality of housing  you build, the happier your peasants will be (See Housing  Quality). The housing bonus can also be negatively affected by  overcrowding. If you do not have enough houses to support your  population (maybe they have been destroyed by an enemy) then this  will lead to overcrowding which will make you unpopular with your  peasants.  

Spirit - Your peasants are spiritual beings. The higher the  proportion of your houses and work buildings that are within  the radius of the temple, the greater the spiritual enlightenment and  the more popular you will be. 

Fear Factor - Do you want to be a good lord or bad lord?  Good lords receive a popularity boost. Bad lords suffer from a  popularity hit.