4 months is early
they need to lose the tongue thrust reflex, and should be able to sit and hold heads up ...
he likes rusk and pudding because it is sweet , and milk is sweeter tasting
around 16 weeks, the vast majority of babies have a growth spurt, and filling their small tummies with anything other than milk, is a bit of a waste, IMO, as a good milk feed is more calorific, filling and nutritious than a rusk ( full of sugar) and pudding ( also sugary )
IMO, you have started too early, and should go back to milk
if he is formula fed, then you could try a formula for hungrier babies, that might be more satisfying
you cannot do any harm by delaying weaning, but there is a possibility of causing harm : ie allergies, intolerances, through weaning early
babies watch everything you do, watching you eat does not mean they want to eat, they don;t know anything other than milk exists