I need some advise on losing my weight and am I doing it right as I am losing weight ?

cutt alot of things down like I don't have any sweets or anything like that and I still eat ok but things like beans and some fish and I do alot and lots of workouts at the gym and ever week I lose 4lbs in weight and I talked to the gym people and they made me a sheet of workout I should do and now i'm losing 4lbs in weight thats alright right but sometime I can lose just 2 or 3 a week. And something is really bugs me before I work out I did a diet before which I only lost 1 or 2 pound a week but now i'm working out and losing more weight my uncle and my sister boyfriend don't want to know but I work so hard on it and my uncle son ever time I tell him oh I lost 3 or 4lb or lost 2 stone all I get is yeah well your still fat or i'll get you still got along way to go should I lister to them as I have lost 2 stone and need to lose 3 more stone