First off, don’t worry too much about the progress at this point. Classes always go slower at the start of the term than later in the year. At this stage, it is much more important to establish yourself with your classes so that you can make progress later.
Next, stop being ‘harsh’. Kids hate staff they see as harsh. They crave ‘fair’. They also like ‘predictable’ so work really really hard on your routines. For example, what does the start of your lesson look like? In my classes, there are always five retrieval questions up on the board as the class arrive. I hand piles of books to the first few pupils for them to give out, everyone else comes in, gets themselves sorted and immediately starts answering the questions in the back of their books because they do it every lesson so know what to do. This gives me time to take the register and deal with any problems (eg the kid without a pen or the child who’s broken their leg and needs a chair to prop it up on). We then talk through the answers - giving me a chance to give some early positive praise to the pupils who need it (probably the same ones that are asking you to change seats). At this point, I am absolutely consistent in insisting on silence. If someone speaks while another child is giving an answer, I make the answeree pause, “you’re doing brilliantly Jack but some people have forgotten to be polite” and then I wait. No names on the board. No shouting. Just wait. Maybe raise an eyebrow. “Thank you, back to Jack”. If they do it again, we wait again. No stress. No shouting. No ‘harsh’. Just wait. They’ll get the idea. If you really need to, add the names to the board at this second point, but don’t get bogged down in it. The important thing is the silence.
When the main task starts, keep it super simple in terms of structure (read this, answer this?) but relatively complicated in terms of challenge. This will mean that your able / focused kids will get on with it and you can go round giving loads of positive praise and picking up any issues. Don’t try anything fancy at the moment. No group work. No ‘latest fad off Twitter’, just straightforward tasks.
To be clear, I’m not claiming to be the world’s best teacher or to be suggesting anything groundbreaking here. I just thought it might help you to understand how experienced staff start their years.
In your situation, I’d try a ‘restart’ next week. Be stood at the door on arrival, “I realised last week’s seating plan didn’t work so I’ve done a new one, please find your seat then start the work on the board/on the sheet/on page 17 of the text book” After those first lessons, be stood there every lesson “good morning, work’s on the board, get yourself started”, “hello Jane, good maths lesson?”
Give it a few weeks of being super consistent and super predictable and you’ll probably feel much more in control 