The problem you are having here is confusing two words that do not mean the same thing, despite the spelling and the situation making them appear similar
back
noun
1.
the rear part of the human body, extending from the neck to the lower end of the spine. The dorsal plane
HOWEVER, back can also be a verb (used with or without an object) :
back
verb (without an object)
23.
to lie at the back of; form a back or background for
You are incorrectly assuming that the "back" referred to in the saying is being used as a verb and therefore has an opposite. It is, however, being used as a noun and therefore cannot have an opposite. There is no opposite of the noun back, only if it is used as a verb.
There is no front or back to your back, there is simply your back. Much like you would say you were behind the tree, or behind the car. The confusion you are adding is that you add the possessive pronoun "your: (a form of the possessive case of you used as an attributive adjective)" this is because simply saying behind THE back could refer to any back. You have to make it clear that it is YOUR back which they are talking/going behind- not the back of the nearest building.
Yes, you can be in front of something or at the back of something, but some somethings have fronts and backs which are NOT relative to the point which you are viewing them from. Your back is still your back because it is a noun, it does not suddenly become the front of something just because that is where you are facing it from. It is a noun, it is non directional. Unlike the verb, which is.