No not confused, there are varous historical reports, (referenced for example here) to toe the line being from prize fighting (as well as up to scratch)
And such I always believed (I know, I know, started a sentence with a conjunction).
However, in trying to prove my point (is there anything worse than a petty pedant), I found two alternative possibilities.
During the days of sail, in the British Royal Navy, part of each Sunday involved the ritual of "divisions," akin to a formal inspection in modern militaries. At this time, the Royal Marines and each navy officer's division would form up at set locations on deck. As a guide for forming neat straight lines, sailors would stand with their toes along a seam of deck planking, and would thereby "toe the line."
OR
The lines between the two front benches in the House of Commons are traditionally two swords length apart. When politicians became two rowdy (hard to believe - how times change!) the speaker would call the to "toe the lines" in order to keep them apart and prevent any real damage being done.
You pays your money and takes your choice.