The SNP will hold an election among its members for a new party Leader. Sturgeon stays as leader and FM until that's completed.
Sturgeon then formally resigns, and Parliament (Holyrood) opens nominations for FM, open to any sitting MSP. The new SNP leader (or deputy, if the leader is not an MSP - very unlikely!) will be nominated, among others - if I remember right, the leader of the smallest party usually is nominated as well, so that there's the appearance of a contest. MSPs vote to decide who will be FM. The appointment of an FM requires Royal Assent, and therefore (similar to WM) it must be someone who's believed to be able to hold the confidence of the Parliament - so usually the leader of the party holding the most seats/the party of government or senior coalition partner.
IF no one makes it through all the hoops to become FM by the legal time limit (28 days after the FM's formal resignation) the Parliament is dissolved and the Presiding Officer (equivalent of the Speaker at WM) proposes a date for an "extraordinary general election". The monarch must approve this before Parliament disbands.
At any time, the Parliament can dissolve itself, but this requires a 2/3 vote of all MSPs - so 86 out of a possible 129, regardless of abstentions. Neither the FM nor the Scottish Government can call an extraordinary general election.
(This is from memory, so any legal bods, please feel free to correct!)