"Most Serene" is an indicator of sovereignty, i.e. an independent country.
'Republic' is just an description of the form of government exercising juridical rights in a geographical unit, e.g. a specific town or region can be a "republic" if the magistrates & judges of that area are appointed by an elected council of its inhabitants, rather than being appointed by a feudal lord, count, bishop or monarch.
But having a "republican" government does not necessarily imply supreme jurisdiction. If the law is such that a case brought before the republic's courts has a right of appeal for review by a higher authority, e.g. to a the courts of a duke or monarch or emperor, then the republic is not, by definition, "sovereign". Or, more bluntly, it is not an independent state, at least not de jure.
"Most Serene" is the customary courtesy reserved to sovereigns and sovereigns only. "Most Serene Prince" or "Most Serene Highness" necessarily means a lord with supreme jurisdiction, i.e. a sovereign king.
You can call any lowly knight or count "Most Illustrious", or "Most Valorous" or "Most Honorable" or what you will, but you can never call them "Most Serene". It would be a scandalous breach of established courtesies. "Most Serene" is a courtesy reserved for the sovereign and not allowed to anybody else.
I am not sure how that came about, or why 'serenity' rather than some other term was chosen to be the exclusive sovereign courtesy. Perhaps because only a sovereign can judge without fear of reversal, that his judgment cannot be reversed or perturbed, i.e. 'serene'? That's speculation. I really don't know. All I know is that it happens to be the customary usage.
Consequently, the "Most Serene" appellation was also used by republics like Venice to make it utterly clear that they are not merely republics (form of government), but specifically [i]sovereign[/i] republics, and thus independent states, i.e. their jurisdiction is supreme, there is no appeal above their courts, no review by any royal or emperor.