What exactly was everyone marked with?
Kotarō: a drug that smells like Cape Jasmine. It can be used to produce genjutsu, but it was not used for that in this case.
Was it just water that used to be a part of Hyo?
Yes and No. Hyō produces Kotarō within her body, it's what she mixes with her water to produce genjutsu mists and the like.
I realize the smell plus lack of chakra is supposed to hide the fact that they were marked, but I am interested in what it exactly it is to see if there's any way Amari could know what it is with her enhanced senses. I don't care whether she can or not, if the technique was that well disguised then I'm fine with it, but I feel like it's still important to know what it is.
It doesn't really hide. It's a drug that smells like caped jasmine, and it's a chemical made
with chakra, but not made
out of chakra.
The smell of caped jasmine was indicative of the drug being spread in the air and that's really all there is to it.
Recognizing it would be more akin to recognizing the drug, which is possible because the drug is also used as a meditative eye drop. Itachi in canon used the drug to stave off his illness and extend his life. Sasuke also uses the drug to cope with the eye irritation caused by the awakening of his Mangekyo.
Itachi isn't sick and Sasuke is dead though, so there aren't really any active utilizes of the medicine anymore.
With that out of the way, including Hyō, there are three people in the world that can make Kotarō. However, the materials needed to make it are hard to come by. While it's possible that someone might recognize the drug, it's extremely unlikely.
Also if it isn't chakra based then how does she have any control over it and how does it help her? The only way I can imagine is if she uses it kind of like Kiba and Akamaru's dynamic marking, but that technique only worked due to their enhanced sense of smell. Not trying to nitpick, just trying to understand.
Yes, its quite similar to the way Kiba and Akamaru use dynamic marking, excellent analysis.
Once the candidates enter the second exam the rain will begin to wash away the drug and as they move around they'll leave behind traces of a chemical trail that Hyō will just follow.
However, she will be further dabbling with the trackers. She only wants to talk to leaf shinobi who were in the last exams, so she'll be acting again later to single them out so she can pick out their trails from all the others.