Stir together the cocoa powder, sugar, salt, and water in a medium saucepan.
Bring to a boil and stir in the milk. Heat until just warm enough to enjoy. Remove from the heat and stir in your vanilla extract.
Divide between 4 cups and serve with marshmallows or whipped cream if desired.