On its own, horseradish tastes strong and spicy. Its heat is felt on the tongue and may bring tears to your eyes. When mixed with vinegar, horseradish still tastes strong and spicy, but it does not cause tears.