How it does work:

Here is what we do: We define the counter

\newcounter{MyCounter}

then we add

\renewcommand\theMyCounter{\roman{MyCounter}}

after it, and it works. Thanks to Axel for his help on this.

 

How it does not work:

When you define a new counter like this

\newcounter{MyCounter}

And later use it like this

\refstepcounter{MyCounter}\label{example}
\roman{MyCounter}. Beispiel eins

And then reference it like this:

And now I reference an example \ref{example}. \end{document}

Then LaTeX still interprets it as something like

\newcommand\theMyCounter{\arabic{MyCounter}}

So it results in an arabic number. This is not what we want. We want to reference the counter in lower roman (or upper roman or alph or what have you), too.