Yes, if I do it like you describe it, it works. The file in itself is accessable without any issuses.
But, we use buttons and link those to the automatically created File Pages to let users download from those instead of having to create pages with only a download button.

We do have a cache plugin on the real site, however I tried on a complete new installation locally on my computer and the problem was still there.