I thought this would be a piece of cake - open up SharePoint Manager, get the XML and done. Unfortunately, that is not the case. The property DisplayFormat holds the value, and the XML produced seems fine but the field kept on displaying the time.
I then saw this article from Microsoft with the answer - the XML element is Format, NOT DISPLAYFORMAT.
Adding Format="DateOnly" to my field XML resolved the problem.