How to keep column attribute when converting object class to Date in R

I have an object mydf:

mydf <- structure(list(USUBJID = structure(c("ID1", "ID2", 
"ID3", "ID4", "ID5", 
"ID6"), label = "ID"), MYDATE = structure(c("2012-03-06", 
"2013-04-09", "2014-06-11", "2011-03-24", "2018-01-08", "2010-03-14"
), label = "Date")), row.names = c("1", "2", 
"3", "4", "5", 
"6"), class = "data.frame")

> mydf
  USUBJID     MYDATE
1     ID1 2012-03-06
2     ID2 2013-04-09
3     ID3 2014-06-11
4     ID4 2011-03-24
5     ID5 2018-01-08
6     ID6 2010-03-14

sapply(mydf, class)
    USUBJID      MYDATE 
"character" "character" 

attributes(mydf$MYDATE)$label
[1] "Date"

When I convert mydf$MYDATE to class Date as follows:

mydf <- mydf %>% mutate(
    MYDATE = as.Date(MYDATE)
)

sapply(mydf, class)
    USUBJID      MYDATE 
"character"      "Date"

attributes(mydf$MYDATE)$label
NULL

It looses the label attribute. How can I convert it to Date, but without removing the label attribute?

mydf$MYDATE[] <- as.Date(mydf$MYDATE)
class(mydf$MYDATE) <- "Date"
attributes(mydf$MYDATE)$label
#[1] "Date"
class(mydf$MYDATE)
#[1] "Date"