How to change the font, axis and tick color in pgfplots

Regarding the axis lines (I assume that's what you meant), add the colour of your choice (and any other options) to the axis line style you already have there. I.e.

axis line style={->, green, very thick}

For the other three <foo> style={<colourname>}, where <foo> is xlabel, ylabel or tick. E.g.

    xlabel style={blue},
    ylabel style={red},
    tick style={cyan, ultra thick}

Those colours look rather ghastly of course, but I'll leave the colour choice to you. :)

documentclass[tikz,border=5mm]{standalone}

% package for plots
usepackage{pgfplots}
usepgfplotslibrary{colorbrewer}
pgfplotsset{compat=newest}

begin{document}
begin{tikzpicture}
    begin{axis}[
        xmin=0,
        xmax=11100,
        ymin=0,
        ymax=5200,
        domain=0:1000,
        xlabel=Generation,
        ylabel=Fitness,
        axis line style={->, green, very thick},
        axis x line*=none,
        axis y line*=none,
        scaled x ticks=false,
        scaled y ticks=false,
        xtick={0,2000,...,11000},
        ytick={0,1000,...,5000},
        tick align=outside,
        every axis plot/.append style={
            line join=round,
            line cap=round,
            clip=false,    
            thick
        },
        tick align=outside,
        x tick label style={
            rotate=45,
            anchor=north east
        },
        axis on top,
        legend style={
            anchor=south,
            at={(0.5,1.1)},
            legend columns=2,
            draw=none,
            fill=none
        },
        cycle list/Dark2,
        xlabel style={blue},
        ylabel style={red},
        tick style={cyan, ultra thick}
        ]
 addplot{rnd};
    end{axis}
end{tikzpicture}
end{document}