Generating customized vector OSM map files from OpenStreetMap and other data?

I am looking for a way (e.g. tutorial?) to generate my own customized (offline) maps in OSM format. By “customized” I mean adding data (not just define a region of interest) to an existing OSM map (e.g. OpenTopoMap), then export the augmented map into a new .osm-file.

What I managed to do:

• load OSM maps (e.g. OpenTopoMap) in QGIS, download the canvas into an .osm-file and convert it to a sqlite db, and import points, polygons, polylines
• load other data into QGIS (e.g. manually generated (vector) points)
• export the canvas to tiles using QTiles either to a directory or to a .mbtiles-file. However, this is the critical point: this export is way too large and the resolution of the tiles is bad, so I am looking to export the canvas in the vectorized form

How can I augment maps in OSM format with own data and store it as a new (vector) .osm-file?

Preferably in QGIS, other programs are fine as long as they are free and run in Linux. The full style e.g. the shading in OpenTopoMap and the elevation curves should be preserved.