For the same reason "left-wing" parties generally oppose the legalization of guns.

"Right-wing" people typically believe in the individual freedom to defend yourself from others, but they do not believe in the freedom to harm yourself. For "Left-wing" people, it's the other way around.

Both tend to take opposite sides on which issues require a security / safety to be prioritized over individual freedom, because their brains are wired to require freedom and security in different areas.

The core policy of right wing parties is to promote the interests of the wealthy - that's what defines them as right wing. However, in most societies, a majority of the population are not wealthy (depending on how you cut it) and thus unlikely to give wholehearted support to such policies.

One strategy to deal with this is to divide the population into opposing groups and set one's party up as the protector and voice for some of those groups against a threatening "other". That "other" can be defined in various ways: racially, nationally, by lifestyle, etc.

Drug use is a lifestyle choice that can be used to isolate such a group (in the same way that gay people were isolated and attacked for much of the 20th century).

The premise is false. In the United States, at least, marijuana prohibition was a mainstream position favored by both major political parties for decades. The party most noted in American politics for favoring marijuana legalization during those decades was the Libertarian party, a right-wing party.

Social conservatism focuses on preventing social behaviours that cause harm. Divorce, for example is taken to harm children. Drug use harms the individual as well as those who are socially close to that individual. Since marijuana is now proven to be a causative agent in mental illness* it is entirely within the traditional domain of social conservatism to keep it illegal.

*It is now incontrovertible that heavy use of cannabis increases the risk of psychosis. There is a dose-response relationship and high potency preparations and synthetic cannabinoids carry the greatest risk. It would be wise to await the outcome of the different models of legalisation that are being introduced in North America, before deciding whether or not to follow suit.

_{Colizzi M, Murray R. Cannabis and psychosis: what do we know and what should we do?. Br J Psychiatry. 2018;212(4):195-196. doi:10.1192/bjp.2018.1}

At least in the United States Republican party, I think it would be more fair to say that there is a diversity of opinion. At National Review, for example, which is widely considered a flagship of the conservative movement in the US, we have an anti-legalization article, but also more-or-less pro-legalization articles here and here.

I hate to say this, but incarceration is profitable, and at least in America the political class is plainly for sale:

The largest private prison corporations, Core Civic and GEO Group, collectively manage over half of the private prison contracts in the United States with combined revenues of $3.5 billion as of 2015.[...]

In 2017 private prison stocks for Core Civic and GEO Group more than doubled after the Department of Justice, under Sessions’ leadership, announced that it would be maintaining contracts with for-profit prisons. While the firms’ stock prices have since declined, in early 2018 they were substantially higher than their 2016 low.

Private prison companies have contributed millions to President Trump’s campaign and associated super PACs. Moreover, at least one prison company appears to be acting in the personal financial interest of President Trump. GEO Group changed the location of its annual meeting from a resort in Boca Raton, Florida to the Trump National Doral Golf Club in Miami. This club is reported to be the “single biggest contributor to Trump’s cash flow.”¹

As I said elsewhere the particular world view of people usually aligns well with their particular interests. The right-wing profiteers of the inhumane incarceration rate in the United States of course think that strict laws and strict enforcement lead to a better society. (Only sticklers would point out that this should be applied equally to tax evasion and insider trading.)

The war on drugs supplies the law enforcement and prison system with a stream of felons and inmates that's never-ending because drugs have been inextricably intertwined with the human existence from the beginning of times, and will ever be. Ending the criminalization of drugs faces the same difficulties as global disarmament: There is a political-industrial-cultural complex profiting from it, and the public is easily convinced of simple "solutions".

Somewhat oversimplified explanation: In many circles (especially right-wing ones) there was limited access to information concerning differences in harm level between different psychoactive substances. So the approach was simple - it's all terribly harmful and should be enforced harshly.

(I highly suspect this is mostly due to a lack of detailed information because I've seen multiple conservatives, who after seeing data concerning actual harm level in comparison to alcohol, changed their views. In the era of the internet, multiple right-wing views are spreading quite well, while on this specific issue people on the right-wing are grudgingly willing to accept an approach more associated with left-wing, so I really think it's an information access issue)

Setting this aside:

• The issue is rather controversial in right-wing circles, as for classical liberals (libertarians) drugs are a perfect example of individual freedom, while for the more religious right, they are more some demonic-like force incarnated in a chemical compound (yes, a bit exaggerated however on this issue one can easily get a forum flame war within right-wingers)
• Having high openness is statistically speaking a predictor of being on the left-wing AND of being more willing to experiment with at least some drugs, so I'd say it's not something really matching people on the right.
• Generally, right-wingers don't like ideas of any grey zones as undermining rule of law (it's sort of illegal, but actually not likely to be prosecuted) and trying to lower sentencing when already existing regulations are failing to discipline people - a huge part of the legalisation path in multiple countries sort of looked as improperly as possible for right-wingers, irrespective of the actual harm of the substance.

From the 2016 (US) Republican Platform, page 40:

Combatting Drug Abuse

The progress made over the last three decades against drug abuse is eroding, whether for cultural reasons or for lack of national leadership. In many jurisdictions, marijuana is virtually legalized despite its illegality under federal law. At the other end of the drug spectrum, heroin use nearly doubled from 2003 to 2013, while deaths from heroin have quadrupled. All this highlights the continuing conflicts and contradictions in public attitudes and public policy toward illegal substances. Congress and a new administration should consider the long-range implications of these trends for public health and safety and prepare to deal with the problematic consequences.

The platform puts marijuana under "drug abuse", albeit acknowledging it is on "the other end of the drug spectrum" as heroin. The last line emphasizes a conservative view about the long-term implications of drug use, possibly alluding to the long-term effects of marijuana usage.

When asking this question about parties, and why they oppose legalization, there are two potential aspects to "why": reasons for the policy as an end in itself, and reasons for the policy as a means to attaining and retaining power. My answer focuses on the latter, as in my experience there does not seem to be a right-wing consensus that drugs (and especially not marijuana) should be illegal. I think my view here is somewhat in line with your observation that drug prohibition is contrary to what you view as conservative values about the role of government.

Groups that hold power, and especially right-wing parties, thrive on rules with harsh penalties that forbid things large portions of the population are doing, that can be selectively enforced against members of groups they want to keep disempowered, including even members of "their own" who step out of line. It facilitates disenfranchisement of groups likely to vote (or, in a non-democratic system, otherwise organize and rise up) against them. Further, the ability to get by with breaking the rule as long as you're one of the "in group" builds feelings of loyalty and unity.

This does not mean every person who supports or identifies with a right-wing party shares the above view or motivation. Many don't actually support drug prohibition, or if they do, subscribe to one or more of the narratives politicians use in justifying it, such as concern that a culture of tolerance of drugs will lead to their children becoming drug addicts or that drug users are a threat to their lives or property.

While my view here is not based on this one account (rather built on a multitude of experiences and assimilation of sources over decades) and I didn't have this in mind when first writing the answer, John Ehrlichman's account of the origins of the drug war supports what I claim:

"The Nixon campaign in 1968, and the Nixon White House after that, had two enemies: the antiwar left and black people," former Nixon domestic policy chief John Ehrlichman told Harper's writer Dan Baum for the April cover story published Tuesday.

"You understand what I'm saying? We knew we couldn't make it illegal to be either against the war or black, but by getting the public to associate the hippies with marijuana and blacks with heroin. And then criminalizing both heavily, we could disrupt those communities," Ehrlichman said. "We could arrest their leaders. raid their homes, break up their meetings, and vilify them night after night on the evening news. Did we know we were lying about the drugs? Of course we did."

Source: https://www.cnn.com/2016/03/23/politics/john-ehrlichman-richard-nixon-drug-war-blacks-hippie/index.html

There's room for debate as to how biased this account is, but it matches what groups who claim to have been targeted have been saying since long before the Ehrlichman interview was published.

Regarding the general principle of selective enforcement of laws widely broken (including drug laws) against groups of people more likely to vote against right-wing parties, the ACLU of NJ has a report from 2015, and it's widely accepted that this kind of disparity exists. Moreover I don't think it's under serious doubt that enforcement tends to prevent (via incarceration) voting against right-wing candidates and tends to break down social and economic stability needed for affected groups to organize politically, or that their doing so wouldn't shift power away from the right.

Outside of the US, Duterte's Philippine Drug War seems to be a similar phenomenon of right-wing use of drug prohibition as a weapon against political opponents.

In terms of the Republican party, it's generally not something that's a major plank issue. Presently, many of the more moderate members are far more likely to support it, especially those of the Libertarian/Classic Liberal wing of the party. The chief reasons they don't press the matter in intraparty politics as those of the libertarian persuasion are trying to focus on the limited government reform platforms and away from the the legalize cannabis issue as one of the stigmas against libertarians is that they are "right wing potheads". As for the rest of the party's wing, there are either no real support one way or the other and the remaining who do want the ban already have it... it's generally easier to support a vote for "No" on an issue than "Yes" in any issue.

Personally, one of the best arguments I have heard is that current tests for usage are unable to determine how much THC is presently active in your system such that it is impairing your judgement (presently, the tests only look for it's presences in your system at all, since whether or not you are currently "high as a kite" really doesn't matter when it's use is illegal. Evidence that you used it at all is good enough for the guilty verdict... Knowing if you are driving under the influence requires knowing if what's in your system is lingering from when you smoked three days ago or is presently impairing your judgement.). Another argument is that the difference in chemical make up for medical cannabis and recreation cannabis can have different effects on the body, but this one is something I haven't verified by any objective measure.

I cannot offer any rationals for the Conservative Party in the UK as I don't follow UK politics closely enough to know the debate to speak to the platforms.