whatId when using custom object

Unfortunately, the above solutions, to send to a contact record or to setToAddresses, will count towards Salesforce limit as explained here: https://developer.salesforce.com/docs/atlas.en-us.salesforce_app_limits_cheatsheet.meta/salesforce_app_limits_cheatsheet/salesforce_app_limits_platform_email.htm

You can send an unlimited amount of email to your org’s internal users, which includes portal users.

Internal user means the proper user record, not a random email address or Contact.

A solution which allows to send email to internal user, and yet allows the use of the whatId is by leveraging Messaging.renderStoredEmailTemplate(templateId, whoId, whatId). Click here for more details on this method: https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_classes_email_outbound_messaging.htm. This method allows to merge the fields without sending an email. Once fields are merged, it's possible to retrieve the subject and body text of the merged template and send an email to internal users without specifying a whatId.

Ultimately, this is what needs to be done

String templateId = ....;
String userId = ....;
String whatId = ....;

Messaging.SingleEmailMessage email = Messaging.renderStoredEmailTemplate(templateId, userId, whatId);

String emailSubject = email.getSubject();
String emailTextBody = email.getPlainTextBody();

email.setTargetObjectId(userId);
email.setSubject(emailSubject);
email.setPlainTextBody(emailTextBody);
email.saveAsActivity = false;

if(!Test.isRunningTest()) {
  Messaging.SendEmailResult[] r = Messaging.sendEmail(new Messaging.SingleEmailMessage[] {email});
}

This is a glaring hole that I'm sad to say has been here since the beginning. Particular frustrating since workflow alerts seem to have no problems merging a custom object template when sending to a user. You can work around it by creating a contact with the users email and then deleting it after the email has been sent.

private static void sendMyTemplate(Id userId, Id mergedObjectId) {
  User user = [select email, firstName, lastName from User where id = :userId];
  Contact tempContact = new Contact(email = user.email, firstName = user.firstName, lastName = user.lastName);
  insert tempContact;
  Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage();
  mail.setTargetObjectId(tempContact.id); 
  mail.setTemplateId(template.Id);
  mail.setWhatId(mergedObjectId); 
  Messaging.sendEmail(new Messaging.singleemailMessage[] {mail});
  delete tempContact;
}

You can get around this by grabbing the user's email address from their user record (you've already got their ID so that's a trivial query) and then using that with the setToAddresses method of SingleEmailMessage, then you shouldn't need to set the target object ID and as such will bypass this restriction.

It would seem that you can't set the WhatId (which you have) when the target object Id is a UserId. You can set it when the target is a contact or such. This post i found seems to confirm that hunch https://stackoverflow.com/questions/11105086/setwhatid-in-salesforce-using-email-template