Should I ask my boss not to come to retrospectives?

I might suggest that talking about this at the retro might be in order.

Retrospectives and the action items that come out of them are often focused on process and how to improve it. However, it can be easy to forget that retros are a part of the process as well, and it is worthwhile to tweak them if they're not working the way they should.

Now, it may a delicate conversation to have; I'm sure your boss won't love when it gets pointed out that the team has much better retros without him, and members of the team might not want attention drawn to them and the difference in their behavior when the boss is away. However, if you want to have open, honest retrospectives, broaching this topic might be a good first step. Treat it like any other potential action item, and try brainstorming with the team about how to fix it.

If the idea of talking about the issue during a retro (while your boss there) is completely unthinkable, well... that also goes a long way towards answering your question.

It sounds like you have team members from high power distance cultures. People may not speak up when the boss is in the room because their values require them to listen and follow, not to advise or lead. You may even notice it happening between junior and senior team members or between yourself and team members. Read more about power distance here: https://en.wikipedia.org/wiki/Power_distance

Cultures with high power distance often have strong collectivism which means individuals want to be part of a whole rather than stand out as individuals. This is contrary to other cultures who have strong individualism and low power distance values.

A short term solution may be to hold two retrospectives where you first collect feedback from the team, then anonymize and consolidate the feedback before presenting "the team's" recommendations to the boss and discussing his ideas with the team. This protects individuals from feeling "picked on" while it allow individualist people to get some limelight.

For a longer term solution, read how the airline industry train their teams to reduce power distance related problems. The ideal is to teach the team to have low power distance (free exchange of information among the team) and high collectivism (recognition and acceptance of team interdependence). https://en.wikipedia.org/wiki/Crew_resource_management

Ps. See Todd A. Jacobs' answer for a good analysis of the scrum process related questions you will have to deal with besides the cultural topics I mention above.

Analysis

There is a chilling effect when he is here. Am I overreacting? Should I just try to build up the confidence of the team? or should I ban my boss from the retrospective?

In my experience, this is a classic case of missing the forest for the trees, and mistaking process problems for interpersonal ones. Let's enumerate some of the issues that underpin your question.

1. You think there's a "chilling effect" that is negatively impacting the Scrum Retrospective event.
2. You are seeking validation from outside the team, rather than asking the team (individually, collectively, privately, or otherwise).
3. You yourself are afraid to communicate about a perceived issue within the retrospective, or directly with the people involved.
4. You have a "boss" on the team, rather than a Scrum Team member.
5. Your boss is unclear on the difference between Scrum Team member (regardless of Scrum role) and line management responsibilities.

In short, you definitely have a communication problem within your process, but it all seems to boil down to a lack of trust and honesty. Fundamentally, if you can't be honest within the team or about reporting out process impediments, then you have a Scrum implementation failure. Scrum simply can't function effectively in a fear-based organizational culture, regardless of whether that culture comes from the team members or "tone at the top."

Solutions

There is no silver bullet. Fear, uncertainty, and doubt will ruin any project management framework. This isn't a problem unique to Scrum. You either need to tackle the source of the team's fears head-on, accept things as they are, or move on as professionally as possible. To address the problem head-on, you and the rest of the team (including your boss) should keep the focus on the Scrum process rather than on interpersonal conflicts, whether real or perceived.

To kick-start the process improvement, you should take a hard look at yourself and your role as Scrum Master. If you perceive a problem, why don't you feel confident about collecting or facilitating honest feedback from the team? More importantly, if you're the Scrum Master (and therefore the person responsible for refereeing the Scrum process), why don't you feel safe in having an honest conversation with the "boss" (whatever his actual Scrum role may be) about his impact on the process?

If you can answer those questions honestly, you'll probably find your own answer. Based on past experience with dysfunctional Scrum implementations, the first step is almost always to improve communications. If you can't or won't do that yourself, then it's unrealistic to expect other team members (especially introverted, "junior," or inexperience team members) to do it either.

Fundamentally, if the whole team (including you) feels unsafe or insecure about being honest with one another, or communicating about process impediments with your organizational leadership, then it doesn't really matter whether your boss is the one at fault or not. The underlying issue is that the team collectively lacks cohesion, trust, situational awareness, and open communications. If the team can't collectively fix those things, then it can't fix the Scrum implementation either.

At the end of the day, you may not be able to change the social dynamics of the team. If that's the case, your project is likely to fail. If that happens, there will be plenty of blame to go around. On the other hand, if you decide to have an open and honest conversation as a team and your boss isn't willing to problem-solve along with everyone else, then he gets to keep both halves of a broken process.

As a Scrum Master, the only time when I would look to ban somebody from a retrospective is if the team asked me to do it.

My recommendation would be to first coach the team and the boss on the importance of making the retrospective a safe space. The team (or the boss) may then act of their own accord.

If that doesn't help I would make the team's behaviour when the boss is present a retrospective topic, referring back to the coaching.

The answer to this question depends a lot on the people in the team and on the boss.

Normally, a Scrum team should not contain people with special roles, especially a boss. The balance of power gets messed up and you can kiss self-organization goodbye because a boss will have a tendency to decide on matters, and want to have the last word in various situations, and decide on things. That's not how a Scrum team should work. In a Scrum team all are peers.

If this boss would be just that, a boss, the answer would be simple: ask the boss not to participate in retrospectives. The problem is that this boss is also a member of the team and, as you mentioned, still has a foot in the technical side and still completes some of the sprint items. You can't kick him out of the retrospectives because he is actually a member of the team and should have a voice. The problem is that his voice is louder or more important than the rest of the team members. How you solve this problem depends on the dynamics of the team.

As an aside, there is one extra issue with the boss being a team member: they are only part time, at least half of the time, as you mention. If the boss needs to take care of "boss matters" while the team depends on him for Sprint items... you have a problem. This causes issues with sprint plannings, with the work itself, or might even cause conflicts if the boss needs to take decisions that go against those of the team or those recommended by Scrum practice. And they can, since they are the boss.

Before you start doing anything, I suggest you discuss this with every team member, including the boss. Depending on the dynamics of the team, you can have these discussions transparently with everyone, or in private. It all depends on the people. You work with them, and you know them better and you should decide how it will be with everyone. Just to get a foot in the door, I would suggest that at your next retrospective, you as the Scrum Master, to ask everyone to fill in a Scrum assessment form just to evaluate your Scrum practices. Like a health check, for example. Have each team member put their name on the form and send it to you in private. You can then say you want to collect further feedback from people and now you have an excuse to have one-on-one meetings with everyone and see how everyone feels about the situation. You can now encourage some people to be more confident, you can ask some people to participate more, and you can tell your boss to be more mindful of how they behave with everyone else.

Aggregate all the results and conclusions and decide based on that, together as a team, what to do. You as a Scrum Master don't really have the power to exclude the boss from the retrospectives, but the team as a whole can decide to vote him out. Hopefully, your boss is open minded enough to accept the team's decision and understand their reasons.

Collect as much information as possible about how everyone is feeling and form a more complete picture before deciding on anything, as things can always turn out for the worst if your boss decides to use his different role in the team to impose his view on things.

If the presence of any team member is reducing the performance and productivity of any activity, remove the problem.

To expand on my answer a bit: a chilling effect brought on by a "superior" in the room is not uncommon. A lot of factors play a part in that, including the degree of "rank" gap between the boss and the team, cultural issues in the firm, lack of boss talent, previous interactions including consequences for raising concerns, and on and on. As a project leader, you're accountable for performance and that means by necessity that you manage down, up, and sideways and you remove the barriers for the team. That also means you have to have the most uncomfortable conversations with whatever stakeholder, properly seasoned with the appropriate level of political salt, when you have identified a root cause to a performance issue.

Since we do not know your boss, we cannot predict how this conversation would go but it is a conversation you must have. In projects, my view is we don't have time to "grow" people. You remove the offending cog, replace it with a new one, and turn the machine back on. In a more of an operations setting, you might take more time to work with the issue to see if you can move the needle. This is your choice to make. Right or wrong, my approach would be to tell the boss (s)he is interfering with the process and ask for an immediate departure from that process. Else, you could see if the boss would address this problem and actively solicit comments in a different way than before.