Enabling and disabling a QAction depending on feature selection

The error message is telling you that your call to iface.activeLayer() is returning None, therefore, your layer object is of NoneType, instead of a QgsVectorLayer.

Basic error handling would be to add a check e.g. if iface.activeLayer():, however if the active layer were a raster layer, you would get a similar error.

The other problem with your code snippet is that your check for selected features is static- it won't update on change of selection or change of active layer. To do this you need to employ signals and slots, and it's not as trivial as you might think, especially when you have vector and raster layers in a project.

Please see my example below which you can paste into an editor in the Python console and run. This enables/ disables a QAction dynamically as you change active layers and modify the feature selection of vector layers. Personally, I like self-contained, clean examples which don't leave loose ends especially redundant signal/slot connections. Therefore, I have put all the code inside a class which also adds a 'Remove' button to the tool bar. Clicking the 'Remove' action, removes the tool bar and cleans up after itself.

You can play around with this example in a project with multiple different types of layers loaded to see the behaviour.

I hope you find this helpful (and hopefully there are no bugs). I will leave it up to you to work out how you can adapt or use this example (if indeed to wish to) but it could be implemented in a plugin etc. fairly easily.

class addAction():
    
    def __init__(self, iface):
        self.iface = iface
        self.mw = self.iface.mainWindow()
        self.msg = QMessageBox(self.mw)
        self.layer = self.iface.activeLayer()
        self.tool_bar = QToolBar('New_toolbar', self.mw)
        self.action_run = QAction('Run', self.tool_bar)
        self.action_run.setEnabled(False)
        self.action_remove = QAction('Remove', self.tool_bar)
        self.mw.addToolBar(self.tool_bar)
        self.tool_bar.addAction(self.action_run)
        self.tool_bar.addAction(self.action_remove)
        self.action_run.triggered.connect(self.run_triggered)
        self.action_remove.triggered.connect(self.remove_toolbar)
        # set enabled status and signals/slots on initialisation
        if self.iface.activeLayer():
            if isinstance(self.layer, QgsVectorLayer):
                self.action_run.setEnabled(True)
                self.layer.selectionChanged.connect(self.selection_changed)
        self.iface.currentLayerChanged.connect(self.active_changed)
        
    def active_changed(self, new_layer):
        '''Checks for selected features, controls enabled status and
        connects selection changed signal to active layer'''
        if isinstance(new_layer, QgsVectorLayer):
            if isinstance(self.layer, QgsVectorLayer):
                self.layer.selectionChanged.disconnect(self.selection_changed)
            self.layer = new_layer
            if self.layer.selectedFeatureCount() > 0:
                self.action_run.setEnabled(True)
            else:
                self.action_run.setEnabled(False)
            self.layer.selectionChanged.connect(self.selection_changed)
        else:
            self.action_run.setEnabled(False)
    
    def selection_changed(self, selected, deselected):
        '''Checks for selected features and controls enabled status
        whenever selection changes'''
        if selected:
            self.action_run.setEnabled(True)
        else:
            self.action_run.setEnabled(False)
            
    def run_triggered(self):
        '''Called when run action clicked'''
        # Delete the lines below and put logic in this method to do
        # whatever you want when action button is clicked
        self.msg.setText('You clicked the Run action!')
        self.msg.show()
                
    def remove_toolbar(self):
        '''Cleans up toolbar, actions and signals/slots'''
        if isinstance(self.layer, QgsVectorLayer):
            self.layer.selectionChanged.disconnect(self.selection_changed)
        self.iface.currentLayerChanged.disconnect(self.active_changed)
        self.tool_bar.removeAction(self.action_run)
        self.tool_bar.removeAction(self.action_remove)
        self.mw.removeToolBar(self.tool_bar)
        del self.action_run
        del self.action_remove


A = addAction(iface)