I was having the same issue after installing Big Sur. I solved it by doing the following:
Change Double Click speed to normal speed. Follow this step, Apple logo from top Mac Menu > System Preferences > Accessibility> Mouse & Trackpad > Now Change Double Click speed to normal on the slider.
Auto Disable or ignore Built-In Trackpad or Mouse when connecting to External or Third-Party Accessories. The last checkbox option is for Auto disable while using an external Trackpad or mouse. Uncheck “Ignore built-in trackpad when mouse or wireless trackpad is present“.