Out of all the drivers being updated, the graphics drivers are probably those that you’ll notice the most, as every refresh usually adds a bit more performance.
Additionally, other functions may have also changed.
You most likely won’t break your system if you don’t update your kernel for this exact reason, but sooner or later you’ll find programs and other packages that require a certain version of the kernel.
In case your system does have a problem with it, you should be able to choose a previous kernel from the boot menu so you can get back into your system.
Then you can delete the offending kernel and make a choice of staying with your current kernel or waiting until a working update appears.
For example, you could take the kernel, patch it up with lots of fixes, tweak other settings, strip out everything you won’t need, and then replace your original kernel with your final product, and it will run just fine (assuming it was done right).