You can do this too, without actually leaving the road. Organisations involved in using and maintaining places you can freely take your 4×4 are quite keen to make a distinction about this.
Using Britain’s network of byways, colloquially known as green-laning, is not off-roading. These are highways just like the M1 is a highway, subject to road laws and accessible to all traffic. These are roads.
There will be some in your area, if you live in England and Wales. (In Scotland, they don’t exist and access rights are different. In Ireland, they don’t either but some very minor rural roads are stony tracks.)
Byways vary from those accessible only by cars with a generous ride height and serious tyres to those you could drive a hatchback down if you didn’t care about it much. I like them all.
If you like using byways often, or just care about their future, I’d recommend joining an association like GLASS (The Green Lane Association) or for two-wheelers the TRF (Trail Riders Fellowship), because you’ll get more intel than an OS map can give you and it’s fair to say the future of access to many roads has been secured by these groups.
Your subs help defend the rights of drivers and riders to use roads as we have always been able to.
But if that’s not hardy enough and you want to give a 4×4 a proper workout, somewhere you can make a mess, get it stuck, get rescued, get stuck again, stop for a cup of tea and giggle while someone else gets stuck before towing them out, to hone a skill and find out just how far a 4×4 can go (and even after trying and seeing it dozens of times, I’m still surprised at that), hire some time in an off-road centre.