Would SBD be hard to do if you were gluten and dairy free?

No, you can absolutely do it. You will just need to cook grains that are gluten free, check labels about casein and look into other sources of cheese and the like (if you are interested in eating those things) like rice cheese, etc. SBD is really based on lots of veggies, protein from beans/animal sources/tofu, some fruits, and healthy grains. You can certainly do this, but as with any diet, you will have to read labels especially to avoid the casein.