Joomla!, like many other web-something, is an evolving thing, and exciting new features pop-up almost on a daily basis, but sometimes there's an hidden price: increased complexity often translates to additional maintenance burden and additional risks, and so decreased overall security.
When talking about "security", it's impossible not to talk about backups, and this, by far, is the best and most important defensive/offensive technique, for one, backups helps you restore, but at the same time they may pose a security threat if the stuff backed up isn't protected accordingly. Let's say, for example, that a funny guy knows where to look for your backups: if he is really mad at you, one of the first things he will try to do is to steal your backup bits!
I discovered my hosting provider is offering me daily scheduled backups, and even if it's just for one day (the latter being overwritten), I suggest you take control and manage the overall backup plan by yourself (I think that by just "having a plan" may suffice).
Outside office, I usually do not have plenty of time for backups, so, for simplicity, I'm assuming just one case scenario:"site online and ready": you just launched your colored Joomla! site and you're satisfied with its overall content and layout (BTW, we are talking about your "oh-not-so-up-to-date" site and not your "daily" blog).
OK, so now your site is online and kicking: you surely wanna forget about it, but if there are issues (like your site being hax0red and such), you''ll have to be able to fix it, and, as time goes by, you wanna be sure to be able to bring it back on it's entirety!
How many backups? Your choice. I don't have time/love for e-mails telling me "backup this and backup that" (got plenty already @ work!), so my philosophy is pretty lame, cheap and simple: do your best to keep it up, try to improve overall security and always backup before you change something and after you're satisfied with the changes. Also, very important, strive to act responsibly in-between and you'll see everything is gonna be alright.
The basic form of backup I'm gonna deal with, it's the daily automated backup, included inside your cheap hosting plan, also known as "the hosting backup", and, despite it's pissing some weirdos by it's simplicity, I must admit it just works. Many providers offer you this form of "automated schedule", that simply makes a copy of your files + db into some folder; this folder is often positioned outside your site's root, and so is not browseable (you may not download its content through HTTP access), hence no robots can scan for its precious content. If your site is not really big stuff, by periodically downloading those files, may keep you on-business.
See? No plugins, no complexity, no additional risks! So assess your hosting provider capabilities and check if this applies to your case! If your hosting is cheaper than mine (see "no backups"), have a look at some form of plugin from the JED at your own risk.
one more thing...: if you happen to have a linux box sitting nearby and doing its "daily-nothing", you may schedule an automated command to be launched by crontab. 99% of hosting providers allow you to ftp to your site (even if this protocol is not encrypted), so you just schedule this bash command and you're done!
lftp -c 'open <server.IP.address>; user <your-ftp-username> <your-ftp-password>; mirror -e <the remote folder> </your/local/mnt/point/folder> ; quit'
Anyway, since Joomla! basically is a db + some bits inside some remote storage, when my hosting provider doesn't offer me an auto-thingie, first I check if my hosting provider allows me SSH acces, so I can develop some custom backup scripts and plan for some form of automation, then, if none apply, as a last resort, I rely on 3rd parties plugins.
There are plenty Joomla! backup techniques around, and even if you don't know, maybe your hosting provider is already servicing you with some basic form of backup, so check that first!
Keeping a copy somewhere (relatively safe), is priceless, so I strongly encourage you to check if that is the case.