Allow /var/log, /var/log/datastore and /var/log/dansguardian to be on different drives
In big (ish) deployments these take up a LOT of space, and disk time. Having the lot on one set of disks means that the disks can't keep up, and there's often problems with making the drive/array/partition big enough to store all the logs an organisation may need (e.g. keeping 12 months for compliance)
Smoothwall install currently is clever enough to spot two disks and provides the option of having the system on one, and /var/log on another.
If this option was simply extended to allow further seperation for /var/log/datastore and /var/log/dansguardian this would help in large installations where there are vast quantities of logs (and the database is also msasive).
I know this is also possible by a bit of Linux configuration by support, but have been told it would be unsupported. Sounds like a relatively simple, quick, OS based change to me.
James Tinmouth commented
Spoke to a support rep yesterday, and I understand the function of the database is going to be scaled back massively so it's possible that the space requirements aren't so onerous. However it still might be an idea for big deployments for speed, especially if summary reports are still going to come out of the database (and of course, the slaves therefore are still going to import everything into the database - even if the retention time is much less.
I don't think our cluster is very big, for example compared with how big one could be, and anything reporting wise takes an age (days) to churn out. I percieve this is where at least some of your competitors are making a killing (although I'm not at all convinced they can actually do the filtering as well in the first place).
James Tinmouth commented
Just to add, I think our current installation is pretty small - only 4 slaves - and our master manages to get behind by ~2.5 million records by the end of the day, spending the whole day tied up with disk I/O. It just about catches up overnight. Support recently rebuilt the database clearing it out, and it took approx 1.5 months to re-import 135 million records and we had no meaningful reporting for that time.
If we run a report - or even a simple grep of the /var/log/datastore for user activity, it takes ~3 days. In fact the report engine usually deletes the report before it's generated. That's with our current 3 month retention. We've just been told that it needs to be 12 months.
Allowing the above (again I think a relatively simple change) will at least give us (and your other customers as big and presumably some much bigger than us) some chance of getting some performance out of the reporting engine.