topic Security and the DBA in Career Discussions

Security and the DBA

shaggyDBA — Thu, 15 Feb 2018 15:04:27 GMT

Maybe someone can shed some light on this for me.

I've been doing database-related security work for ~6 years and it seems there there is no "formal" guidance concerning databases. Sure there are plenty of consultants out there capable of assisting and there certainly is no shortage of guidance from the likes of DISA on what you should be doing, but I haven't seen anything formalized as to "here's how you should secure your database" and, more concerning "here's how you should manage a forensic investigation of a database in the event of a breach".

I'm from an Oracle background, so I know David Knox & company have a ton of material out there. I also know Pete Finnigan has recently published a fantastic book "Oracle Incident Response and Forensics". But I rarely see anything from (ISC)2 beyond data security recommendations.

Data generation and consumption is increasing at an exponential rate and it is up to the DBA to maintain the CIA of said data. But how does the DBA respond to a breach? What factors should the DBA be considering? These are the things I don't see much of in the public domain, whether it be (ISC)2, SANS, etc.

Very curious about this and interested in hearing others insight/experience on this topic.

Shawn

Re: Security and the DBA

JoePete — Fri, 16 Feb 2018 00:48:04 GMT

An interesting question. I suppose one approach is that a database is just a different data store. It is really at the tail end of a long line of decisions. This might explain why there is so little focus on database security and forensics - much like there probably isn't a lot written about file security. I remember back in the late 90s when all the big data names were trying "webify" their products that had traditionally followed the server client model. What was kind of shock was how these mission critical databases generally had no table security. The security had always been pushed into the client. Now that these things were moving onto the web, they kind of followed the same model - wide open database with a lot of the security being handled by some application server. While I have seen some best practice recommendations, they tend to be pretty sparse and almost always come at the problem not from the database side from middleware/application side - in short assuming that will be the vector of attack.

Re: Security and the DBA

Baechle — Wed, 21 Feb 2018 22:34:56 GMT

Shawn,

I’m going to build onto what Joe has started.

Nearly every application you can think of is based upon a database of some kind. Some applications collocate their databases within the file structure alongside the application, while others offload them to dedicated Database products from vendors like Oracle. Securing the client, the application, the file system, and any networking and server systems between them follows a similar conceptual path. Because the manufacturers of applications configure their products differently, there is no neat one-size-fits-all set guidelines you could say is separate and exclusively for “databases”. You have the (a) application security; (b) operating system security; (c) network/communications security; and (d) storage access control security basically all playing together. How you configure each one would be the domain of that particular manufacturer’s published best practices combined with mitigative controls to address vulnerabilities as they arise.

For example, the manufacturer of one product may use a centralized Database-dedicated user account that fetches files on behalf of the application. In this case, a security design should account for and limit access to the file storage to the dedicated application account. The application and the server on which it resides must be configured in accordance with best practices to perform the authentication work for real users on behalf of the Database user account.

On the other hand, the manufacturer may refer database access back through the operating system/file system authentication mechanism. In this scenario the individual user relies upon their own rights to access the records rather than a centralized Database-dedicated user account. This means there is a completely different set of file access permission logic that needs to be developed.

That’s only one example of what you would need to account for. With each change in front end application or the back-end database, or even the operating system or storage solution they are built on comes a best-practice that must be assessed and addressed within the architecture. One thing you may want to consider is the CISSP Concentration for Architecture, the ISSAP. This concentration of knowledge leans in the direction you are pointed – but still requires you to keep abreast of and constantly assess the best practices for your particular configuration.

Sincerely,

Eric B.

Re: Security and the DBA

Stevandrag — Fri, 23 Feb 2018 15:51:21 GMT

I am currently doing research around database security and especially on architecture models coming from the industry i.e. what all different companies do to secure databases not only on premises but also in the cloud.

I would say there are some compliance demands that apply on databases especially inside PCI, HIPAA, GDPR.

Although I am not focused on forensic, I am happy to have more discussion with you.

Re: Security and the DBA

shaggyDBA — Tue, 27 Feb 2018 20:39:42 GMT

I agree that there is no "tried and true" database security solution, simply because there are so many players in that market. Each vendor has reams of documentation and a plethora of consultative advice on how to adequately secure "their" database.

But what prompted my post is more about forensics/incident response than securing the database. My question may be a little naive, as I haven't done much forensic work (outside of academics). As security professionals we seem to have a great deal of resources on how to perform forensics up to the database. This obviously varies based on the vendor software under investigation, but there is a "boilerplate" if you will of what should be done.

I haven't seen anything remotely close to this for the database side of the house. Each of the major database vendors supports audit logging, but if those logs are somehow manipulated how can we identify what has been compromised? We take immense care when processing a disk, laptop, or server for forensic analysis, but how do we apply the same level of rigor to a database?

I really started pondering this while I was reading the only book I've ever seen on this subject - "Oracle Incident Response and Forensics; Preparing for and Responding to Data Breaches" by Pete Finnigan.

For the geeks among us, it is a great read and got me thinking about why we don't have any additional focus on this area. Databases are more than just bit-buckets, otherwise we'd still be storing data in flat files. There is an incredible amount of processing power within a database and if someone is able to breach the database shouldn't we have some process identified to guide us through database forensics?

With the volumes of data generated daily, more powerful functionality is being introduced into database to enable data to be pinned in memory, how would one analyze this in the event of a breach? I find this fascinating and troubling at the same time!

Is there anyone in the group that has dealt with forensic analysis of databases as part of an incident response team? I am very curious to know what processes you followed to maximize the data gathered and ensure its integrity/authenticity/chain of custody.

Re: Security and the DBA

Baechle — Wed, 28 Feb 2018 17:37:01 GMT

Shawn,

I appreciate the (re)focus of what it is we’re talking about. In my opinion, you are naming two exclusive but technologically and procedurally overlapping disciplines (incident response and forensics), but discussing it in the framework of yet a third discipline (application security architecture). In my opinion each exists for its own purpose, approaches evaluating the problem and formulating a process differently, and is conducted by groups of people with different base skill sets.

First, it appears as though we were talking about information technology/systems security planning and architecture. You discuss the configuration and review of audit logs from database management applications. This is more the typical preventative and detective control discussions that are had in traditional application security design circles. The ability to recover relevant information during incident response, and for later forensic examination should be inputs into this discussion – but this is not forensics itself.

Now on to forensics. Let me paraphrase my definition of forensics: The process of identifying, evaluating, and presenting facts to or on behalf of a court of law. So, to clarify, the purpose of forensics as it relates to a database is not to ensure that the log files are unaltered or even exist at all. The purpose of forensic investigations is to first identify the possible areas where information about an activity may reside (including the apparent lack of information), and then collect, examine, and make conclusions about that information in a verifiable and repeatable way.

There is an unfortunate fact in the perspective of forensics, for the purposes of your discussion, that the database application, the database management application, the database store, the log files, and other relevant data is just that - data. Forgive me if my understanding of what you are talking about is off, but the hard and fast procedures that you're talking about in relation to recovering "forensic images" of hard disks and file systems applies to databases as well, because these are the contents of those higher level systems. Getting into looking at the content, regardless of if that is the structure of FAT, NTFS, HFS, Windows configuration files, log files, applications, data, etc. is the Examination phase of forensics. It's the collection phase, ensuring that the material is collected in a way that that minimizes or documents any potential alteration that has the most procedurally stringent controls. Examination is left up to the examiner to plan and develop.

The method to formulate a plan is based on –current best practice—about how to collect that information from the contents of data storage. Document the process including any deviations from the plan, errors encountered, failures and successes, to establish as fact that the information collected is a true and accurate representation of the information from the source – not that the content of that information is true or unaltered by an adverse party. And finally, to collect information from all available sources to recreate an accurate narrative of the activity at the time. That means looking beyond the database itself to determine what happened, such as network logs, operating system logs on servers and workstations, application logs, the state of the files themselves, etc.

I can tell you from experience, that imaging a system or data storage area is extraordinarily easy – but this is _not_ forensics. It’s a part of forensics. There are several automated tools that will practically do this for us by clicking “next” a bunch of times. The rest of forensics is fetching or developing a best practice for a newly encountered application, or a revision of an application that’s 20 years old, or so new a best practice hasn’t been developed yet. What is hard, is explaining to lay persons (a jury for example) the condition that exists in the system without speculating how the system arrived in that condition. Finally, I spend more time refuting a speculation from the opposite party by running tests to recreate the speculative condition: Running a copy of the database in a virtual machine while attempting different scenarios like adding, removing, modifying records, compacting/garbage collection processes, creating enough events to partially rollover/overwrite logs, and explaining why the proposed process does not create the condition that the adverse party speculates it might. I even once had an entire case built around a bug in a feature of a database application that I was able to diffuse by going back through the change log history to determine that the actual version of the database being used hadn’t implemented the feature yet. Forensics is way more than setting up the database “the right way”.

Sincerely,

Eric B.

Re: Security and the DBA

JoePete — Wed, 28 Feb 2018 19:05:58 GMT

@shaggyDBA wrote:
There is an incredible amount of processing power within a database and if someone is able to breach the database shouldn't we have some process identified to guide us through database forensics?

I think this also becomes an issue with the growing array (no pun) of data possibilities - especially as we venture into cloud-based systems (e.g. data dispersion). Imagine a database, in a virtual striped RAID array, residing on some cloud-based object storage infrastructure. That's like the turducken of data - one thing stuffed inside another - and if your guests get food poisoning, how do you figure if or which item had salmonella? You have a good question/observation going on here. From a forensic standpoint, is there a difference between examining a standard size check and 100-page contract for forgery? And to make that contract an accurate analog for today's database - it has also been sent through a paper shredder. While there may be a daunting difference between the two, both are still documents of the same type (paper). Maybe this is why database forensics seems an absent topic; a database is just seen as another file or set of files, but I think your question suggests that even though that may be true, the complexity of those files perhaps demands a more comprehensive solution.

Re: Security and the DBA

Baechle — Wed, 28 Feb 2018 19:29:58 GMT

Joe,

The forensic science process has different phases. These include, collection, analysis/examination, and presentation.

First there is the evidence collection phase, or seizure. This is where there is the most risk in destroying or catastrophically altering the evidence. In digital forensics this can be one step or two (depending on if there is first a physical seizure of equipment followed by a logical seizure of data from the equipment). This often gets confused as being the sole purpose of forensics since it is often the most visible step (assuming the evidence does not get presented in a trial) to non-forensic practitioners. The basic premise is that the inviolability of the evidence is preserved as well as possible while still making samples available for the analysis/examination phase. This is most often accomplished by taking an image of the storage media (either physically or logically) and then making at least two additional copies of that image (a reference image, and a working copy or copies). This is done in a structured way that so that it appears identical on similar platforms (e.g. all laptops are done the same way; all external hard drives are done the same way; all SAN/NAS/Cloud data is done the same way per that solution.)

Second, there is the analysis/examination phase. In this phase, scientific tests are conducted to answer a legal question. This could be comparing a hair sample, tool marks, DNA, etc. Sometimes these are destructive tests (such as cutting a piece of cloth and subjecting it to a chemical solution). Destructive tests typically have pre-determined protocols because once you conduct the test, that’s it. Everyone must be on the same page about the quality and veracity of the results of the test because it won’t be repeatable. In digital forensics we use non-destructive tests because the analysis/examination is conducted on a working copy made from the reference copy of the image. It’s authenticated as being a true copy through cryptographic hash comparisons. The tests here are what you’re referring to when talking about “database forensics”. The tests are constructed as needed to answer legal questions. This is why when you get to this level, there are no strict guidelines on what actions to perform – as opposed to the evidence collection phase where the evidence itself could be destroyed or in destructive tests.

The last phase is the presentation phase. In this phase, you are presenting the collection methodology and the tests that were conducted using lay language. Here you not only describe the rationale of the tests that were conducted but often defend your tests against opposition attack, and refute the speculations of the opposition about alternative tests and results.

Sincerely,

Eric B.

Re: Security and the DBA

JoePete — Wed, 28 Feb 2018 20:36:44 GMT

@Baechle

Eric, your response is part of what I was getting at. The database component of this is a sliver (as you note - the series of tests in the examination phase) of a larger picture. I do think as we continue to move toward dispersed data types - especially what we find in cloud environments - it's getting harder technically and legally to obtain that bit-by-bit reference copy.

Re: Security and the DBA

Baechle — Wed, 28 Feb 2018 21:06:20 GMT

@JoePete;

I believe I understand where you're going with your points.

What I am attempting to add is that this isn't a "Database Forensics" problem for a forensic investigation. Instead, it's a data storage acquisition problem for the investigation. The reason that there is a lack of specific material addressing this problem in the context of "Database Forensics," is because the problem doesn't exist.

If we are talking about a database in a cloud environment, we're typically talking about either (1) a virtual server in a dedicated hardware frame; (2) a virtual server in a shared hardware frame; or (3) a database store running on a shared database, in a shared virtual server, on shared hardware.

So, you either (1) have access to the virtual machine and related storage, (2) have access to the virtual machine and virtual storage, or (3) have a legal issue to work out about obtaining access to the data store, logs, operating system, etc. when there is the possibility of shared liability.

In all of the scenarios, you haven't even gotten to the point of forensically examining the data (the database). You wouldn't reasonably be able to do that until you've obtained the forensic working copy. Or... if you're attempting to do this on a live system, have serious defensibility issues once you're on the stand trying to explain why you're not working with a forensic copy.

Sincerely,

Eric B.

@JoePete wrote:
@Baechle

Eric, your response is part of what I was getting at. The database component of this is a sliver (as you note - the series of tests in the examination phase) of a larger picture. I do think as we continue to move toward dispersed data types - especially what we find in cloud environments - it's getting harder technically and legally to obtain that bit-by-bit reference copy.

Re: Security and the DBA

shaggyDBA — Wed, 28 Feb 2018 21:23:21 GMT

The database may be a sliver, but isn't the data held therein the theoretical prize?

So here's my concern (and I openly accept that this might be naive) - let's say a breach has occurred and an investigation is underway. How does one state with any level of confidence what data was compromised without investigating the database internals?

If I'm depending on file access or log analysis how can I be assured that I have an accurate picture of what was compromised? What if sensitive data is logically separated (via different schemas), but still exist in a datafile with non-sensitive data? How would we determine which schema had been compromised as part of our analysis? Or do we move under the assumption that if a datafile was accessed any data in said file could've been accessed?

I'm not trying to stir a pot or make databases the end-all-be-all, but I do have some concerns/questions about the how for narrowing the scope of a data breach.

Thanks to all for the feedback on this!

Re: Security and the DBA

Baechle — Wed, 28 Feb 2018 21:36:44 GMT

@shaggyDBA

Your scenario and questions are not naive at all. And I say what I have to say from the standpoint that based upon my training and experience, I am confident in the methods that I would employ right now. That could change based upon a best practice that comes out as soon as I post this reply.

I would say that your first step is to stop. Take a deep breath. Verify that you’re conducting a forensic investigation and not incident response, and if the answer is a forensic investigation then answer this question: What predicated your belief that there was a data breach?

Sincerely,

Eric B.

@shaggyDBA wrote:
The database may be a sliver, but isn't the data held therein the theoretical prize?

So here's my concern (and I openly accept that this might be naive) - let's say a breach has occurred and an investigation is underway. How does one state with any level of confidence what data was compromised without investigating the database internals?

If I'm depending on file access or log analysis how can I be assured that I have an accurate picture of what was compromised? What if sensitive data is logically separated (via different schemas), but still exist in a datafile with non-sensitive data? How would we determine which schema had been compromised as part of our analysis? Or do we move under the assumption that if a datafile was accessed any data in said file could've been accessed?

I'm not trying to stir a pot or make databases the end-all-be-all, but I do have some concerns/questions about the how for narrowing the scope of a data breach.

Thanks to all for the feedback on this!

Re: Security and the DBA

JoePete — Thu, 01 Mar 2018 15:38:52 GMT

@shaggyDBA an interesting wrinkle on your scenario is rather than thinking of a database sitting on some physical volume. Think of it as a database on a virtual machine, that itself is a database record, in some larger virtualization application, that sits, dispersed, on some cloud-based object storage. Moreover, the always connected, always-on world of the cloud diminishes the requisite of permanent storage - for example a temporary database or even complete virtual environment held nowhere else than in the RAM of thousands of interconnected devices. I am not sure "database forensics" is a full or accurate topic, but what's a database other than breaking complex data into discreet chunks and then relying on a schema to keep track of and assemble that data on demand? That model is repeating itself as we now use new data types and technologies to increase our computing capacity.

Re: Security and the DBA

yevgeng — Sat, 03 Mar 2018 09:33:41 GMT

You need to look at infrastructure and organization as a whole and see how databases fit into organization-wide security effort. Securing your databases is essentially applying certain techniques at the expense of some other aspect. Let's consider an account to access database over non-encrypted internet connection. If we revoke this kind of access we reduce possibility of all kinds of man-in-the middle attacks as well as data leak to third party. By doing this we give up convenience because now you need to get to db server somehow, execute commands locally and then get the output into a suitable place for further processing. If we encrypt client to server connection then we reduce the risk that someone might snoop at our session traffic. What do we lose--most likely bandwidth and some endpoint resources that will be spent on encryption\decryption. The perceived benefits in turn might be negated if connection is established from a non-patched application server which could be easily exploited. Or you might be on a dedicated vlan that doesn't take outside public connections directly (the traffic goes through firewalls, specialized IDS,IPS, packet analyzing appliances and is subject to heavy white/black-listing).

That's why enterprise-wide security effort and posture of entire organization needs to be considered against potential threats.

In regards to breach response, most likely you will be involved as subject matter expert during breach investigation or be a member special task force team that will be formed by security leadership specifically for breach investigation (3rd party resources might also be included).

Re: Security and the DBA

Baechle — Mon, 05 Mar 2018 07:26:14 GMT

As far as a DBA is concerned, I 100% agree with @yevgeng. Conducting a reactive investigation is about identifying, generating, and following specific leads. How you answer, "What predicated your belief that your system was compromised?" often changes what you do next as an investigator - as opposed to trying to diagnose the database system itself.

@yevgeng wrote:

In regards to breach response, most likely you will be involved as subject matter expert during breach investigation or be a member special task force team that will be formed by security leadership specifically for breach investigation (3rd party resources might also be included).