Audit logging and export to centralized logging (ProGet / BuildMaster)

dbojak

Hello,

We are currently evaluating ProGet and other Inedo products for enterprise use. As part of that evaluation, we need to understand what administrative and security events are captured, and how those events can be exported to our internal centralized logging platform.

Examples of gaps we have noticed currently in the system logging:

Changes to policies on a feed. Today we can see feed creation in proget event logs in the UI but thats about it.
Creation or deletion of API keys are not logged
Creation of User is logged but which permission was given to the user is not logged
If an anonymous user is provided access to something it only logs the privilege ID and doesn't provide more information that an anon user was provided administer, Manage Feed, Publish Package etc.

Please provide below if available:

Whether a documented, exhaustive list of audited event types exists per product
Whether the events are accessible via API or webhook, or whether direct database access is the only option, and if the latter, whether the schema is considered stable across product versions
Whether audit events are subject to the same automatic purge as Diagnostic Center messages, and whether the retention period is configurable

Thank you,
Dev

stevedennis

Hi @dbojak,

Great questions, and we're happy to help.

We're currently working on developing our best practices for application activity, and I'd like to share our current guidance, which is a work in progress. This is based on working closely with customers over many years, both proactively (developing policies) and reactively (actively helping investigate after an incident).

(Work in Progress)

Before looking to "ingest all the logs", identify a clear scenario that's contextualized to the product. For example, what does the Information Security team have in mind for monitoring, incident investigation, and compliance? What kind of incidents? What monitoring do they anticipate? And compliance with what?

ProGet attempts to provide appropriate "visibility into user activity and administrative actions", so if there are gaps I would start there. What is missing? How does duplicating data help? Etc.

For example, knowing which users download which Microsoft-owned packages on which day/time is not only impractical, but it serves no conceivable business benefit nor reduces any risk. But it could impact developer productivity and will increase costs across the board.

In addition, Microsoft Sentinel doesn't provide any contextualized visibility and, chances are, you will just end up back in ProGet searching for data anyway. So all the time and bandwidth spent exporting/duplicating data just ends up being waste.

That said, we don't have any built-in functionality to export this data, but it's relatively easy to export/ingest from the EventOccurrences table. These work differently in each product, but are similar in design.

[1] We do not have a documented, exhaustive list of audited event types exists per product, but it's relatively easy to attain through SELECT * FROM EventTypes type of query. That's the "source of truth" for events

[2] Direct database is the only option; we consider it quite stable, and it hasn't changed since like 1.0 of the products; there hasn't been any real demand to redesign or enhance it

[3] Audit events are never automatically purged, nor can they be deleted from the UI/API; they can only be manually purged from the database

As for the gaps you identified, those are things we can certainly consider addressing, provided it's within the confines of the existing design. It's relatively easy to add a new event type (if needed) or add additional audit data.

Overall, it's meant to be an "audit trail" instead of "change log" -- so it's not always the most user-friendly, uses more system-level concepts, and prefers immutable IDs over names.

Hope that helps!

STeve

dbojak

@stevedennis said:

dbojak
16 days ago

Hello,

We are currently evaluating ProGet and other Inedo products for enterprise use. As part of that evaluation, we need to understand what administrative and security events are captured, and how those events can be exported to our internal centralized logging platform.

Examples of gaps we have noticed currently in the system logging:

Changes to policies on a feed. Today we can see feed creation in proget event logs in the UI but thats about it.
Creation or deletion of API keys are not logged
Creation of User is logged but which permission was given to the user is not logged
If an anonymous user is provided access to something it only logs the privilege ID and doesn't provide more information that an anon user was provided administer, Manage Feed, Publish Package etc.
Please provide below if available:

Whether a documented, exhaustive list of audited event types exists per product
Whether the events are accessible via API or webhook, or whether direct database access is the only option, and if the latter, whether the schema is considered stable across product versions
Whether audit events are subject to the same automatic purge as Diagnostic Center messages, and whether the retention period is configurable
Thank you,
Dev

Offline
stevedennis
INEDO-ENGINEER
14 days ago

Hi @dbojak,

Great questions, and we're happy to help.

We're currently working on developing our best practices for application activity, and I'd like to share our current guidance, which is a work in progress. This is based on working closely with customers over many years, both proactively (developing policies) and reactively (actively helping investigate after an incident).

(Work in Progress)

Before looking to "ingest all the logs", identify a clear scenario that's contextualized to the product. For example, what does the Information Security team have in mind for monitoring, incident investigation, and compliance? What kind of incidents? What monitoring do they anticipate? And compliance with what?

ProGet attempts to provide appropriate "visibility into user activity and administrative actions", so if there are gaps I would start there. What is missing? How does duplicating data help? Etc.

For example, knowing which users download which Microsoft-owned packages on which day/time is not only impractical, but it serves no conceivable business benefit nor reduces any risk. But it could impact developer productivity and will increase costs across the board.

In addition, Microsoft Sentinel doesn't provide any contextualized visibility and, chances are, you will just end up back in ProGet searching for data anyway. So all the time and bandwidth spent exporting/duplicating data just ends up being waste.

That said, we don't have any built-in functionality to export this data, but it's relatively easy to export/ingest from the EventOccurrences table. These work differently in each product, but are similar in design.

[1] We do not have a documented, exhaustive list of audited event types exists per product, but it's relatively easy to attain through SELECT * FROM EventTypes type of query. That's the "source of truth" for events

[2] Direct database is the only option; we consider it quite stable, and it hasn't changed since like 1.0 of the products; there hasn't been any real demand to redesign or enhance it

[3] Audit events are never automatically purged, nor can they be deleted from the UI/API; they can only be manually purged from the database

As for the gaps you identified, those are things we can certainly consider addressing, provided it's within the confines of the existing design. It's relatively easy to add a new event type (if needed) or add additional audit data.

Overall, it's meant to be an "audit trail" instead of "change log" -- so it's not always the most user-friendly, uses more system-level concepts, and prefers immutable IDs over names.

Hope that helps!

STeve

Hi Steve,

Thanks for your reply.

Our requirements are primarily focused on cybersecurity monitoring and supplementing incident investigations

The audit gaps we would like addressed are:

Feed security-policy changes:
- Log who made the change, when and what was changed plus new values.
API-key lifecycle events:
- Capture creation, deletion, revocation, and permission changes, identifying the responsible actor and affected key.
Role and anonymous-access changes:
- Clearly identify the actor, affected identity, permissions granted or removed, and resource.

Package download logs also support supply chain incident investigations, allowing us to attribute a specific package and version download to a human or non human identity. We will be using large language model (LLM) agents for development, so attribution to the relevant user, agent, or service account is important for assessing potential exposure.

Integration with Security Information and Event Management (SIEM) tools would simplify log extraction and forwarding without relying on direct database access. It would also support correlation with identity, endpoint, and network activity and preserve evidence separately from ProGet.

We would like these audit enhancements and SIEM integration added to your product roadmap or a tracked enhancement plan.

Thank you,
Dev

stevedennis

Hi @dbojak ,

We do User-driven Development, so this is something we could work on after customer onboarding.

The audit changes you describe seem reasonable, although we would obviously need to work together to translate that to product-specific terms; e.g. there are no "feed security policies", just scoped privilege/restrictions that associate a principal with a task, and those are either are added/deleted. You'll understand these as you learn our product a bit more, however.

And obviously there are also some technical limitations far beyond ProGet - for example, knowing which user downloaded a package requires that user first authenticating to the feed. Otherwise these kind of changes are relatively easy to implement and something we can often add in maintenance releases.

However, it's unlikely we will add SIEM integration as a feature; it'd be a substantial undertaking that is like 95% outside of ProGet's core functionality, and would require continuously supporting third-party log aggregation platforms we have zero exposure to. We would need to see (1) a lot more demand, and (2) user-created prototypes that extract/convert from the database.

A handful of users have written "log exporters" to tools like Splunk, but they are so highly specific to their requirements that it would be impossible to generalize as an open-source tool, let alone a product feature.

Thanks,
Steve