Hi Ali,
By default, an artifact retention policy operates on artifacts alone and does not consider builds/releases/etc.
There is no current way to say "500MB of artifacts per release", however, you can combine 2 retention policies that should be close to what you want:
Build retention policy:
Only purge builds in deployed releases
Only purge rejected builds
Keep last (whatever you want)
Artifact retention policy:
Only purge artifacts in deployed releases
Only purge artifacts in rejected builds
Purge orphaned artifacts
The key is "purge orphaned artifacts" which will only delete artifacts after its corresponding build is deleted. This allows you to provide the necessary granularity on the builds themselves, and let the artifact policy come in after and clean up the artifacts that no longer have an existing build. This can be extended to releases in a way, since any releases that get deleted will also have all their builds deleted.
Hope this helps,
-John