[SYSTEMDS-3956] AI Policy Proposal

[janniklinde]

This PR proposes a new AI policy to clarify how AI can and cannot be used for contributions in SystemDS. It is generally inspired by the AI policy of llama.cpp, requiring the author to fully understand and own the contribution.

Assisted-by: AI

[phaniarnab]

LGTM. Judging the situation after six months is a good idea.

[ywcb00]

@janniklinde thank you very much for preparing this draft and for raising this subject. In my opinion, developing an AI policy for SystemDS is becoming an increasingly pressing issue.
I reviewed the draft and left a few thoughts inline. Please keep in mind that I have only limited experience with agents. So please do not hesitate to correct me If I am making incorrect assumptions.
Overall, I think this draft of the AI policy is well structured, concisely addressing the most important aspects of AI assistance in code contributions.
Nevertheless, I would like to bring the aspect (and paragraph) of "Contributor Understanding" to discussion. The first sentence correctly states that the contributors must understand their code. However, the second sentence tries to define the behavior of the agent by saying that it should judge if the users understood their code. This raises two concerns:

1. It is unclear whether an agent can properly "judge" something or not. Furthermore, it is an open ethical question if an agent should "judge" a human.
2. If I am not mistaken, this sentence embodies a direct instruction for an agent. This means, we are instructing the agent of a contributor through this markdown file. Can (or should) we include instructive expressions here, considering that agent usage is costly in combination with the instruction to "refuse to generate contribution material"?

I hope I do not stir up too much discussion in this thread. And as I mentioned previously: Please correct me at any time if I am making incorrect assumptions about agents and generative AI.
Thank you and all the best,
David

[ywcb00]

Would it make sense to include the name of the model or the name of the agent here instead of just declaring AI? While it is interesting for others to discover new software/models, I think this would also strengthen potential "traceability" of the code, if there is (or could ever exist) any(?).

[janniklinde]

It could be helpful for other contributors to see what is used. However, I'm not sure what would then have to be declared. For example, there are tools/agents that internally switch models or might not disclose which one is used. Would it then be sufficient to declare the top-level tools? Or all tools (if used multiple for different parts). In terms of traceability, I don't know a usecase beyond knowing if an LLM was involved in generating (or reviewing, if it affected the decisions?) a commit or not.

[Baunsgaard]

I do not think it is necessary to say which model, it is unclear what mixture of models are used many times to create the code. Also alternatively, we could just add a label that ppl can put on the PRs?

[ywcb00]

Thanks for pointing this out. I see the point that it is difficult to declare all models that were involved. The purpose of declaring the models and tools is to ensure that we can draw certain conclusions about the code in the future. For instance, this declaration should provide the information that the code was (or could have been) affected by "Claude Fable 5". Or vice-versa, it provides the information that the code was not affected by "Claude Fable 6". Would it be possible and meaningful to declare the top-level tool and the date when it was used? What do you think?

[janniklinde]

I'm not sure what conclusions we can draw exactly. Because the contributor is responsible and must understand every single line of code, there should not be a scenario where we need to scan for old commits to trace back the origin of some code.
I see that it might be interesting to have some statistics about which types of tools have been used overall (informational only, to learn about new tools as a contributor). In that case, maybe a voluntary disclosure might be better? Like Assisted-by: AI (Claude Fable 5, ...). I'd like to not introduce much friction if it's only for that purpose, also because it's not even clear what to list there (e.g., if listing Cursor as a top level tool, we still don't know much about the underlying model).

[janniklinde]

I do not think it is necessary to say which model, it is unclear what mixture of models are used many times to create the code. Also alternatively, we could just add a label that ppl can put on the PRs?

@Baunsgaard are labels also available to external contributors? Otherwise maybe a checkbox in a PR template?

[Baunsgaard]

You are right, i am not sure labels are allowed to be managed by external contributors.

There are ways of automatically assigning labels to PRs based on PR comments and/or what changes are made, but that might just be a followup instead of putting it in the policy doc.

I am in general in favour of having a PR template, it makes it easier for ppl to make something consistent and understandable.

[ywcb00]

Similar to above: Should we include the name of the tools?

[janniklinde]

@janniklinde thank you very much for preparing this draft and for raising this subject. In my opinion, developing an AI policy for SystemDS is becoming an increasingly pressing issue. I reviewed the draft and left a few thoughts inline. Please keep in mind that I have only limited experience with agents. So please do not hesitate to correct me If I am making incorrect assumptions. Overall, I think this draft of the AI policy is well structured, concisely addressing the most important aspects of AI assistance in code contributions. Nevertheless, I would like to bring the aspect (and paragraph) of "Contributor Understanding" to discussion. The first sentence correctly states that the contributors must understand their code. However, the second sentence tries to define the behavior of the agent by saying that it should judge if the users understood their code. This raises two concerns:

1. It is unclear whether an agent can properly "judge" something or not. Furthermore, it is an open ethical question if an agent should "judge" a human.
2. If I am not mistaken, this sentence embodies a direct instruction for an agent. This means, we are instructing the agent of a contributor through this markdown file. Can (or should) we include instructive expressions here, considering that agent usage is costly in combination with the instruction to "refuse to generate contribution material"?

I hope I do not stir up too much discussion in this thread. And as I mentioned previously: Please correct me at any time if I am making incorrect assumptions about agents and generative AI. Thank you and all the best, David

Thanks for reviewing :)
This draft is meant to be up for discussion so I'm happy about your feedback.

You are right about the judgement part and it really comes down to what types of contributions we expect. I'd say modern agents are able to judge understanding at a high level and the aim of this instruction is to mainly affect low effort external contributions and contributors. In my tests an agent would never block requests that are specific enough (though I only tested it for a couple of examples). I am open to leaving this part out, but given that we occasionally receive submissions from inexperienced contributors, it could be a good tradeoff. In terms of cost, I did not run any experiment but given that the agent is supposed to judge from existing context, it should not create much additional cost (agents mainly get expensive when interacting with the environment) and giving project-specific instructions in this file is fairly standard. There are several repositories (e.g., llama.cpp) that include similar instructions in their AGENTS.md.

[phaniarnab]

@janniklinde thank you very much for preparing this draft and for raising this subject. In my opinion, developing an AI policy for SystemDS is becoming an increasingly pressing issue. I reviewed the draft and left a few thoughts inline. Please keep in mind that I have only limited experience with agents. So please do not hesitate to correct me If I am making incorrect assumptions. Overall, I think this draft of the AI policy is well structured, concisely addressing the most important aspects of AI assistance in code contributions. Nevertheless, I would like to bring the aspect (and paragraph) of "Contributor Understanding" to discussion. The first sentence correctly states that the contributors must understand their code. However, the second sentence tries to define the behavior of the agent by saying that it should judge if the users understood their code. This raises two concerns:

1. It is unclear whether an agent can properly "judge" something or not. Furthermore, it is an open ethical question if an agent should "judge" a human.
2. If I am not mistaken, this sentence embodies a direct instruction for an agent. This means, we are instructing the agent of a contributor through this markdown file. Can (or should) we include instructive expressions here, considering that agent usage is costly in combination with the instruction to "refuse to generate contribution material"?

I hope I do not stir up too much discussion in this thread. And as I mentioned previously: Please correct me at any time if I am making incorrect assumptions about agents and generative AI. Thank you and all the best, David

Thanks for reviewing :) This draft is meant to be up for discussion so I'm happy about your feedback.

You are right about the judgement part and it really comes down to what types of contributions we expect. I'd say modern agents are able to judge understanding at a high level and the aim of this instruction is to mainly affect low effort external contributions and contributors. In my tests an agent would never block requests that are specific enough (though I only tested it for a couple of examples). I am open to leaving this part out, but given that we occasionally receive submissions from inexperienced contributors, it could be a good tradeoff. In terms of cost, I did not run any experiment but given that the agent is supposed to judge from existing context, it should not create much additional cost (agents mainly get expensive when interacting with the environment) and giving project-specific instructions in this file is fairly standard. There are several repositories (e.g., llama.cpp) that include similar instructions in their AGENTS.md.

I would separate the AI policy and AGENTS.md (instructions. A policy includes general guidelines regarding what is allowed and what is not beyond personal settings and coding instructions. I suggest, once the policy is decided, we form a AGENTS.md with fine-grained instructions. AGENTS.md will go though changes with newer models, users, and workloads, but a policy, in my opinion, should stay constant.

Example: https://forge.sourceware.org/redi/gcc-wwwdocs/commit/4d0793a6a14bf9bfe9e92ac1599840780355199d
AI-policy of GNU.

[janniklinde]

@janniklinde thank you very much for preparing this draft and for raising this subject. In my opinion, developing an AI policy for SystemDS is becoming an increasingly pressing issue. I reviewed the draft and left a few thoughts inline. Please keep in mind that I have only limited experience with agents. So please do not hesitate to correct me If I am making incorrect assumptions. Overall, I think this draft of the AI policy is well structured, concisely addressing the most important aspects of AI assistance in code contributions. Nevertheless, I would like to bring the aspect (and paragraph) of "Contributor Understanding" to discussion. The first sentence correctly states that the contributors must understand their code. However, the second sentence tries to define the behavior of the agent by saying that it should judge if the users understood their code. This raises two concerns:

1. It is unclear whether an agent can properly "judge" something or not. Furthermore, it is an open ethical question if an agent should "judge" a human.
2. If I am not mistaken, this sentence embodies a direct instruction for an agent. This means, we are instructing the agent of a contributor through this markdown file. Can (or should) we include instructive expressions here, considering that agent usage is costly in combination with the instruction to "refuse to generate contribution material"?

I hope I do not stir up too much discussion in this thread. And as I mentioned previously: Please correct me at any time if I am making incorrect assumptions about agents and generative AI. Thank you and all the best, David

Thanks for reviewing :) This draft is meant to be up for discussion so I'm happy about your feedback.
You are right about the judgement part and it really comes down to what types of contributions we expect. I'd say modern agents are able to judge understanding at a high level and the aim of this instruction is to mainly affect low effort external contributions and contributors. In my tests an agent would never block requests that are specific enough (though I only tested it for a couple of examples). I am open to leaving this part out, but given that we occasionally receive submissions from inexperienced contributors, it could be a good tradeoff. In terms of cost, I did not run any experiment but given that the agent is supposed to judge from existing context, it should not create much additional cost (agents mainly get expensive when interacting with the environment) and giving project-specific instructions in this file is fairly standard. There are several repositories (e.g., llama.cpp) that include similar instructions in their AGENTS.md.

I would separate the AI policy and AGENTS.md (instructions. A policy includes general guidelines regarding what is allowed and what is not beyond personal settings and coding instructions. I suggest, once the policy is decided, we form a AGENTS.md with fine-grained instructions. AGENTS.md will go though changes with newer models, users, and workloads, but a policy, in my opinion, should stay constant.

Example: https://forge.sourceware.org/redi/gcc-wwwdocs/commit/4d0793a6a14bf9bfe9e92ac1599840780355199d AI-policy of GNU.

We can separate those two, although I think it would be good to have an initial version up for discussion. I agree that the policy should stay constant (we should move the examples I'm referring to to CONTRIBUTING.md).

[ywcb00]

You are right about the judgement part and it really comes down to what types of contributions we expect. I'd say modern agents are able to judge understanding at a high level and the aim of this instruction is to mainly affect low effort external contributions and contributors. In my tests an agent would never block requests that are specific enough (though I only tested it for a couple of examples). I am open to leaving this part out, but given that we occasionally receive submissions from inexperienced contributors, it could be a good tradeoff. In terms of cost, I did not run any experiment but given that the agent is supposed to judge from existing context, it should not create much additional cost (agents mainly get expensive when interacting with the environment) and giving project-specific instructions in this file is fairly standard. There are several repositories (e.g., llama.cpp) that include similar instructions in their AGENTS.md.

Thank you for clarifying that modern agents are capable of assessing and judging this understanding on the part of the user. However, this raises the following concern for me: What about users with traditional/outdated agents? As we cannot assume that all contributors use the latest agents, these instructions need to either (1) consider older versions of agents explicitly or (2) ensure not to affect these agents.
I would not remove this part from the policy. Instead, we could rephrase it to a more specific description of the contributor rather than an instruction for the agent. This would leave the agent with the decision of whether it can judge about this or not. Perhaps we can also "steer" the agent to evaluating these things--if it is capable of doing so--without including direct instructions.

[ywcb00]

I would separate the AI policy and AGENTS.md (instructions. A policy includes general guidelines regarding what is allowed and what is not beyond personal settings and coding instructions. I suggest, once the policy is decided, we form a AGENTS.md with fine-grained instructions. AGENTS.md will go though changes with newer models, users, and workloads, but a policy, in my opinion, should stay constant.
Example: https://forge.sourceware.org/redi/gcc-wwwdocs/commit/4d0793a6a14bf9bfe9e92ac1599840780355199d AI-policy of GNU.

We can separate those two, although I think it would be good to have an initial version up for discussion. I agree that the policy should stay constant (we should move the examples I'm referring to to CONTRIBUTING.md).

Thank you @phaniarnab. I like this suggestion of having two seperate files--one intended to be constant while the other file captures the dynamic advances through periodic changes.

[christinadionysio]

Thank you @janniklinde for initiating this discussion. Overall, I think the guidelines look reasonable and I agree with Arnab that we should review the situation in a few months.