CS 224R Deep Reinforcement Learning

Course Calendar

A course calendar with details of lectures, TA sessions, office hours, and miscellaneous course events is available in a variety of formats:

• Google Calendar
• iCalendar file
• raw view

Grading and Course Policies

Homeworks (40%): There are three graded homework assignments. Homework 1 is worth 10% of the grade, while Homework 2 and Homework 3 are each worth 15% of the grade. Assignments will require training neural networks in PyTorch. All assignments are due on Gradescope at 9 pm Pacific Time on the respective due date.

Exam (25%): There's an in-class midterm exam on Friday, May 15 at 9:30 am, which covers course content up until May 8th. We will host an exam review session a few days prior to the exam with details TBA.

• On-campus students. We won't be able to accommodate conflicts since it is during class time, and a missed exam defaults to a zero grade. If you have an emergency (including illness) that means you have to miss an exam, please contact us via staff mailing list. We can't promise to accommodate, but we will see what we can do. Please keep in mind that if you email us on exam day or the night before, the course staff will be busy with exam logistics and we may take a day or two to get back to you.
• CGOE students. Due to the size of this class, we cannot accommodate in-person exams (even if you live near campus) to help with room capacity and staffing for the on-campus students. CGOE students will need to nominate exam monitors for both exams and coordinate the exam process with the CGOE exams team. Please refer to this link for more information on the process. For any additional questions, please reach out to the CGOE exams team.

Project (35%): The course project allows you to apply some of what you have learned to an application or research question of your choosing. The project may be done in groups of 1-3 students. You may choose between a custom final project (where students define a project pertaining to the course topics) and a default project (where students start by implementing a basic LLM RL set-up and try to improve it). Examples of both projects are available here, though the default project has been adjusted since Spring 2025.

• Default Project Guidelines (starter code)
• Custom Project Guidelines

Extra Credit (up to 2%): Extra credit will be awarded for outstanding projects and for outstanding Ed participation, at the discretion of the teaching team. The number of outstanding projects, as well as the standard for awarding extra credit for Ed solutions will be determined by the teaching team.

Late Days: You have 5 total late days across homeworks, the project proposal, and the project milestone. Late days are not applicable to the project poster (due to its live nature) and the project report (due to the university grading deadline). You may use a maximum of 2 late days for any single assignment, and assignments received after this will not be graded. Late days used for group project deliverables apply to all members of the group. Once you have used all 5 late days, the penalty is 2% off the final course grade for each additional late day.

• Late days are meant to give you some flexibility when unexpected situations come up (like illness or travel). If something like that happens, you should first use your late days instead of requesting an extension. We do not provide extra accommodations for students who join the course late.

Honor Code and Using AI Tools: Collaboration with other students and AI tools is allowed as part of the problem-solving process. However, unless noted otherwise, we expect students to write down solutions and code independently. Assistance from AI tools is treated analogously to assistance from another person. It is an honor code violation to copy, refer to, or look at written or code solutions from other students, from AI tools, and from previous course offerings, including the use of code autocomplete systems. Furthermore, it is an honor code violation to post your assignment solutions online, such as on a public git repo. For more information, see The Stanford Honor Code, The Stanford Honor Code Pertaining to CS Courses, and the Generative AI Policy Guidance.

Academic Accommodations

Students who may need academic accommodations based on the impact of a disability should initiate the request with the Office of Accessible Education (OAE) and notify us as soon as possible (no later than the add-drop deadline). It is the student's responsibility to reach out to the course staff regarding their accommodations on exams and assignments in advance. Please email letters to cs224r-staff-spr2526@cs.stanford.edu.

General Guidelines:

• Contact the OAE to obtain an official accommodation letter if you have a health condition that may impact your coursework.
• Only accommodations explicitly stated in your current OAE letter will be accommodated. If you need additional or revised accommodations, you must request an updated letter from the OAE.
• The teaching team will not be able to approve any retroactive accommodation requests.
• If you have any questions or concerns regarding your accommodations, please contact your disability advisor.

Deadlines and Assignments:

• Students with OAE letters must request a separate extension for each assignment (you cannot auto-apply your OAE extensions to all assignments).
  • Please note that we will not be able to apply OAE extensions to the final project poster (due to its live nature) and final project report (due to the university grading deadline).
• Students with the standard OAE extension accommodation must request an extension before the original due date, as it says in your OAE letter. Retroactive extension requests will not be approved.
• The teaching team will only approve what is outlined in the OAE letter. If you have any questions or concerns, please reach out to your OAE advisor.

Exams: If you plan to use your OAE-approved exam accommodations for a specific exam, students must provide their letter and inform the instructor by:

• 10 calendar days prior to midterm date.

You only need to submit your letter once per quarter. For urgent OAE-related accommodation needs that arise after the deadline, please consult your OAE adviser. If you are not yet registered with OAE, contact the office directly at oae-contactus@stanford.edu.

Additional Information

Lecture Attendance: While we do not require lecture attendance, students are encouraged to join the live lecture. If you want to watch the lecture remotely, please see the Panopto tab of Canvas. For those who cannot join the live lectures, lecture recordings will also be available on Canvas shortly following the lecture. Video cameras located in the back of the room will capture the instructor presentations in this course. For your convenience, you can access these recordings by logging into the course Canvas site. These recordings might be reused in other Stanford courses, viewed by other Stanford students, faculty, or staff, or used for other education and research purposes. Note that while the cameras are positioned with the intention of recording only the instructor, occasionally a part of your image or voice might be incidentally captured. If you have questions, please contact a member of the teaching team.

AIWG: This course is participating in the proctoring pilot overseen by the Academic Integrity Working Group (AIWG). The purpose of this pilot is to determine the efficacy of proctoring and develop effective practices for proctoring in-person exams at Stanford. To find more details on the pilot or the working group, please visit the AIWG's webpage.

Note on Financial Aid

All students should retain receipts for books and other course-related expenses, as these may be qualified educational expenses for tax purposes. If you are an undergraduate receiving financial aid, you may be eligible for additional financial aid for required books and course materials if these expenses exceed the aid amount in your award letter. For more information, review your award letter or visit the Student Budget website.