\documentclass[12pt]{article}
\usepackage{fullpage,enumitem,amsmath,amssymb,graphicx,url,hyperref}
\usepackage{sectsty}

\hypersetup{
    colorlinks=true,
    linkcolor=blue,
}

\sectionfont{\fontsize{15}{20}\selectfont}

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\E}{\mathbb{E}}
\usepackage{tgpagella}
\usepackage{tcolorbox}
\usepackage{forest}



\begin{document}

\begin{center}
{\Large \textbf{CS224R Spring 2025 Homework 1 \\ Imitation Learning}}
\\ {\large Due 4/18/2025}

\begin{tabular}{rl}
SUNet ID: & $\hspace{6cm}$\\
Name: & \\
Collaborators: & 
\end{tabular}
\end{center}

\noindent By turning in this assignment, I agree by the Stanford honor code and declare
that all of this is my own work. 
%\noindent \textbf{Note:} As an update to the course policy, TA-approved Ed answers and posts will receive extra credit. The total possible extra credit you may receive is up to 2\% of your grade.

\section*{Overview}

\textbf{Goals:} In this assignment you will experiment with two common methods of imitation learning: behavior cloning and DAgger. You will then have the opportunity to train and tune your agents in a variety of environments in the MuJoCo physics simulator (\url{https://mujoco.org/}). In lieu of a human demonstrator, we will provide demonstrations from a pre-trained expert policy.
\begin{enumerate} 
    \item Implement and train an imitation learning agent that can learn from expert data.
    \item Analyze the shortcomings of behavior cloning and how DAgger can address some of these issues. 
    \item Experiment with hyperparameters and explore how they affect performance.
\end{enumerate}

\vspace{0.2cm}
\noindent\textbf{Submitting the PDF}: Make a PDF report containing: Table 1 for a table of results from Question 1.2, Figure 1 for Question 1.3, and Figure 2 with results from Question 2.2 as well as the requested brief explanations. \\ 

\noindent\textbf{Submitting the Code and Experiment Runs}: In order to turn in your code and experiment logs, create a folder that contains the following:
\begin{itemize}
\item A folder named data with experiments for both the behavioral cloning (sub-problem 1, not sub-problem 2) exercise and the DAgger exercise. Note that you can include multiple runs per exercise if you’d like, but you must include at least one run (of any task/environment) per exercise. These folders can be copied directly from the \texttt{cs224r/data} folder into this new folder. \textbf{Important: Disable video logging for the runs that you submit, otherwise the files size will be too large! You can do this by setting the flag --video log freq -1}
\item A README file that lists the commands (with clear hyperparameters) that we need in order to run the code and produce the numbers that are in your figures/tables. The following command is an example:\\
\\
\texttt{To generate the numbers for Question 1 Subproblem 1, run: \\
python cs224r/scripts/run\_hw1.py \textbackslash \\
    --expert\_policy\_file cs224r/policies/experts/Ant.pkl \textbackslash \\
    --env\_name Ant-v4 --exp\_name bc\_ant --n\_iter 1 \textbackslash \\
    --expert\_data cs224r/expert\_data/expert\_data\_Ant-v4.pkl }
\item The cs224r folder with all the .py files, with the same names and directory structure as the original homework repository. 
\end{itemize} 

\noindent\textbf{Gradescope}: Submit both the PDF and the code and experiment runs in the appropriate assignment on Gradescope. An autograder will be provided to evaluate the performance of your policies from the generated tensorboard files. \textbf{Note: the autograder score will only consist of 50\% of your homework grade. The consistency of the code and tensorboard files will be checked after submission.}\\

\noindent\textbf{Note:} once you have the files for submission e.g. in a folder named \texttt{submit}, run the command \textcolor{red}{\texttt{zip -r submit.zip submit/}} to generate the zip file. For an example of what the zip should look like, see the following page. In the past, we've found programs like WinRAR seem to be incompatible with Gradescope.\\

\noindent\textbf{Use of AI Tools (e.g. ChatGPT, Cursor):} For the sake of deeper understanding on implementing imitation learning methods, assistance from generative models to write code for this homework is prohibited. See the course website for more details.
\newpage


\section*{Sample File Submission}
As an example, the unzipped version of your submission should result in the following file structure. \textbf{Make sure that the submit.zip file is below 15MB and that the experiment log folders include the prefix \texttt{q1\_} and \texttt{q2\_}.} \\
\\
\begin{forest}
  for tree={
    font=\ttfamily,
    grow'=0,
    child anchor=west,
    parent anchor=south,
    anchor=west,
    calign=first,
    edge path={
      \noexpand\path [draw, \forestoption{edge}]
      (!u.south west) +(7.5pt,0) |- node[fill,inner sep=1.25pt] {} (.child anchor)\forestoption{edge label};
    },
    before typesetting nodes={
      if n=1
        {insert before={[,phantom]}}
        {}
    },
    fit=band,
    before computing xy={l=15pt},
  }
[submit.zip
  [data
    [q1\_bc\_Ant
        [events.out.tfevents\ldots]
    ]
    [q2\_dagger\_Ant
        [events.out.tfevents\ldots]
    ]
    [$\ldots$]
  ]
  [cs224r
    [agents
        [bc\_agent.py]
        [$\ldots$]
    ]
    [policies
        [$\ldots$]
    ]
    [$\ldots$]
  ]
  [README.md]
]
\end{forest}


\section*{Codebase}
\includegraphics[width=\textwidth]{Homework1/figures/hw1_codebase.png}
We have provided you with starter code for this homework. A simplistic diagram of its structure is shown above. In the \texttt{agents} directory, you will define your main behavior cloning agent. This agent will contain contain references to its policy (or actor) and replay buffer. This agent will be fed into the BC trainer, which will contain its training loop.\\
\\
We recommend that you read the files in the following order. Your task is to fill in the blanks labeled \texttt{TODO}. 

\begin{itemize}
\item \texttt{scripts/run\_hw1.py (read-only file)}
\item \texttt{infrastructure/bc\_trainer.py}
\item \texttt{agents/bc\_agent.py (read-only file)}
\item \texttt{policies/MLP\_policy.py}
\item \texttt{infrastructure/replay\_buffer.py}
\item \texttt{infrastructure/utils.py}
\item \texttt{infrastructure/pytorch\_util.py}
\end{itemize}

\newpage
\section*{Problem 1: Behavior Cloning}
\begin{enumerate}
\item Run behavioral cloning (BC) and report results on two tasks: the Ant environment, where a behavioral cloning agent should achieve at least $30\%$ of the performance of the expert, and one environment of your choosing where it does not. For your reference, the other environments with expert data include Hopper, Walker2d, and HalfCheetah. A policy that achieves greater than $30\%$ of the expert on the Ant task will receive full credit on the autograder. Here is how you can run the Ant task:
\begin{tcolorbox}[width=\linewidth, sharp corners=all, colback=white!95!black]
\begin{verbatim}
python cs224r/scripts/run_hw1.py \
    --expert_policy_file cs224r/policies/experts/Ant.pkl \
    --env_name Ant-v4 --exp_name bc_ant --n_iter 1 \
    --expert_data cs224r/expert_data/expert_data_Ant-v4.pkl \
    --video_log_freq -1
\end{verbatim}
\end{tcolorbox}

\noindent When providing results, report the mean and standard deviation of your policy’s return over multiple rollouts in a table, and state which task was used. When comparing one that is working versus one that is not working, be sure to set up a fair comparison in terms of network size, amount of data, and number of training iterations. \\
\\
\noindent \textbf{Note:} To report the mean and standard deviation, your eval batch size should be greater than \texttt{ep\_len}, so that you’re collecting multiple rollouts when evaluating the performance of your trained policy. For example, if \texttt{ep\_len} is $1000$ and \texttt{eval\_batch\_size} is $5000$, then you’ll be collecting approximately 5 trajectories (maybe more if any of them terminate early), and the logged \texttt{Eval\_AverageReturn} and \texttt{Eval\_StdReturn} represents the mean and standard deviation of your policy over these 5 rollouts. \\
\\
\textbf{Tip:} To generate videos of the policy, remove the flag \texttt{--video\_log\_freq -1}. However, this is slower, and so you will want to keep this flag on while debugging. \\
\\
\textbf{\color{red}Include your table here.}

\item Experiment with one set of hyperparameters that affects the performance of the behavioral cloning agent, such as the amount of training steps, the amount of expert data provided, or something that you come up with yourself. For one of the tasks used in the previous question, show a graph of how the BC agent’s performance varies with the value of this hyperparameter. State the hyperparameter and a brief one or two sentence rationale for why you chose it. The plot should contain clearly labeled axes.\\
\\
\textbf{\color{red}Include your chosen hyperparameter, plot, and rationale here.}

\end{enumerate}

\newpage

\section*{Problem 2: DAgger}
\begin{enumerate}
\item Once you’ve filled in all of the TODO commands, you should be able to run DAgger.
\begin{tcolorbox}[width=\linewidth, sharp corners=all, colback=white!95!black]
\begin{verbatim}
python cs224r/scripts/run_hw1.py \
  --expert_policy_file cs224r/policies/experts/Ant.pkl \
  --env_name Ant-v4 --exp_name dagger_ant --n_iter 10 \
  --do_dagger \
  --expert_data cs224r/expert_data/expert_data_Ant-v4.pkl \
  --video_log_freq -1
\end{verbatim}
\end{tcolorbox}
\item Run DAgger and report results on the two tasks you tested previously with behavioral cloning (i.e., Ant + another environment). Report your results in the form of a learning curve, plotting the number of DAgger iterations on the x-axis versus the policy’s mean return on the y-axis, with error bars to show the standard deviation. Include the performance of the expert policy and your behavioral cloning agent on the same plot. State which task you used, and any details regarding network architecture, amount of data, etc. (as in the previous section).\\
\\
\textbf{\color{red}Include the plots of the two tasks here.}
\end{enumerate}

%\bibliography{Homework1/references}
%\bibliographystyle{unsrt}

\end{document}
