As a graduate seminar, the main focus of the course will be on research projects. There will be no exams, and all marks will be focused on the research project. The research project can be completed in pairs, or individually. A successful outcome will be a paper, that could be submitted to a workshop or conference, with that paper actually formatted appropriately for the venue. My goal in this course is to help you specify a feasibly-sized project, that can be completed within the course. This will help ensure that you are exposed to completing a paper. The learning goals for this course are to introduce you to more advanced optimization principles for reinforcement learning. It is expected that you have taken a reinforcement learning course. The topics covered will include Bellman operators; some discussion of convergence; optimization objectives in reinforcement learning; difficulties in optimizing objectives in reinforcement learning; step-size selection; eligibility traces; and off-policy learning. I will lecture on some background material, there will be some guest lectures and otherwise each student will be responsible for giving a 20 minute overview of a paper in the readings.