Autonomous vehicles still underperform humans by a wide margin at one human intervention per 5k miles - a figure which improved by just 10% from 2016 to 2017. While ready-at-the-wheel drivers and remote operators make self-driving a viable technology today, we believe that the right type of simulator can drastically accelerate progress by giving more people the opportunity to contribute, allowing them to do so in software-only, and by creating more fluid competition between approaches.

Unfortunately, we were unable to license GTA V in a way that allowed us to distribute our previous versions, including Deepdrive 1.0 and Deepdrive Universe. We've now rebuilt everything on Unreal Engine, and with its full access to the engine source code, a great graphical editor, and the general ability to modify any part of the simulation, are elated about what we can bring to you in terms of capability, hackability, and transparency.

We think that something akin to a MNIST for self-driving is extremely important for beginners to get up and running easily and also for allowing researchers to quickly evaluate new ideas. Specifically, we see great promise in the ability to try methods like reinforcement learning in our initial Deepdrive 2.0 release, something that would be too dangerous to experiment with in the real world. We also know that matching the richness and variability of real-world driving within the simulation will be one of the keys to our success, and we will work aggressively towards creating the most true-to-life driving simulation possible.

Definitely! We plan to test in physical cars early and often by integrating with systems like Comma.ai's or Polysync's DriveKit. If you're interested in partnering with us on a hardware integration, please reach out at [email protected].

Mac support requires one of two things, a shared memory implementation or sending capture data over sockets. The latter would be slower but would enable better distributed processing on all platforms. Apple is incubating external GPU support which will bring MacOS on par with Linux and Windows in terms of single-machine deep learning setups. The simulation itself already runs on Mac, so it’s just a matter of sending and processing the data from the environment with sharedmem/sockets and GPU accelerated ML respectively. Slower-than-realtime support will be possible when we add synchronous stepping (see next FAQ).

The agent and environment currently operate asynchronously from each other except for resets and registering cameras. So if the environment's frame rate drops below the agent's step rate, the agent will receive blank frames from the environment, and if the opposite happens, frames will be skipped. Existing RL baselines like A2C use synchronized stepping across multiple environments through vectorized environments which we will support in v3. Also, thanks to the Carla.org team for their technical guidance on ways to do synchronized stepping within Unreal.

Sending sensor data via sockets was a bottleneck in our testing. Getting 8 HD cameras and depth at 60FPS, uncompressed back to the CPU gets too close to the limits of what, even highly tuned, sockets can do. We are still working on other bottlenecks (like sequential rendering of cameras), but it’s important, and possible to simulate a modern self-driving stack in real-time on one machine - and we will stay committed to doing that so developers with one machine can test their agents.

The baseline agent was trained on 8.2 hours of driving data using a variant of DAgger where the human labels are replaced by an oracle path follower (defined in Car.cpp). See tensorflow_agent/agent.py for exactly how we collect data. Starting from BVLC alexnet pretrained weights, we fine tuned for nearly 8 hours on a GTX 980. You can view our complete Tensorboard events here. N.B. we needed to mix Linux and Windows collection in order to have the agent perform well on both platforms (we used 80/20% Windows/Linux). Similarly, we mixed data rendered in the Unreal Editor with data rendered in the packaged version of the sim to get equal performance in both places. We think this is due to rendering differences between OpenGL, DirectX, and rendering optimizations made during packaging which points to the importance of using capture augmentation in addition to using domain adaptation techniques to increase transferability.