ISC High Performance: International Conference on High Performance Computing., Ever growing interest and usage of deep learning rises a question on the performance of various infrastructures suitable for training of neural networks. We present here our approach and first results of tests performed with TensorFlow Benchmarks which use best practices for multi-GPU and distributed training. We pack the Benchmarks in Docker containers and execute them by means of uDocker and Singularity container tools on a single machine and in the HPC environment. The Benchmarks comprise a number of convolutional neural network models run across synthetic data and e.g. the ImageNet dataset. For the same Nvidia K80 GPU card we achieve the same performance in terms of processed images per second and similar scalability between 1-2-4 GPUs as presented by the TensorFlow developers. We therefore do not obtain statistically significant overhead due to the usage of containers in the multi-GPU case, and the approach of using TF Benchmarks in a Docker container can be applied across various systems., uDocker is being developed within the DEEP HybridDataCloud project, which receives funding from the European Union’s Horizon 2020 research and innovation program under agreement RIA 777435. A part of this work was performed on the computational resource ForHLR-II funded by the Ministry of Science, Research and the Arts Baden-Wuerttemberg and DFG (“Deutsche Forschungsgemeinschaft”).