Hi there,
If I submit code written in python3, can your backend handle it? Otherwise, any support for specifying a docker image?
Thank you!
Lisheng
Posted by: lise_sun @ Dec. 6, 2017, 3:57 p.m.Unfortunately we have disabled the ability for participants to specify a docker to run their scripts in. Competition organizers can still specify a special image.
You could create an image with py2 and py3 inside and specify the version for your script? Instead of `python script.py` you say `python3 script.py` in your metadata.
Posted by: eric2 @ Dec. 6, 2017, 8:44 p.m.The default docker is ckcollab/codalab-legacy. It can be found on Github at https://github.com/codalab/docker-codalab-legacy-worker.
Posted by: guyon @ Dec. 6, 2017, 11:05 p.m.Dear organizers,
We would also love to use Python3, and potentially an up-to-date version of scikit-learn (0.19.1). Could you please let us know whether this is currently supported or whether you plan to support this, for example by installing a separate AnaConda with Python3 in the docker container or by using a different docker image. We'd prefer to not ship a full python environment with every submission.
Please let us know whether we can help you with this somehow.
Best,
AAD Freiburg
We are looking at various possibilities and will get back to you.
Isabelle
Dear organizers,
are there any news on this?
Best,
AAD Freiburg
We created a new docker codalab/codalab-legacy:py3
Can you please check whether it works with your software?
Here is the docker file:
https://github.com/codalab/docker-codalab-legacy/blob/master/Dockerfile#L26
We are checking that rolling it will be backward compatible. If this works for you, we'll switch over to this new docker.
Posted by: guyon @ Feb. 13, 2018, 9:30 a.m.Also, you may need to change your metadata in submissions from python yourscript.py to python3 yourscript.py
Posted by: eric2 @ Feb. 13, 2018, 11:10 a.m.Thanks a lot for taking care of this. We assume that this one here is the actual dockerfile: https://github.com/codalab/docker-codalab-legacy/blob/py3/Dockerfile
While it installs python3, it does not install AnaConda's standard libraries. They will only be aivalable in their python2 version and can thus not be loaded. We did a minor, hacky modification to actually install scikit-learn and pandas as well (and removed COCO and the image loading library):
# Python requirements
FROM debian:8.5
# Install anaconda
RUN apt-get update --fix-missing && apt-get install -y wget bzip2 ca-certificates \
libglib2.0-0 libxext6 libsm6 libxrender1 \
git mercurial subversion
RUN echo 'export PATH=/opt/conda/bin:$PATH' > /etc/profile.d/conda.sh && \
wget --quiet https://repo.continuum.io/archive/Anaconda2-2.5.0-Linux-x86_64.sh && \
/bin/bash /Anaconda2-2.5.0-Linux-x86_64.sh -b -p /opt/conda && \
rm /Anaconda2-2.5.0-Linux-x86_64.sh
RUN apt-get install -y curl grep sed dpkg && \
TINI_VERSION=`curl https://github.com/krallin/tini/releases/latest | grep -o "/v.*\"" | sed 's:^..\(.*\).$:\1:'` && \
curl -L "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini_${TINI_VERSION}.deb" > tini.deb && \
dpkg -i tini.deb && \
rm tini.deb && \
apt-get clean
ENV PATH /opt/conda/bin:$PATH
# http://bugs.python.org/issue19846
# > At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK.
ENV LANG C.UTF-8
RUN conda install python=3.6
# Fix for 'xgboost' missing
RUN conda install libgcc
# Java, C#, R, etc. requirements
RUN apt-get update && apt-get install -y --fix-missing openjdk-7-jre r-base mono-runtime libgomp1 libc6
# Helpful python libraries
RUN pip install \
theano==0.9.0 \
Cython==0.27.3 \
numpy==1.14.0 \
scipy==1.0.0 \
scikit-learn==0.19.1 \
pandas==0.22.0
Okay, we checked this a little bit more, and pyyaml is missing in python3, too. Also, AnaConda would by default call python3 if a user calls python, and I assume that you'd need to set an alias or specify the python version in the metadata file.
Besides that, the starter kit can be adapted to be run in both python2 and python3 with very little effort (replacing tabs by spaces, replacing print statements by calls to the print function) to maintain backwards compatibility.
Posted by: aad_freiburg @ Feb. 14, 2018, 1:37 p.m.Ah, my mistake for the oversight on missing packages for python 3. I could have sworn I tested and made sure at least a couple packages were available -- must have overlooked that.
The previously pasted Dockerfile, is that the latest working one? Feel free to submit a PR here:
https://github.com/codalab/docker-codalab-legacy/tree/py3
or paste the latest and I'll update it!
Posted by: eric2 @ Feb. 14, 2018, 10:51 p.m.We created a PR to the github repository.
Posted by: aad_freiburg @ Feb. 20, 2018, 3:20 p.m.Alright, here's a new version of the docker image to test:
https://github.com/codalab/docker-codalab-legacy/tree/py3
The image name:
codalab/codalab-legacy:py3
You can test your scripts locally like so:
docker run -it codalab/codalab-legacy:py3 python <your script.py>
***OR for Python 3:***
docker run -it codalab/codalab-legacy:py3 python3 <your script.py>
python points to version 2, python3 points to version 3
Please feel free to shoot me an email eric@ckcollab.com if you have any questions or urgent problems! I installed Python3 a different way from previous versions, please let me know if there is some package missing or something else, I didn't have any Python3 submissions to test with.
Posted by: eric2 @ Feb. 22, 2018, 7:13 p.m.Hi, we just tested the new docker image with a Python3 port of the starter kit and it works both locally and online. To make it easier for you and others to run Python3 code, we made our submission public.
We'll check later today whether our actual submission is running smoothly on the new docker image.
Posted by: aad_freiburg @ Feb. 27, 2018, 9:18 a.m.Excellent news! thank you very much for sharing your submission aad_freiburg
Best