Hello!
I am trying to simulate multiple social vehicles and all my experiments (included the provided rnn_example) are failing at some early point with the following:
traci.exceptions.FatalTraCIError: connection closed by SUMO
2019-11-06 09:31:58,162 INFO trial_runner.py:615 -- Attempting to recover trial state from last checkpoint.
(pid=14455) sumo: /build/sumo-tHynv2/sumo-1.2.0+dfsg1/src/microsim/lcmodels/MSLCM_LC2013.cpp:1905: static double MSLCM_LC2013::distanceAlongNextRoundabout(double, const MSLane*, const std::vector<MSLane*>&): Assertion `initialLane == 0' failed.
The whole output is:
2019-11-06 09:40:39,599 ERROR trial_runner.py:569 -- Error processing event.
Traceback (most recent call last):
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 515, in _process_trial
result = self.trial_executor.fetch_result(trial)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py", line 351, in fetch_result
result = ray.get(trial_future[0])
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/worker.py", line 2121, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(FatalTraCIError): ray_worker (pid=16644, host=elena-HP-Pavilion-Laptop-14-bf1xx)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 418, in train
raise e
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 407, in train
result = Trainable.train(self)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/tune/trainable.py", line 176, in train
result = self._train()
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py", line 129, in _train
fetches = self.optimizer.step()
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/optimizers/multi_gpu_optimizer.py", line 140, in step
self.num_envs_per_worker, self.train_batch_size)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/optimizers/rollout.py", line 29, in collect_samples
next_sample = ray_get_and_free(fut_sample)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/utils/memory.py", line 33, in ray_get_and_free
result = ray.get(object_ids)
ray.exceptions.RayTaskError(FatalTraCIError): ray_worker (pid=16643, host=elena-HP-Pavilion-Laptop-14-bf1xx)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/evaluation/rollout_worker.py", line 469, in sample
batches = [self.input_reader.next()]
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/evaluation/sampler.py", line 56, in next
batches = [self.get_data()]
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/evaluation/sampler.py", line 99, in get_data
item = next(self.rollout_provider)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/evaluation/sampler.py", line 340, in _env_runner
base_env.send_actions(actions_to_send)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/env/base_env.py", line 332, in send_actions
self.vector_env.vector_step(action_vector)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/ray/rllib/env/vector_env.py", line 110, in vector_step
obs, r, done, info = self.envs[i].step(actions[i])
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/gym_hiway/env/competition_env.py", line 139, in step
observation, reward, agent_dones = self._engine.step(action_msg)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/hiway/game_engine.py", line 53, in step
return self._step(action_msg)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/hiway/game_engine.py", line 89, in _step
sim.step_traffic(dt)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/hiway/bullet_simulation.py", line 173, in step_traffic
social_vehicle_state = self._traffic_sim.step(dt, agent_vehicle_states)
File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/hiway/sumo_traffic_simulation.py", line 181, in step
self._traci_conn.simulationStep(self._cumulative_sim_seconds)
File "/usr/share/sumo/tools/traci/connection.py", line 329, in simulationStep
result = self._sendExact()
File "/usr/share/sumo/tools/traci/connection.py", line 105, in _sendExact
raise FatalTraCIError("connection closed by SUMO")
traci.exceptions.FatalTraCIError: connection closed by SUMO
(pid=16643) sumo: /build/sumo-tHynv2/sumo-1.2.0+dfsg1/src/microsim/lcmodels/MSLCM_LC2013.cpp:1905: static double MSLCM_LC2013::distanceAlongNextRoundabout(double, const MSLane*, const std::vector<MSLane*>&): Assertion `initialLane == 0' failed.
(pid=16643) ERROR:GameEngine:Simulation crashed with exception.Attempting to cleanly shutdown.
(pid=16643) ERROR:GameEngine:connection closed by SUMO
(pid=16643) Traceback (most recent call last):
(pid=16643) File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/hiway/game_engine.py", line 53, in step
(pid=16643) return self._step(action_msg)
(pid=16643) File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/hiway/game_engine.py", line 89, in _step
(pid=16643) sim.step_traffic(dt)
(pid=16643) File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/hiway/bullet_simulation.py", line 173, in step_traffic
(pid=16643) social_vehicle_state = self._traffic_sim.step(dt, agent_vehicle_states)
(pid=16643) File "/home/elena/anaconda3/envs/huawei/lib/python3.7/site-packages/hiway/sumo_traffic_simulation.py", line 181, in step
(pid=16643) self._traci_conn.simulationStep(self._cumulative_sim_seconds)
(pid=16643) File "/usr/share/sumo/tools/traci/connection.py", line 329, in simulationStep
(pid=16643) result = self._sendExact()
(pid=16643) File "/usr/share/sumo/tools/traci/connection.py", line 105, in _sendExact
(pid=16643) raise FatalTraCIError("connection closed by SUMO")
(pid=16643) traci.exceptions.FatalTraCIError: connection closed by SUMO
(pid=16643) Step #0.00 (0ms ?*RT. ?UPS, TraCI: 268ms, vehicles TOT 0 ACT 0 BUF 0)
2019-11-06 09:40:39,612 INFO trial_runner.py:615 -- Attempting to recover trial state from last checkpoint.
Which version of SUMO are you using? We found this error with older versions of SUMO but it should go away if you upgrade to SUMO 1.3.1.
Posted by: HuaweiUK @ Nov. 6, 2019, 10:06 a.m.Indeed, I am using 1.2:
sumo:
Installed: 1.2.0+dfsg1-14~xenial
Candidate: 1.2.0+dfsg1-14~xenial
Version table:
*** 1.2.0+dfsg1-14~xenial 500
500 http://ppa.launchpad.net/sumo/stable/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
0.25.0+dfsg1-2 500
500 http://gb.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
However, I am not sure how to upgrade it. Running "sudo apt-get install sumo=1.3.1" returns that version 1.3.1 for sumo was not found. I have also added the repository "ppa:/sumo/stable", but it does not have the latest version cause I am getting no upgrades when running apt-get update. I also tried to install it from the source file (https://sourceforge.net/projects/sumo/) and follow the instructions here https://sumo.dlr.de/docs/Installing/Linux_Build.html, but I am getting errors.
Perhaps there is an easy way to upgrade?
Posted by: Team_29 @ Nov. 6, 2019, 12:53 p.m.Ah, installation from source worked. Looks like there's no exceptions now!
Thanks!
Posted by: Team_29 @ Nov. 6, 2019, 1:17 p.m.