| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-24 | |||
| 19:52:43 | JamesBenson | dansmith, trust me, this is the first time and only because we are getting software that is based off of the xen hypervisor. | |
| 19:52:54 | JamesBenson | needless to say, I'm finding bugs. | |
| 19:53:03 | dansmith | I'm assuming this is the nova-compute log, but it looks really strange. definitely seems like you're failing to import os_xenapi for some reason that I can't help you with | |
| 19:53:27 | JamesBenson | yes, nova-compute log | |
| 19:53:31 | dansmith | the db accesses are super strange, but I'd resolve the import issue first in case that is kicking it into some weird state where it's trying to survey the db when it shouldn't | |
| 19:53:33 | JamesBenson | it's coming from kolla-ansible | |
| 19:53:54 | JamesBenson | So in the docker container, I can open python console and import sucessfully | |
| 19:54:49 | JamesBenson | I did have to tar the dir in site-packages, cp it over and expand... but console did work. | |
| 19:55:28 | dansmith | it's hard to tell from the traces, but I guess it's possible that os_xenapi is trying to access the db on init and that is preventing it from being imported, but that has been disallowed for many years | |
| 19:55:32 | dansmith | so I wouldn't know what to say | |
| 20:01:06 | JamesBenson | dansmith, are you familiar with kolla? | |
| 20:02:21 | dansmith | nope | |
| 20:09:04 | JamesBenson | Is it possible to check that nova is using openstack's os-xenapi instead of python-os-xenapi? | |
| 20:09:19 | JamesBenson | I've never played with nova code explicitly. | |
| 20:15:39 | efried | JamesBenson: The nova requirements.txt file lists: | |
| 20:15:39 | efried | requirements.txt:os-xenapi>=0.3.3 # Apache-2.0 | |
| 20:16:02 | efried | which means whatever happens when you say pip install 'os-xenapi>=0.3.3' | |
| 20:16:56 | efried | aspiers: done, thanks for the reminder. | |
| 20:18:51 | efried | sean-k-mooney: re libvirt-video-device-models, the action from last week was for kashyap to have a look at it and I would approve it if he was on board. | |
| 20:19:25 | efried | I know kashyap has been one-armed-wallpaperer busy for the past few days... | |
| 20:36:10 | sean-k-mooney | efried: ya that fine | |
| 20:36:18 | sean-k-mooney | it cna wait until after the ptg | |
| 20:36:37 | sean-k-mooney | i was going to add it to the runway list but noticed it wasnt approved so held off | |
| 20:36:55 | sean-k-mooney | i have set a workflow -1 on the nova patch in the inerim | |
| 20:40:47 | JamesBenson | efried, so I wonder why it wasn't installed by default | |
| 20:41:16 | efried | sorry, I wasn't following the conversation, let me read scrollback... | |
| 20:41:39 | gmann | efried: regarding PT schedule, can we move 'API inconsistency cleanup' L276 on Thursday or friday ? I would not be available on Saturday due to my flight | |
| 20:41:46 | gmann | PTG | |
| 20:42:03 | efried | gmann: Roger that, will shuffle, thanks for the heads up. | |
| 20:42:17 | gmann | efried: thanks . | |
| 20:42:18 | JamesBenson | efried, sean-k-mooney could you provide insight too, I've seen you in kolla :-) | |
| 20:42:36 | efried | you haven't seen *me* in kolla :P | |
| 20:42:52 | JamesBenson | sorry to pick on you explicitly, oh man, sorry efried! | |
| 20:43:00 | sean-k-mooney | JamesBenson: context? | |
| 20:43:16 | efried | (I have nothing against kolla, just completely ignorant/inexperienced there) | |
| 20:43:44 | sean-k-mooney | reading back | |
| 20:43:57 | JamesBenson | sean-k-mooney: I'm doing kolla with xen, the nova_compute container is having issues with os_xenapi and six, I had to manually install them, but still vomits on itself. | |
| 20:44:34 | JamesBenson | sean-k-mooney, check out the paste for the error log... | |
| 20:45:41 | sean-k-mooney | JamesBenson: i honestly did not know kolla-ansible supproted xen i always used it with kvm | |
| 20:45:58 | JamesBenson | likewise, until we got software that needs a xenblanket... | |
| 20:46:48 | sean-k-mooney | i would gues in addtion to installing xen you might need to run some db migrations | |
| 20:47:27 | sean-k-mooney | oh xen server is supported now https://github.com/openstack/kolla-ansible/blob/23415bf8bb8179a966d1c10bd46e6cb9045f2b5d/doc/source/reference/compute/xenserver-guide.rst | |
| 20:47:28 | JamesBenson | doing this little experiment, I'm finding bugs in several different openstack processes... :-( | |
| 20:47:37 | JamesBenson | yeah | |
| 20:47:54 | sean-k-mooney | what version of python are you using | |
| 20:48:02 | sean-k-mooney | i assume python 2 | |
| 20:48:03 | JamesBenson | this is version 2 still | |
| 20:48:24 | sean-k-mooney | ya it 2.7 from the logs cool | |
| 20:49:05 | sean-k-mooney | so this is in the cells scater gather code | |
| 20:49:26 | sean-k-mooney | we indocuded explcitly use fo eventlets there | |
| 20:49:43 | JamesBenson | ? | |
| 20:50:08 | sean-k-mooney | in stien we intoduced a parallel cell list feature | |
| 20:50:14 | JamesBenson | yes, I think it was failing at the cells spot in ansible | |
| 20:50:23 | JamesBenson | I'm deploying using 7.0.1 | |
| 20:50:54 | sean-k-mooney | it was implemented using eventlets expclitly but that means in stien you need to monkey patch the nova api now | |
| 20:51:39 | sean-k-mooney | but some people have been having issue with monkey patched nova api under uwsigi also | |
| 20:51:50 | sean-k-mooney | do you know how this is run in in kolla | |
| 20:52:37 | JamesBenson | honestly, starting to get to the limits of my knowledge here now. | |
| 20:52:59 | sean-k-mooney | im checking the kolla repos to see | |
| 20:54:11 | JamesBenson | I do know this needs to get fixed in kolla-ansible: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/bootstrap_xenapi.yml#L3 and L4, | |
| 20:54:24 | efried | mriedem: pike-em good to go at this point? | |
| 20:55:32 | sean-k-mooney | JamesBenson: the command is nova-api so i think it running under pythons built in web server with haproxy in front | |
| 20:56:13 | sean-k-mooney | at least based on https://github.com/openstack/kolla-ansible/blob/7.0.1/ansible/roles/nova/templates/nova-api.json.j2#L2 | |
| 20:58:16 | sean-k-mooney | JamesBenson: what people were seeing was under uwsgi instead of mod_wsig the nova-api was not sending/reciving heartbeats to rabbitmq and teh connect to amqp was dropping | |
| 20:58:34 | sean-k-mooney | although im not sure that is related to your issue | |
| 21:00:05 | sean-k-mooney | JamesBenson: how did you install os_xenapi | |
| 21:00:31 | sean-k-mooney | did you rebuild the images using the template overide file | |
| 21:01:38 | JamesBenson | it was a multistep process.... but tar'ed the package locally from python2.7/site-packages, docker cp, untar... I issued the commands using docker exec -u 0 | |
| 21:02:08 | sean-k-mooney | oh there is a standard way to extend kolla images | |
| 21:02:11 | JamesBenson | I verified inside the container by logging into it, running python, and importing the packages without issue | |
| 21:02:28 | JamesBenson | yeah, I didn't want to rebuild the images as I was sure there would be more issues. | |
| 21:02:43 | sean-k-mooney | https://github.com/openstack/kolla/blob/master/doc/source/admin/image-building.rst#generic-customisation | |
| 21:03:02 | sean-k-mooney | did you pull the image form the docker hub | |
| 21:03:07 | JamesBenson | yes | |
| 21:03:21 | sean-k-mooney | ok | |
| 21:03:27 | sean-k-mooney | let me check something | |
| 21:04:24 | JamesBenson | so I'm good at debugging but don't do some processes often enough to remember how to do it, like submitted a bug, but now submitting the bug fix :-/. lol, downfall of academia and not industry, it's not a tool we use all too often. | |
| 21:04:42 | JamesBenson | I know :how: to fix it, just submitting it... | |
| 21:05:56 | sean-k-mooney | well i was going to check if they had a configuration option in the kolla build config to enable building with xen then compare that to the build setting used in the gate to publish the image | |
| 21:06:06 | sean-k-mooney | but no no buil option that i can see in the config | |
| 21:07:16 | sean-k-mooney | and just checking you rna the xen_bootsrap command | |
| 21:08:23 | JamesBenson | yes had issues, but seemed to build the facts.json: https://da.gd/E0FIl | |
| 21:09:53 | JamesBenson | and I can literally submit about 3 patches to fix the issues I had with xen, if I can bug you about the process privately. | |
| 21:10:26 | sean-k-mooney | well i havent working on kolla in about 18 months but if i can help sure | |
| 21:10:50 | sean-k-mooney | wel proably should also bring this to the openstack-kolla channel | |
| 21:11:05 | JamesBenson | because there are bugs in kolla-ansible, openstack os-xenapi, and what ever the issue is here or kolla with not installing the packages correctly in the docker container. | |
| 21:11:55 | sean-k-mooney | im just loggin off for the night but feel free to ping me tommorow | |
| 21:12:07 | sean-k-mooney | or ping me while im away and ill respond tomorow | |
| 21:12:25 | JamesBenson | Lets talk tomorrow, I'm logging off soon too | |
| 21:13:11 | sean-k-mooney | cool my irc client is connected to #openstack-kolla also so feel free to ping me there so we dont distub the nova folks for the packaging side | |
| 21:13:28 | sean-k-mooney | the nova issue we can prably debug more here | |
| 21:13:34 | sean-k-mooney | is it repoducable? | |
| 21:13:48 | JamesBenson | most likely. | |
| 21:14:09 | sean-k-mooney | only with xen or do you think i would hit it with libvir kvm too | |
| 21:14:27 | sean-k-mooney | this does not look realted to xen to be honest | |
| 21:14:36 | sean-k-mooney | well the db issues dont | |
| 21:15:19 | sean-k-mooney | there are some other issue with the xenapi too at the end | |
| 21:15:25 | mriedem | efried: yeah i've had the patch up since earlier today | |
| 21:16:38 | JamesBenson | I'm not sure where it exactly crosses over, but I know definitely those two packages have issues, kolla-ansible and os-xenapi. Nova will need to dig deeper into to see where it breaks. | |
| 21:16:49 | JamesBenson | But I've never dug into nova this deep. | |