| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-24 | |||
| 20:36:55 | sean-k-mooney | i have set a workflow -1 on the nova patch in the inerim | |
| 20:40:47 | JamesBenson | efried, so I wonder why it wasn't installed by default | |
| 20:41:16 | efried | sorry, I wasn't following the conversation, let me read scrollback... | |
| 20:41:39 | gmann | efried: regarding PT schedule, can we move 'API inconsistency cleanup' L276 on Thursday or friday ? I would not be available on Saturday due to my flight | |
| 20:41:46 | gmann | PTG | |
| 20:42:03 | efried | gmann: Roger that, will shuffle, thanks for the heads up. | |
| 20:42:17 | gmann | efried: thanks . | |
| 20:42:18 | JamesBenson | efried, sean-k-mooney could you provide insight too, I've seen you in kolla :-) | |
| 20:42:36 | efried | you haven't seen *me* in kolla :P | |
| 20:42:52 | JamesBenson | sorry to pick on you explicitly, oh man, sorry efried! | |
| 20:43:00 | sean-k-mooney | JamesBenson: context? | |
| 20:43:16 | efried | (I have nothing against kolla, just completely ignorant/inexperienced there) | |
| 20:43:44 | sean-k-mooney | reading back | |
| 20:43:57 | JamesBenson | sean-k-mooney: I'm doing kolla with xen, the nova_compute container is having issues with os_xenapi and six, I had to manually install them, but still vomits on itself. | |
| 20:44:34 | JamesBenson | sean-k-mooney, check out the paste for the error log... | |
| 20:45:41 | sean-k-mooney | JamesBenson: i honestly did not know kolla-ansible supproted xen i always used it with kvm | |
| 20:45:58 | JamesBenson | likewise, until we got software that needs a xenblanket... | |
| 20:46:48 | sean-k-mooney | i would gues in addtion to installing xen you might need to run some db migrations | |
| 20:47:27 | sean-k-mooney | oh xen server is supported now https://github.com/openstack/kolla-ansible/blob/23415bf8bb8179a966d1c10bd46e6cb9045f2b5d/doc/source/reference/compute/xenserver-guide.rst | |
| 20:47:28 | JamesBenson | doing this little experiment, I'm finding bugs in several different openstack processes... :-( | |
| 20:47:37 | JamesBenson | yeah | |
| 20:47:54 | sean-k-mooney | what version of python are you using | |
| 20:48:02 | sean-k-mooney | i assume python 2 | |
| 20:48:03 | JamesBenson | this is version 2 still | |
| 20:48:24 | sean-k-mooney | ya it 2.7 from the logs cool | |
| 20:49:05 | sean-k-mooney | so this is in the cells scater gather code | |
| 20:49:26 | sean-k-mooney | we indocuded explcitly use fo eventlets there | |
| 20:49:43 | JamesBenson | ? | |
| 20:50:08 | sean-k-mooney | in stien we intoduced a parallel cell list feature | |
| 20:50:14 | JamesBenson | yes, I think it was failing at the cells spot in ansible | |
| 20:50:23 | JamesBenson | I'm deploying using 7.0.1 | |
| 20:50:54 | sean-k-mooney | it was implemented using eventlets expclitly but that means in stien you need to monkey patch the nova api now | |
| 20:51:39 | sean-k-mooney | but some people have been having issue with monkey patched nova api under uwsigi also | |
| 20:51:50 | sean-k-mooney | do you know how this is run in in kolla | |
| 20:52:37 | JamesBenson | honestly, starting to get to the limits of my knowledge here now. | |
| 20:52:59 | sean-k-mooney | im checking the kolla repos to see | |
| 20:54:11 | JamesBenson | I do know this needs to get fixed in kolla-ansible: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/bootstrap_xenapi.yml#L3 and L4, | |
| 20:54:24 | efried | mriedem: pike-em good to go at this point? | |
| 20:55:32 | sean-k-mooney | JamesBenson: the command is nova-api so i think it running under pythons built in web server with haproxy in front | |
| 20:56:13 | sean-k-mooney | at least based on https://github.com/openstack/kolla-ansible/blob/7.0.1/ansible/roles/nova/templates/nova-api.json.j2#L2 | |
| 20:58:16 | sean-k-mooney | JamesBenson: what people were seeing was under uwsgi instead of mod_wsig the nova-api was not sending/reciving heartbeats to rabbitmq and teh connect to amqp was dropping | |
| 20:58:34 | sean-k-mooney | although im not sure that is related to your issue | |
| 21:00:05 | sean-k-mooney | JamesBenson: how did you install os_xenapi | |
| 21:00:31 | sean-k-mooney | did you rebuild the images using the template overide file | |
| 21:01:38 | JamesBenson | it was a multistep process.... but tar'ed the package locally from python2.7/site-packages, docker cp, untar... I issued the commands using docker exec -u 0 | |
| 21:02:08 | sean-k-mooney | oh there is a standard way to extend kolla images | |
| 21:02:11 | JamesBenson | I verified inside the container by logging into it, running python, and importing the packages without issue | |
| 21:02:28 | JamesBenson | yeah, I didn't want to rebuild the images as I was sure there would be more issues. | |
| 21:02:43 | sean-k-mooney | https://github.com/openstack/kolla/blob/master/doc/source/admin/image-building.rst#generic-customisation | |
| 21:03:02 | sean-k-mooney | did you pull the image form the docker hub | |
| 21:03:07 | JamesBenson | yes | |
| 21:03:21 | sean-k-mooney | ok | |
| 21:03:27 | sean-k-mooney | let me check something | |
| 21:04:24 | JamesBenson | so I'm good at debugging but don't do some processes often enough to remember how to do it, like submitted a bug, but now submitting the bug fix :-/. lol, downfall of academia and not industry, it's not a tool we use all too often. | |
| 21:04:42 | JamesBenson | I know :how: to fix it, just submitting it... | |
| 21:05:56 | sean-k-mooney | well i was going to check if they had a configuration option in the kolla build config to enable building with xen then compare that to the build setting used in the gate to publish the image | |
| 21:06:06 | sean-k-mooney | but no no buil option that i can see in the config | |
| 21:07:16 | sean-k-mooney | and just checking you rna the xen_bootsrap command | |
| 21:08:23 | JamesBenson | yes had issues, but seemed to build the facts.json: https://da.gd/E0FIl | |
| 21:09:53 | JamesBenson | and I can literally submit about 3 patches to fix the issues I had with xen, if I can bug you about the process privately. | |
| 21:10:26 | sean-k-mooney | well i havent working on kolla in about 18 months but if i can help sure | |
| 21:10:50 | sean-k-mooney | wel proably should also bring this to the openstack-kolla channel | |
| 21:11:05 | JamesBenson | because there are bugs in kolla-ansible, openstack os-xenapi, and what ever the issue is here or kolla with not installing the packages correctly in the docker container. | |
| 21:11:55 | sean-k-mooney | im just loggin off for the night but feel free to ping me tommorow | |
| 21:12:07 | sean-k-mooney | or ping me while im away and ill respond tomorow | |
| 21:12:25 | JamesBenson | Lets talk tomorrow, I'm logging off soon too | |
| 21:13:11 | sean-k-mooney | cool my irc client is connected to #openstack-kolla also so feel free to ping me there so we dont distub the nova folks for the packaging side | |
| 21:13:28 | sean-k-mooney | the nova issue we can prably debug more here | |
| 21:13:34 | sean-k-mooney | is it repoducable? | |
| 21:13:48 | JamesBenson | most likely. | |
| 21:14:09 | sean-k-mooney | only with xen or do you think i would hit it with libvir kvm too | |
| 21:14:27 | sean-k-mooney | this does not look realted to xen to be honest | |
| 21:14:36 | sean-k-mooney | well the db issues dont | |
| 21:15:19 | sean-k-mooney | there are some other issue with the xenapi too at the end | |
| 21:15:25 | mriedem | efried: yeah i've had the patch up since earlier today | |
| 21:16:38 | JamesBenson | I'm not sure where it exactly crosses over, but I know definitely those two packages have issues, kolla-ansible and os-xenapi. Nova will need to dig deeper into to see where it breaks. | |
| 21:16:49 | JamesBenson | But I've never dug into nova this deep. | |
| 21:18:11 | sean-k-mooney | well there seams to be 2 basic erros in the nova log | |
| 21:18:15 | sean-k-mooney | DBNotAllowed: nova-compute | |
| 21:18:20 | sean-k-mooney | and then the xen api one | |
| 21:18:59 | JamesBenson | agreed | |
| 21:19:24 | JamesBenson | Is it possible that kolla mucked up the install of that package? | |
| 21:19:52 | JamesBenson | But I thought that was pulled from the nova repo. | |
| 21:19:53 | JamesBenson | ? | |
| 21:20:47 | sean-k-mooney | well its posibel that the xen dirver need other config option to be set that kolla is nto setting in the nova.conf | |
| 21:22:28 | sean-k-mooney | the default for the databse are set here https://github.com/openstack/kolla-ansible/blob/2dd69e9140b1ce1bd248c5c09217fb3a6502a9fc/ansible/roles/nova/defaults/main.yml#L161-L167 and used to generate teh config nova.conf here https://github.com/openstack/kolla-ansible/blob/2dd69e9140b1ce1bd248c5c09217fb3a6502a9fc/ansible/roles/nova/templates/nova.conf.j2#L156-L164 | |
| 21:23:01 | sean-k-mooney | that code should be virt dirver independent | |
| 21:23:25 | mriedem | melwitt: heh yeah that PS4 on the transactional create patch is about a billion times cleaner | |
| 21:25:58 | mriedem | efried: https://review.opendev.org/#/c/652868/ | |
| 21:30:21 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.opendev.org/642899 | |
| 21:32:23 | efried | mriedem: Right, just wanted to make sure that was the final | |
| 21:32:39 | efried | mriedem: and presumably the one on top fails tests until the bottom one merges and the tag is created. | |
| 21:36:17 | jaypipes | dansmith, mriedem: is it possible to update metadata on a server that failed to boot properly? (i.e. the instance went to cell0) | |
| 21:36:35 | dansmith | jaypipes: metadata? like user metadata? | |
| 21:37:10 | mriedem | efried: i'm trying to figure out why the one on top is failing | |
| 21:37:12 | jaypipes | dansmith: no, sorry, metadata was a bad term. any attribute of the instance... say, it's name. | |
| 21:37:27 | efried | mriedem: It's because the tags aren't created yet | |
| 21:37:44 | efried | mriedem: I guess the job isn't smart enough to infer them from the releases in the predecessor patch. | |
| 21:37:57 | dansmith | jaypipes: shouldn't really be anything cell0 specific I don't think.. if you can do it in error, then you can do it | |
| 21:38:06 | dansmith | jaypipes: anything that requires talking to the host the instance is on won't work, of course | |