| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-24 | |||
| 20:47:28 | JamesBenson | doing this little experiment, I'm finding bugs in several different openstack processes... :-( | |
| 20:47:37 | JamesBenson | yeah | |
| 20:47:54 | sean-k-mooney | what version of python are you using | |
| 20:48:02 | sean-k-mooney | i assume python 2 | |
| 20:48:03 | JamesBenson | this is version 2 still | |
| 20:48:24 | sean-k-mooney | ya it 2.7 from the logs cool | |
| 20:49:05 | sean-k-mooney | so this is in the cells scater gather code | |
| 20:49:26 | sean-k-mooney | we indocuded explcitly use fo eventlets there | |
| 20:49:43 | JamesBenson | ? | |
| 20:50:08 | sean-k-mooney | in stien we intoduced a parallel cell list feature | |
| 20:50:14 | JamesBenson | yes, I think it was failing at the cells spot in ansible | |
| 20:50:23 | JamesBenson | I'm deploying using 7.0.1 | |
| 20:50:54 | sean-k-mooney | it was implemented using eventlets expclitly but that means in stien you need to monkey patch the nova api now | |
| 20:51:39 | sean-k-mooney | but some people have been having issue with monkey patched nova api under uwsigi also | |
| 20:51:50 | sean-k-mooney | do you know how this is run in in kolla | |
| 20:52:37 | JamesBenson | honestly, starting to get to the limits of my knowledge here now. | |
| 20:52:59 | sean-k-mooney | im checking the kolla repos to see | |
| 20:54:11 | JamesBenson | I do know this needs to get fixed in kolla-ansible: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/bootstrap_xenapi.yml#L3 and L4, | |
| 20:54:24 | efried | mriedem: pike-em good to go at this point? | |
| 20:55:32 | sean-k-mooney | JamesBenson: the command is nova-api so i think it running under pythons built in web server with haproxy in front | |
| 20:56:13 | sean-k-mooney | at least based on https://github.com/openstack/kolla-ansible/blob/7.0.1/ansible/roles/nova/templates/nova-api.json.j2#L2 | |
| 20:58:16 | sean-k-mooney | JamesBenson: what people were seeing was under uwsgi instead of mod_wsig the nova-api was not sending/reciving heartbeats to rabbitmq and teh connect to amqp was dropping | |
| 20:58:34 | sean-k-mooney | although im not sure that is related to your issue | |
| 21:00:05 | sean-k-mooney | JamesBenson: how did you install os_xenapi | |
| 21:00:31 | sean-k-mooney | did you rebuild the images using the template overide file | |
| 21:01:38 | JamesBenson | it was a multistep process.... but tar'ed the package locally from python2.7/site-packages, docker cp, untar... I issued the commands using docker exec -u 0 | |
| 21:02:08 | sean-k-mooney | oh there is a standard way to extend kolla images | |
| 21:02:11 | JamesBenson | I verified inside the container by logging into it, running python, and importing the packages without issue | |
| 21:02:28 | JamesBenson | yeah, I didn't want to rebuild the images as I was sure there would be more issues. | |
| 21:02:43 | sean-k-mooney | https://github.com/openstack/kolla/blob/master/doc/source/admin/image-building.rst#generic-customisation | |
| 21:03:02 | sean-k-mooney | did you pull the image form the docker hub | |
| 21:03:07 | JamesBenson | yes | |
| 21:03:21 | sean-k-mooney | ok | |
| 21:03:27 | sean-k-mooney | let me check something | |
| 21:04:24 | JamesBenson | so I'm good at debugging but don't do some processes often enough to remember how to do it, like submitted a bug, but now submitting the bug fix :-/. lol, downfall of academia and not industry, it's not a tool we use all too often. | |
| 21:04:42 | JamesBenson | I know :how: to fix it, just submitting it... | |
| 21:05:56 | sean-k-mooney | well i was going to check if they had a configuration option in the kolla build config to enable building with xen then compare that to the build setting used in the gate to publish the image | |
| 21:06:06 | sean-k-mooney | but no no buil option that i can see in the config | |
| 21:07:16 | sean-k-mooney | and just checking you rna the xen_bootsrap command | |
| 21:08:23 | JamesBenson | yes had issues, but seemed to build the facts.json: https://da.gd/E0FIl | |
| 21:09:53 | JamesBenson | and I can literally submit about 3 patches to fix the issues I had with xen, if I can bug you about the process privately. | |
| 21:10:26 | sean-k-mooney | well i havent working on kolla in about 18 months but if i can help sure | |
| 21:10:50 | sean-k-mooney | wel proably should also bring this to the openstack-kolla channel | |
| 21:11:05 | JamesBenson | because there are bugs in kolla-ansible, openstack os-xenapi, and what ever the issue is here or kolla with not installing the packages correctly in the docker container. | |
| 21:11:55 | sean-k-mooney | im just loggin off for the night but feel free to ping me tommorow | |
| 21:12:07 | sean-k-mooney | or ping me while im away and ill respond tomorow | |
| 21:12:25 | JamesBenson | Lets talk tomorrow, I'm logging off soon too | |
| 21:13:11 | sean-k-mooney | cool my irc client is connected to #openstack-kolla also so feel free to ping me there so we dont distub the nova folks for the packaging side | |
| 21:13:28 | sean-k-mooney | the nova issue we can prably debug more here | |
| 21:13:34 | sean-k-mooney | is it repoducable? | |
| 21:13:48 | JamesBenson | most likely. | |
| 21:14:09 | sean-k-mooney | only with xen or do you think i would hit it with libvir kvm too | |
| 21:14:27 | sean-k-mooney | this does not look realted to xen to be honest | |
| 21:14:36 | sean-k-mooney | well the db issues dont | |
| 21:15:19 | sean-k-mooney | there are some other issue with the xenapi too at the end | |
| 21:15:25 | mriedem | efried: yeah i've had the patch up since earlier today | |
| 21:16:38 | JamesBenson | I'm not sure where it exactly crosses over, but I know definitely those two packages have issues, kolla-ansible and os-xenapi. Nova will need to dig deeper into to see where it breaks. | |
| 21:16:49 | JamesBenson | But I've never dug into nova this deep. | |
| 21:18:11 | sean-k-mooney | well there seams to be 2 basic erros in the nova log | |
| 21:18:15 | sean-k-mooney | DBNotAllowed: nova-compute | |
| 21:18:20 | sean-k-mooney | and then the xen api one | |
| 21:18:59 | JamesBenson | agreed | |
| 21:19:24 | JamesBenson | Is it possible that kolla mucked up the install of that package? | |
| 21:19:52 | JamesBenson | But I thought that was pulled from the nova repo. | |
| 21:19:53 | JamesBenson | ? | |
| 21:20:47 | sean-k-mooney | well its posibel that the xen dirver need other config option to be set that kolla is nto setting in the nova.conf | |
| 21:22:28 | sean-k-mooney | the default for the databse are set here https://github.com/openstack/kolla-ansible/blob/2dd69e9140b1ce1bd248c5c09217fb3a6502a9fc/ansible/roles/nova/defaults/main.yml#L161-L167 and used to generate teh config nova.conf here https://github.com/openstack/kolla-ansible/blob/2dd69e9140b1ce1bd248c5c09217fb3a6502a9fc/ansible/roles/nova/templates/nova.conf.j2#L156-L164 | |
| 21:23:01 | sean-k-mooney | that code should be virt dirver independent | |
| 21:23:25 | mriedem | melwitt: heh yeah that PS4 on the transactional create patch is about a billion times cleaner | |
| 21:25:58 | mriedem | efried: https://review.opendev.org/#/c/652868/ | |
| 21:30:21 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.opendev.org/642899 | |
| 21:32:23 | efried | mriedem: Right, just wanted to make sure that was the final | |
| 21:32:39 | efried | mriedem: and presumably the one on top fails tests until the bottom one merges and the tag is created. | |
| 21:36:17 | jaypipes | dansmith, mriedem: is it possible to update metadata on a server that failed to boot properly? (i.e. the instance went to cell0) | |
| 21:36:35 | dansmith | jaypipes: metadata? like user metadata? | |
| 21:37:10 | mriedem | efried: i'm trying to figure out why the one on top is failing | |
| 21:37:12 | jaypipes | dansmith: no, sorry, metadata was a bad term. any attribute of the instance... say, it's name. | |
| 21:37:27 | efried | mriedem: It's because the tags aren't created yet | |
| 21:37:44 | efried | mriedem: I guess the job isn't smart enough to infer them from the releases in the predecessor patch. | |
| 21:37:57 | dansmith | jaypipes: shouldn't really be anything cell0 specific I don't think.. if you can do it in error, then you can do it | |
| 21:38:06 | dansmith | jaypipes: anything that requires talking to the host the instance is on won't work, of course | |
| 21:39:01 | mriedem | jaypipes: like dansmith says it depends on the API and what it restricts, | |
| 21:39:09 | mriedem | but i don't see any vm_state restrictions on the server update API | |
| 21:39:29 | dansmith | and what I meant was, there could be vm_state restrictions, but none that are specific to cell0 that I know of, | |
| 21:39:33 | dansmith | except something that ends up with an rpc call | |
| 21:39:56 | jaypipes | dansmith, mriedem: this is some custom Yahoo! API plugin ... it grabs the Instance object, updates some custom attribute on it and then calls instance.save(). Unfortunately, that isn't working in a cellsv2 environment... | |
| 21:39:59 | mriedem | interestingly there are restrictions on updating things like metadata and tags | |
| 21:40:17 | mriedem | jaypipes: likely b/c it's not targeted to the cell0 db | |
| 21:40:24 | dansmith | jaypipes: should be fine, just .. yeah, make sure it's targeted, | |
| 21:40:31 | mriedem | it's not cells-aware | |
| 21:40:35 | jaypipes | yeahg | |
| 21:40:44 | jaypipes | juno code. | |
| 21:40:51 | jaypipes | being converted to ocata. ugh. | |
| 21:40:54 | mriedem | if it were smart and used the common.get_instance method, the Instance._context would already be targeted | |
| 21:40:54 | dansmith | juno we don't give a shit, right? | |
| 21:41:08 | jaypipes | ocata. | |
| 21:41:23 | jaypipes | none of this is "smart" :) | |
| 21:41:30 | dansmith | JU .. NO | |
| 21:41:35 | mriedem | can i count verizon media group as a fallback employer to fix these old forks in case i get canned? | |
| 21:41:52 | dansmith | can anyone count on verizon media group as an employer? | |