Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
20:31:45 efried mordred: indeeed
20:33:08 mriedem odyssey4me: are the /etc/ files captured?
20:33:28 odyssey4me mriedem unfortunately not - we had to turn that off 'cos we were eating up all the infra inodes :/
20:33:50 odyssey4me we could add that back, but tar.gz them so we eat fewer inodes :)
20:33:58 mriedem ok, i can't tell if the [database]/connection value in nova.conf that the nova-compute is using is the same database connection as the cell1 this creates
20:36:42 mriedem odyssey4me: anyway i think we can see that the compute_nodes entry is created here for the compute host http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz#_2018-02-05_18_15_57_469
20:36:49 mriedem at 18:15:57.469
20:36:58 mriedem it shows up in the service list here:
20:36:58 mriedem http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_15_43_557748
20:37:09 mriedem "end": "2018-02-05 18:15:43.505982",
20:37:31 odyssey4me mriedem db config template is here, if that helps a bit: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/templates/nova.conf.j2#L213-L233
20:37:56 odyssey4me each db will have a unique name, and each db user/password will also be unique
20:38:17 dansmith kashyap: I explained my concern in more detail, maybe you could dig into docs/code tomorrow with that extra info to validate one way or the other?
20:39:33 mriedem odyssey4me: looks ok
20:39:37 openstackgerrit melanie witt proposed openstack/nova master: Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442
20:41:44 dansmith melwitt: nice
20:41:48 mriedem odyssey4me: also throw the --verbose option on create_cell
20:43:18 mriedem ultimately my guess is discover_hosts isn't picking up the compute host and mapping it to the cell
20:43:26 mriedem just not sure why
20:43:43 mriedem could be a timing issue, but that's what the openstack service list check is for
20:44:28 melwitt dansmith: thanks for pointing it out on the original review
20:44:36 dansmith melwitt: yar
20:44:48 efried mriedem: What's your trick for seeing those nova-new results before they're reported to gerrit?
20:44:57 dansmith melwitt: that race is the whole reason I wrote this eventing stuff so I get all sad when it gets broken
20:45:01 odyssey4me yeah, that's my one suspicion... a race condition of some sort - the other is that perhaps there's an online migration required which wasn't executed
20:45:19 mriedem efried: status.openstack.org/zuul
20:45:37 mriedem odyssey4me: i see that the online data migrations are skipped
20:45:37 efried Oh, look, those are links!
20:45:52 odyssey4me mriedem thanks again for looking through all that, and for sanity checking the process to verify that everything looks like it's doing what it should... I'll get a better set of test logs/configs going and we can revisit when those are good
20:46:16 mriedem odyssey4me: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_03_50_709904
20:46:22 mriedem <openstack1> Task "Disable the online migrations requirement" has been omitted from the job because the conditional "[u"inventory_hostname == groups['nova_conductor'][0]", u'not data_migrations | skipped', u'data_migrations | succeeded']" was evaluated as "False"
20:46:37 odyssey4me mriedem yep, they're skipped due to the check to see whether all hosts are upgraded before running them... and I don't think we have something in these role tests which actually check for that and execute them, so I think I'll patch that up
20:47:01 odyssey4me I thought we had one, but perhaps it got patched in later and not ported back
20:47:23 efried eff me, guess I need to wait for some of those four hundred patches to clear out of the check queue.
20:47:57 melwitt dansmith: understandable. I'm pretty new to the neutron event scene and have been learning
20:54:29 efried D'oh. The check queue seems to be smart enough not to run functional tests if no .py files were touched.
20:55:58 mriedem someone want to fast approve this? https://review.openstack.org/#/c/539978/
20:56:04 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: user headers https://review.openstack.org/541427
20:56:29 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: invalidate https://review.openstack.org/541430
20:56:45 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: connection params https://review.openstack.org/541432
20:57:14 efried mriedem: There is no fast. There is only zuul.
20:58:18 sean-k-mooney efried: zull used to be fast for small projects. now zuul is fair to all projects. as some who also works on small projects this make me sad
20:58:43 efried Seems snappy for nova-powervm - unless the whole world is broken.
21:00:16 sean-k-mooney efried: ah well the upstream zuul used to round robin taking the first job from each queue in a pipline so small pojects with short to no queue were more likely to have jobs run. now it handels jobs in the order it arrived being fair to all pojects
21:01:29 cdent melwitt, mriedem: would you have any objection to me setting a little goal for myself for nova: remove code from all the __init__.py files (to avoid inadvertent imports)? That is, if I were to do it, is there a chance it would get merged.
21:01:31 sean-k-mooney efried: if you have your own zuul instance for thirdparty testing then it is still fast for your on defintion(policy of job selection) of fast
21:02:41 mriedem cdent: that's somehow related to splitting out placement right?
21:03:22 cdent mriedem: I discovered it because of that, yes, but it's also beneficial for the rest of nova: the fact that it is a common behavior means that various processes are often importing many more modules than they actually need.
21:03:42 mriedem i'd say it has as good a chance at getting merged as anything else
21:04:24 sean-k-mooney cdent: mriedem even if it not related to splitting out placement can we do it anyway. haveing side effect form importing moduels makes testing harder and is really unintuitive to people who have not worked with that module before
21:04:46 cdent mriedem: for latest example see my note to self on https://review.openstack.org/#/c/541435/
21:04:51 mriedem sean-k-mooney: you act like i said no one could push patches
21:05:24 cdent would a spec or blueprint be of any value?
21:05:40 sean-k-mooney mriedem: hehe sorry i know that is not what you ment. i just hate seeing __init__.py files with anything but constants in them
21:05:47 mriedem cdent: idk
21:06:11 sean-k-mooney mriedem: its a pet pev of mine.
21:06:20 mriedem cdent: if it's a non-trivial amount of work, and is a depednency for the eventual placement split, then i could see a specless bp
21:07:04 cdent yeah, sadly it would probably be non-trivial as it often ends with wiggle here causing ripples way over there
21:07:56 cdent I'll put it my queue, thanks.
21:08:22 cdent and I think that can be the end of my work day
21:08:38 sean-k-mooney cdent: that part of why i dont like that patten in python. for example import anyting in nova.cmd results in your process being monkey patched https://github.com/openstack/nova/blob/master/nova/cmd/__init__.py
21:09:18 cdent yup
21:09:51 cdent 'night all
21:16:04 openstackgerrit Merged openstack/nova master: Add PowerVM to feature-classification https://review.openstack.org/533237
21:16:13 efried edleafe: yt?
21:21:58 kashyap dansmith: Okay, after watching the Falcon launch, I came back to IRC
21:22:17 kashyap dansmith: Saw your comment in the review. Much clearer.
21:22:35 kashyap So, I asked a libvirt dev who does migration to look into it -- Will the 'bandwidth' parameter of migrateToURI2() will be ignored if it was previously set by the API migrateSetMaxSpeed()?
21:23:09 mriedem dansmith: i'm going to announce this here publicly, but melwitt has never been asked to give a sizing
21:23:19 mriedem sdague: ^ is that an ibm only thing?
21:23:56 edleafe efried: ?
21:24:05 mriedem melwitt: this was much more fun before you said LOEs
21:24:08 efried edleafe: Wanted to do a quick brainstorm if you have a mo.
21:24:12 melwitt mriedem: haha, sorry
21:24:24 sean-k-mooney mriedem: a sizing of a task? if so how, i get ask that all the time for stuff i dont even work on
21:24:32 kashyap dansmith: Will respond on that review once I dig into the Python bindings and upstream relevant APIs to confirm what you asked. (Also double-confirm with a libvirt upstream dev.)
21:24:40 edleafe efried: sure
21:24:42 mriedem sean-k-mooney: of course you do, because you work at a corporation
21:25:12 sean-k-mooney mriedem: :) ah so thats where i went wrong.
21:25:20 efried edleafe: Trying to think through an easy way to get NUMA affinity in placement modeling. First off, do you understand the stuff Jay has been talking about, something about aggregate "distance"?
21:25:37 edleafe efried: vaguely
21:26:03 efried edleafe: Can you help me understand it?
21:26:10 openstackgerrit Takashi NATSUME proposed openstack/nova-specs master: List/show all server migration types https://review.openstack.org/540256
21:26:34 openstackgerrit Takashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration https://review.openstack.org/334732
21:26:50 sean-k-mooney mriedem: also because of that we are ment to track everything we do upstream and in jira and in downstream tools with vendors ... corporate processes :)
21:26:53 openstackgerrit Takashi NATSUME proposed openstack/nova master: Transform live_migration_post_dest notification https://review.openstack.org/469784
21:27:20 edleafe efried: my weak understanding is that it would favor RAM that is associated with a given NUMA node, but that if there wasn't enough, it would use RAM that was "further away"
21:27:37 efried edleafe: Okay, but how does this "distance" thing get defined?
21:27:52 edleafe efried: that's the part that I don't have any clue about
21:28:03 openstackgerrit Takashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells https://review.openstack.org/521400
21:28:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells https://review.openstack.org/521400
21:28:21 mriedem sean-k-mooney: do you track how much time you aren't spending working on grant requests when you should be?
21:28:29 efried edleafe: Okay, me neither. So starting from first principles, I was thinking about how one could model NUMA affinity. And here's what I came up with:
21:28:34 edleafe efried: you'd have to have Jay give a lesson on this (to all of us)
21:29:21 efried edleafe: GET /allocation_candidates?<...the usual resource requests...>&affine_resources=<resource_class>,<resource_class>,...
21:29:36 mriedem efried: L142 https://etherpad.openstack.org/p/nova-ptg-queens-placement
21:29:46 sean-k-mooney mriedem: hehe if it did i would cry
21:30:23 efried mriedem: Thanks. Will dig.
21:30:38 efried edleafe: So first of all, it would only apply to the unnumbered request group. Because you're already getting same-provider "affinity" in the numbered groups.
21:31:29 efried edleafe: So within the unnumbered request group, the API would force the resources from classes you list to come from either the same provider, or the same aggregate.

Earlier   Later