| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 20:43:18 | mriedem | ultimately my guess is discover_hosts isn't picking up the compute host and mapping it to the cell | |
| 20:43:26 | mriedem | just not sure why | |
| 20:43:43 | mriedem | could be a timing issue, but that's what the openstack service list check is for | |
| 20:44:28 | melwitt | dansmith: thanks for pointing it out on the original review | |
| 20:44:36 | dansmith | melwitt: yar | |
| 20:44:48 | efried | mriedem: What's your trick for seeing those nova-new results before they're reported to gerrit? | |
| 20:44:57 | dansmith | melwitt: that race is the whole reason I wrote this eventing stuff so I get all sad when it gets broken | |
| 20:45:01 | odyssey4me | yeah, that's my one suspicion... a race condition of some sort - the other is that perhaps there's an online migration required which wasn't executed | |
| 20:45:19 | mriedem | efried: status.openstack.org/zuul | |
| 20:45:37 | mriedem | odyssey4me: i see that the online data migrations are skipped | |
| 20:45:37 | efried | Oh, look, those are links! | |
| 20:45:52 | odyssey4me | mriedem thanks again for looking through all that, and for sanity checking the process to verify that everything looks like it's doing what it should... I'll get a better set of test logs/configs going and we can revisit when those are good | |
| 20:46:16 | mriedem | odyssey4me: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_03_50_709904 | |
| 20:46:22 | mriedem | <openstack1> Task "Disable the online migrations requirement" has been omitted from the job because the conditional "[u"inventory_hostname == groups['nova_conductor'][0]", u'not data_migrations | skipped', u'data_migrations | succeeded']" was evaluated as "False" | |
| 20:46:37 | odyssey4me | mriedem yep, they're skipped due to the check to see whether all hosts are upgraded before running them... and I don't think we have something in these role tests which actually check for that and execute them, so I think I'll patch that up | |
| 20:47:01 | odyssey4me | I thought we had one, but perhaps it got patched in later and not ported back | |
| 20:47:23 | efried | eff me, guess I need to wait for some of those four hundred patches to clear out of the check queue. | |
| 20:47:57 | melwitt | dansmith: understandable. I'm pretty new to the neutron event scene and have been learning | |
| 20:54:29 | efried | D'oh. The check queue seems to be smart enough not to run functional tests if no .py files were touched. | |
| 20:55:58 | mriedem | someone want to fast approve this? https://review.openstack.org/#/c/539978/ | |
| 20:56:04 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: user headers https://review.openstack.org/541427 | |
| 20:56:29 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: invalidate https://review.openstack.org/541430 | |
| 20:56:45 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: connection params https://review.openstack.org/541432 | |
| 20:57:14 | efried | mriedem: There is no fast. There is only zuul. | |
| 20:58:18 | sean-k-mooney | efried: zull used to be fast for small projects. now zuul is fair to all projects. as some who also works on small projects this make me sad | |
| 20:58:43 | efried | Seems snappy for nova-powervm - unless the whole world is broken. | |
| 21:00:16 | sean-k-mooney | efried: ah well the upstream zuul used to round robin taking the first job from each queue in a pipline so small pojects with short to no queue were more likely to have jobs run. now it handels jobs in the order it arrived being fair to all pojects | |
| 21:01:29 | cdent | melwitt, mriedem: would you have any objection to me setting a little goal for myself for nova: remove code from all the __init__.py files (to avoid inadvertent imports)? That is, if I were to do it, is there a chance it would get merged. | |
| 21:01:31 | sean-k-mooney | efried: if you have your own zuul instance for thirdparty testing then it is still fast for your on defintion(policy of job selection) of fast | |
| 21:02:41 | mriedem | cdent: that's somehow related to splitting out placement right? | |
| 21:03:22 | cdent | mriedem: I discovered it because of that, yes, but it's also beneficial for the rest of nova: the fact that it is a common behavior means that various processes are often importing many more modules than they actually need. | |
| 21:03:42 | mriedem | i'd say it has as good a chance at getting merged as anything else | |
| 21:04:24 | sean-k-mooney | cdent: mriedem even if it not related to splitting out placement can we do it anyway. haveing side effect form importing moduels makes testing harder and is really unintuitive to people who have not worked with that module before | |
| 21:04:46 | cdent | mriedem: for latest example see my note to self on https://review.openstack.org/#/c/541435/ | |
| 21:04:51 | mriedem | sean-k-mooney: you act like i said no one could push patches | |
| 21:05:24 | cdent | would a spec or blueprint be of any value? | |
| 21:05:40 | sean-k-mooney | mriedem: hehe sorry i know that is not what you ment. i just hate seeing __init__.py files with anything but constants in them | |
| 21:05:47 | mriedem | cdent: idk | |
| 21:06:11 | sean-k-mooney | mriedem: its a pet pev of mine. | |
| 21:06:20 | mriedem | cdent: if it's a non-trivial amount of work, and is a depednency for the eventual placement split, then i could see a specless bp | |
| 21:07:04 | cdent | yeah, sadly it would probably be non-trivial as it often ends with wiggle here causing ripples way over there | |
| 21:07:56 | cdent | I'll put it my queue, thanks. | |
| 21:08:22 | cdent | and I think that can be the end of my work day | |
| 21:08:38 | sean-k-mooney | cdent: that part of why i dont like that patten in python. for example import anyting in nova.cmd results in your process being monkey patched https://github.com/openstack/nova/blob/master/nova/cmd/__init__.py | |
| 21:09:18 | cdent | yup | |
| 21:09:51 | cdent | 'night all | |
| 21:16:04 | openstackgerrit | Merged openstack/nova master: Add PowerVM to feature-classification https://review.openstack.org/533237 | |
| 21:16:13 | efried | edleafe: yt? | |
| 21:21:58 | kashyap | dansmith: Okay, after watching the Falcon launch, I came back to IRC | |
| 21:22:17 | kashyap | dansmith: Saw your comment in the review. Much clearer. | |
| 21:22:35 | kashyap | So, I asked a libvirt dev who does migration to look into it -- Will the 'bandwidth' parameter of migrateToURI2() will be ignored if it was previously set by the API migrateSetMaxSpeed()? | |
| 21:23:09 | mriedem | dansmith: i'm going to announce this here publicly, but melwitt has never been asked to give a sizing | |
| 21:23:19 | mriedem | sdague: ^ is that an ibm only thing? | |
| 21:23:56 | edleafe | efried: ? | |
| 21:24:05 | mriedem | melwitt: this was much more fun before you said LOEs | |
| 21:24:08 | efried | edleafe: Wanted to do a quick brainstorm if you have a mo. | |
| 21:24:12 | melwitt | mriedem: haha, sorry | |
| 21:24:24 | sean-k-mooney | mriedem: a sizing of a task? if so how, i get ask that all the time for stuff i dont even work on | |
| 21:24:32 | kashyap | dansmith: Will respond on that review once I dig into the Python bindings and upstream relevant APIs to confirm what you asked. (Also double-confirm with a libvirt upstream dev.) | |
| 21:24:40 | edleafe | efried: sure | |
| 21:24:42 | mriedem | sean-k-mooney: of course you do, because you work at a corporation | |
| 21:25:12 | sean-k-mooney | mriedem: :) ah so thats where i went wrong. | |
| 21:25:20 | efried | edleafe: Trying to think through an easy way to get NUMA affinity in placement modeling. First off, do you understand the stuff Jay has been talking about, something about aggregate "distance"? | |
| 21:25:37 | edleafe | efried: vaguely | |
| 21:26:03 | efried | edleafe: Can you help me understand it? | |
| 21:26:10 | openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: List/show all server migration types https://review.openstack.org/540256 | |
| 21:26:34 | openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration https://review.openstack.org/334732 | |
| 21:26:50 | sean-k-mooney | mriedem: also because of that we are ment to track everything we do upstream and in jira and in downstream tools with vendors ... corporate processes :) | |
| 21:26:53 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Transform live_migration_post_dest notification https://review.openstack.org/469784 | |
| 21:27:20 | edleafe | efried: my weak understanding is that it would favor RAM that is associated with a given NUMA node, but that if there wasn't enough, it would use RAM that was "further away" | |
| 21:27:37 | efried | edleafe: Okay, but how does this "distance" thing get defined? | |
| 21:27:52 | edleafe | efried: that's the part that I don't have any clue about | |
| 21:28:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells https://review.openstack.org/521400 | |
| 21:28:10 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells https://review.openstack.org/521400 | |
| 21:28:21 | mriedem | sean-k-mooney: do you track how much time you aren't spending working on grant requests when you should be? | |
| 21:28:29 | efried | edleafe: Okay, me neither. So starting from first principles, I was thinking about how one could model NUMA affinity. And here's what I came up with: | |
| 21:28:34 | edleafe | efried: you'd have to have Jay give a lesson on this (to all of us) | |
| 21:29:21 | efried | edleafe: GET /allocation_candidates?<...the usual resource requests...>&affine_resources=<resource_class>,<resource_class>,... | |
| 21:29:36 | mriedem | efried: L142 https://etherpad.openstack.org/p/nova-ptg-queens-placement | |
| 21:29:46 | sean-k-mooney | mriedem: hehe if it did i would cry | |
| 21:30:23 | efried | mriedem: Thanks. Will dig. | |
| 21:30:38 | efried | edleafe: So first of all, it would only apply to the unnumbered request group. Because you're already getting same-provider "affinity" in the numbered groups. | |
| 21:31:29 | efried | edleafe: So within the unnumbered request group, the API would force the resources from classes you list to come from either the same provider, or the same aggregate. | |
| 21:32:19 | efried | edleafe: I'm not sure an anti_affine_resources makes any sense, but could be added if it does. | |
| 21:32:46 | edleafe | efried: "affine"? | |
| 21:33:12 | edleafe | efried: How would that specify RAM, PCI deveice, etc., associated with a given NUMA? | |
| 21:33:33 | efried | edleafe: Okay, let me give a more complete example, make more sense. | |
| 21:34:29 | sean-k-mooney | edleafe: anti_affine_resources for nic bonding e.g. antiafintiy of the pf for the VF request | |
| 21:34:40 | sean-k-mooney | efried: ^ | |
| 21:34:44 | efried | sean-k-mooney: ack | |
| 21:35:20 | efried | edleafe: To start with, model NUMA nodes as subtrees under the root compute host RP (which may not in fact provide any resources). So maybe the first tier children provide CPU & memory, and the second tier (grandchildren) are the network devices providing VFs. | |
| 21:35:53 | efried | edleafe: Associate each child/grandchildren in a given NUMA node with the same aggregate UUID. | |
| 21:36:49 | efried | edleafe: Then your request would be like GET /allocation_candidates?resources=VCPU:2,MEMORY_MB:2048,SRIOV_NET_VF:1,DISK_GB=2048&affine_resources=VCPU,MEMORY_MB,SRIOV_NET_VF | |
| 21:37:03 | edleafe | why can't the network devices be children of the NUMA node? | |
| 21:37:11 | efried | They are | |
| 21:37:14 | efried | swhat I said | |
| 21:37:33 | efried | Sorry, to rephrase: | |
| 21:37:36 | sean-k-mooney | efried: but the cpu and memory shoudl also be childeren of the numa node too | |
| 21:37:51 | edleafe | "and the second tier (grandchildren) are the network devices providing VFs." | |
| 21:37:53 | efried | sean-k-mooney: The CPU/mem are resources provided by the numa node which is the RP. | |