| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 20:31:02 | mriedem | melwitt: test coverage is obviously ok, and removing dead code | |
| 20:31:27 | melwitt | k | |
| 20:32:14 | mriedem | jaypipes: let me know before you push anything since i'd like to see a run with https://review.openstack.org/#/c/488187/ | |
| 20:32:43 | mriedem | sdague: i think the list_cells thing will have to be a next week thing | |
| 20:32:48 | mriedem | brain is fried on placement stuff today | |
| 20:33:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/488125 | |
| 20:33:28 | sdague | mriedem: that's fine, as long as we can still move it in | |
| 20:39:49 | jaypipes | mriedem: will do. | |
| 20:40:28 | mriedem | should have results in ~15 minutes for the live migration job | |
| 20:50:50 | mriedem | sdague: looks like https://review.openstack.org/#/c/487246/ broke the powervm in tree CI | |
| 20:50:55 | mriedem | efried: thorst: ^ | |
| 20:51:03 | mriedem | i'm seeing that here http://184.172.12.213/87/488187/1/check/nova-in-tree-pvm/9c61778/logs/ | |
| 20:51:17 | mriedem | openstack --os-cloud devstack-admin --os-region RegionOne compute service list --host powervm-ci-powervm-devstacked-28324 --service nova-compute -c ID -f value | |
| 20:51:19 | efried | mriedem looking | |
| 20:52:38 | mriedem | shit and it killed ironic | |
| 20:52:38 | mriedem | http://logs.openstack.org/46/487246/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/bec38d6/logs/devstacklog.txt.gz#_2017-07-26_11_55_57_117 | |
| 20:52:58 | sdague | mriedem: ah... gah | |
| 20:53:37 | sdague | right, all the compute for multiple hosts drivers | |
| 20:54:13 | sdague | mriedem: ok, hitting revert | |
| 20:54:48 | mriedem | ack | |
| 20:54:56 | mriedem | the one time i actually look at in-tree powervm CI results :) | |
| 20:54:59 | sdague | heh | |
| 20:55:13 | sdague | you get credit for finding it first | |
| 20:55:16 | mriedem | yay | |
| 20:55:21 | mriedem | i have notified the ironicers | |
| 20:55:33 | sdague | well the revert is submitted and approved | |
| 20:55:49 | mriedem | yeah i see it | |
| 20:56:17 | sdague | so... actually, why isn't the host set that way on powervm and ironic? | |
| 20:56:32 | openstackgerrit | Ed Leafe proposed openstack/nova master: Handle hash ring rebalancing in ironic flavor migration https://review.openstack.org/487954 | |
| 20:56:48 | sdague | efried: you have a powervm setup somewhere that you can query? | |
| 20:57:01 | efried | esberglu needs to be involved here. | |
| 20:59:17 | edleafe | dansmith: ^^ incorporated rloo's suggestions | |
| 20:59:47 | efried | That powervm failure *might* be unrelated. We shouldn't be trying to connect to localhost. | |
| 21:00:01 | efried | sdague Did that change *when* the compute service gets started? | |
| 21:00:02 | sdague | mriedem: I'm actually not sure why hostname wouldn't match in the db | |
| 21:00:15 | sdague | efried: ?? | |
| 21:00:23 | mriedem | efried: no | |
| 21:00:27 | efried | sdague Yeah, I wouldn't have thought so. | |
| 21:00:29 | mriedem | efried: it's polling for the compute node to show up | |
| 21:00:31 | mriedem | by the hostname | |
| 21:00:35 | efried | So the net is, we're looking into it. | |
| 21:01:00 | sdague | efried: I'm ok with a revert atm because it broke ironic, and we had enough breaks on them this week | |
| 21:01:10 | sdague | but I am curious why those don't seem to line up | |
| 21:01:55 | thorst | I think the main thing for powervm is it shouldn't be taking that long to start up...so that's what we're looking into :-/ | |
| 21:03:02 | mriedem | oh right i forgot it takes 10 minutes for the powervm node to register | |
| 21:03:04 | mriedem | in init_host | |
| 21:03:46 | tonyb | mikal, sdague, melwitt: I don't knwo if this email was wider distrubuted but you know how we moved last_bytes recently .. it seems it was used by nova-lxd | |
| 21:04:15 | melwitt | I think I saw that email | |
| 21:04:17 | sdague | tonyb: they are out of tree, kind of don't care | |
| 21:04:31 | tonyb | mikal, sdague, melwitt: having the (out of tree) nova-lxd driver call into the libvirt code isn't cool :( so shoudl I revert it? | |
| 21:04:42 | tonyb | sdague: Well that was my initial response | |
| 21:04:53 | edmondsw | mriedem I started a change to get the powervm driver up faster but put it aside when we closed things down for pike | |
| 21:05:04 | tonyb | sdague: especially as when we get to queens we move it again | |
| 21:05:09 | sdague | tonyb: no, show up and interact in the community if you are using internal functions in the rest of the tree | |
| 21:05:10 | melwitt | tonyb: didn't they say they're already copy-pasting it somewhere? | |
| 21:05:27 | mriedem | edmondsw: i don't see how nova has control over how fast the backend node comes up | |
| 21:05:32 | tonyb | melwitt: not in the email I have but there may be more | |
| 21:05:40 | mriedem | edmondsw: you were just working on auto-enable the service i thought | |
| 21:05:52 | melwitt | okay, lemme check. maybe I misunderstood it | |
| 21:05:55 | edmondsw | mriedem https://review.openstack.org/#/c/471773/ | |
| 21:07:00 | edmondsw | should complete init_host much faster | |
| 21:07:25 | sdague | tonyb: I wasn't on any such email, but my patience is low for out of tree driver that's not working in the community | |
| 21:08:12 | melwitt | tonyb: okay, I just saw it was the last comment in the review https://review.openstack.org/#/c/472228/ and he's saying that they've now copy-pasted it, not that they had been until now. so I misread it | |
| 21:08:20 | tonyb | okay so we more or less want to say, "Sorry. this is part of the move to privsep so until xxx merged you'll just need to work around it in your driver' | |
| 21:09:01 | tonyb | melwitt: Thanks. | |
| 21:09:15 | tonyb | I'll contact them after breakfast. | |
| 21:09:30 | melwitt | yeah, if it's going to be available again in privsep after the work is done, that seems not so bad to me | |
| 21:09:31 | tonyb | I just wanted to make check we're on the same page | |
| 21:09:37 | tonyb | cool | |
| 21:13:35 | sdague | tonyb: yes, also, work upstream :P | |
| 21:23:25 | efried | sdague mriedem Okay, we've figured out how that change affected us. We need to do some stuff to the systemctl service file and restart compute. As of now, we're doing that after devstack finishes, not caring that compute wasn't coming up all the way during the stack. With this change, it started to matter that compute wasn't starting. | |
| 21:23:52 | efried | So we're going to figure out how to tweak the service file before stacking starts. Should make compute come right up during the stack process. | |
| 21:24:28 | mriedem | live migration job is done on my sanity check change http://logs.openstack.org/87/488187/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/482b6f4/ | |
| 21:27:20 | mriedem | oh i've got a bug in there | |
| 21:33:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Sanity check delete_allocation_for_instance https://review.openstack.org/488187 | |
| 21:35:45 | dansmith | jaypipes: https://bugs.launchpad.net/nova/+bug/1707071 | |
| 21:35:46 | openstack | Launchpad bug 1707071 in OpenStack Compute (nova) "Compute nodes will fight over allocations during migration" [Undecided,New] | |
| 21:35:52 | dansmith | dude how awesome is that bug number? | |
| 21:36:03 | dansmith | palindromic and all primes | |
| 21:37:49 | dansmith | well, I guess zero isn't a prime.. damn | |
| 21:40:37 | mriedem | hmm, so on rebuild to the same host | |
| 21:40:54 | mriedem | conductor calls select_destinations | |
| 21:41:11 | mriedem | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L770 | |
| 21:41:23 | mriedem | will we double allocate that host then? | |
| 21:43:27 | mriedem | oh nvm | |
| 21:43:29 | mriedem | that's if not host | |
| 21:44:51 | mriedem | whew | |
| 21:47:44 | smcginnis | dansmith: You'll have to try to catch 1737371 | |
| 21:52:15 | dansmith | smcginnis: yeah | |
| 21:56:34 | melwitt | time to buy a lotto ticket | |
| 22:07:58 | mriedem | jaypipes: dansmith: ok, https://review.openstack.org/#/c/483566/ | |
| 22:08:06 | mriedem | there is one thing in there that worries me | |
| 22:08:19 | mriedem | https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@171 | |
| 22:09:31 | dansmith | hrm | |
| 22:10:13 | mriedem | oh L86 | |
| 22:10:13 | mriedem | if len(selected_hosts) < num_instances: | |
| 22:10:42 | dansmith | what happens if we return less than enough hosts to whatever calls us? | |
| 22:11:00 | dansmith | heh yeah that | |
| 22:11:30 | dansmith | and min/max_instances is only a quota check, so no problem there | |
| 22:12:29 | mriedem | ok +2 | |